当字段使用不同的值时，Protobuf的序列化长度会有所不同

Question

My Protobuf message consists of 3 doubles

syntax = "proto3";

message TestMessage{
  double input = 1;
  double output = 2;
  double info = 3;
}

When I set these values to

test.set_input(2.3456);
test.set_output(5.4321);
test.set_info(5.0);

the serialized message looks like

00000000  09 16 fb cb ee c9 c3 02  40 11 0a 68 22 6c 78 ba  |........@..h"lx.|
00000010  15 40 19                                          |.@.|
00000013

when using test.serializeToArray and could not be deserialized successfully by a go program using the same protobuf message. When trying to read it from a c++ program I got a 0 as info, so the message seems to be corrupted.

When using test.serializeToOstream I got this message, which could be deserialized successfully by both go and c++ programs.

00000000  09 16 fb cb ee c9 c3 02  40 11 0a 68 22 6c 78 ba  |........@..h"lx.|
00000010  15 40 19 00 00 00 00 00  00 14 40               |.@........@|
0000001b

When setting the values to

test.set_input(2.3456);
test.set_output(5.4321);
test.set_info(5.5678);

the serialized messages, both produced by test.serializeToArray and test.serializeToOstream look like

00000000  09 16 fb cb ee c9 c3 02  40 11 0a 68 22 6c 78 ba  |........@..h"lx.|
00000010  15 40 19 da ac fa 5c 6d  45 16 40                 |.@....\mE.@|
0000001b

and could be successfully read by my go and cpp program.

What am I missing here? Why is serializeToArray not working in the first case?

EDIT: As it turns out, serializeToString works fine, too.

Here the code I used for the comparison:

file_a.open(FILEPATH_A);
file_b.open(FILEPATH_B);

test.set_input(2.3456);
test.set_output(5.4321);
test.set_info(5.0);

//serializeToArray
int size = test.ByteSize();
char *buffer = (char*) malloc(size);
test.SerializeToArray(buffer, size);
file_a << buffer;

//serializeToString
std::string buf;
test.SerializeToString(&buf);
file_b << buf;

file_a.close();
file_b.close();

Why does serializeToArray not work as expected?

EDIT2:

When using file_b << buf.data() instead of file_b << buf.data(), the data gets corrupted as well, but why?

Answer 1

I think the error you're making is treating binary as character data and using character data APIs. Many of those APIs stop at the first nil byte (0), but that is a totally valid value in protobuf binary.

You need to make sure you don't use any such APIs basically - stick purely to binary safe APIs.

Since you indicate that size is 27, this all fits.

Basically, the binary representation of 5.0 includes 0 bytes, but you could easily have seen the same problem for other values in time.