golang将struct作为原始数据写入

Question

I am working on a new type of database, using GO. One of the things I would like to do is have a distributed disk so that I can distribute queries over multiple machines (think Pi type architectures). This means building my own structures on raw disk.

My challenge is that I can't find a GO package that will let me write N bytes from a pointer to a structure. All the IO packages limit the access to []byte slices.

That's nice for protection, but if I have to buffer everything through a byte array via some form of encoding it will slow down the access to a specific object.

Anyone got any idea on how to do raw IO? Or am I going to have to handle GOBs as my unit of IO and suffer the penalty for encoding/decoding?

Answer 1

Big warning first: don't do it: it is neither safe nor portable

For a given struct, you can reflect over it to figure out the in-memory size of the actual struct, then unsafely cast it to a []byte using unsafe.

eg: (*[in-mem size]byte)(unsafe.Pointer(&mystruct))

This will give you something C-ish with absolutely no safety guarantees or portability.

I'll quote the Go spec:

A package using unsafe must be vetted manually for type safety and may not be portable.

You can find a lot more details in this Go and Memory layout post, including all the steps you need to unsafely treat structs as just bytes.

Overall, it's fascinating to examine how Go functions on a low level, but this is absolutely the wrong thing to do in your case. Any real data infrastructure will need storage logic way more complicated than just dumping in-memory structs to disk anyway.