Go：将数组的指针传递给gob而不进行复制？

Question

I have a very, very large array (not slice) of maps that I am then trying to encode. I really need to avoid making a copy of the array but I can't figure out how to do this.

So I far I have this:

func doSomething() {
 var mygiantvar [5]map[string]Searcher
 mygiantvar = Load()
 Save(`file.gob.gz`, &mygiantvar)
}

func Save(filename string, variable *[5]map[string]Searcher) error {
    // Open file for writing
    fi, err := os.Create(filename)
    if err !=nil {
        return err
    }
    defer fi.Close()
    // Attach gzip writer
    fz := gzip.NewWriter(fi)
    defer fz.Close()
    // Push from the gob encoder
    encoder := gob.NewEncoder(fz)
    err = encoder.Encode(*variable)
    if err !=nil {
        return err
    }
    return nil
}

From my understanding that will pass a pointer of mygiantvar to Save, which saves the first copy. But then the entire array will surely be copied into encoder.Encode which will then copy it around many more functions, right?

This mygiantvar variable will be something like 10GB in size. So it must avoid being copied ever.

But then again perhaps only the actual array [5] part is copied but the maps inside of this are pointers inside an array, so the array of pointers to maps would be copied instead of the maps themselves? I have no idea about this - it's all very confusing.

Any ideas?

Answer 1

Note that Encoder.Encode will pass around an interface{}.

func (enc *Encoder) Encode(v interface{}) error {

That means a kind of a pointer to whatever you will be passing to it, as I described in "what is the meaning of interface{} in golang?"
(see also "Why can't I assign a *Struct to an *Interface?")

An interface value isn't the value of the concrete struct (as it has a variable size, this wouldn't be possible), but it's a kind of pointer (to be more precise a pointer to the struct and a pointer to the type)

That means it won't copy the full content of your map (or here of your array).
Since array is a value, you could slice it to avoid any copy during the call to Encode():

err = encoder.Encode(*variable[:])

See "Go Slices: usage and internals"

This is also the syntax to create a slice given an array:

x := [3]string{"Лайка", "Белка", "Стрелка"}
s := x[:] // a slice referencing the storage of x

If that doesn't work, you can keep *variable (here an array: [5]map[string]Searcher), as map types are reference types, like pointers or slices: the copy won't be huge.
See "Go maps in action".

While the array will be copied when passed to interface{}, the map content won't be copied.
See this <kbd>play.golang.org</kbd> example:

package main

import "fmt"

func main() {
    var a [1]map[string]int
    a[0] = make(map[string]int)
    a[0]["test"] = 0
    modify(a)
    fmt.Println(a)
}

func modify(arr interface{}) {
    a := arr.([1]map[string]int)
    a[0]["test"] = -1
}

Output:

[map[test:-1]]