使用binary.PutVarint（…）时索引超出范围

Question

http://play.golang.org/p/RqScJVvpS7

package main

import (
    "fmt"
    "math/rand"
    "encoding/binary"
)

func main() {
    buffer := []byte{0, 0, 0, 0, 0, 0, 0, 0}
    num := rand.Int63()
    count := binary.PutVarint(buffer, num)

    fmt.Println(count)
}

I had this working awhile ago when num was just an incrementing uint64 and I was using binary.PutUvarint but now that it's a random int64 and binary.PutVarint I get an error:

panic: runtime error: index out of range

goroutine 1 [running]:
encoding/binary.PutUvarint(0x1042bf58, 0x8, 0x8, 0x6ccb, 0xff9faa4, 0x9acb0442, 0x7fcfd52, 0x4d658221)
    /usr/local/go/src/encoding/binary/varint.go:44 +0xc0
encoding/binary.PutVarint(0x1042bf58, 0x8, 0x8, 0x6ccb, 0x7fcfd52, 0x4d658221, 0x14f9e0, 0x104000e0)
    /usr/local/go/src/encoding/binary/varint.go:83 +0x60
main.main()
    /tmp/sandbox010341234/main.go:12 +0x100

What am I missing? I would have thought this to be a trivial change...

EDIT: I just tried extending my buffer array. For some odd reason it works and I get a count of 10. How can that be? int64 is 64 bits = 8 bytes, right?

Answer 1

Quoting the doc of encoding/binary:

The varint functions encode and decode single integer values using a variable-length encoding; smaller values require fewer bytes. For a specification, see https://developers.google.com/protocol-buffers/docs/encoding.

So the binary.PutVarint() is not a fixed, but a variable-length encoding. When passing an int64, it will need more than 8 bytes for large numbers, and less than 8 bytes for small numbers. Since the number you're encoding is a random number, it will have random bits even in its highest byte.

See this simple example:

buffer := make([]byte, 100)
for num := int64(1); num < 1<<60; num <<= 4 {
    count := binary.PutVarint(buffer, num)
    fmt.Printf("Num=%d, bytes=%d
", num, count)
}

Output:

Num=1, bytes=1
Num=16, bytes=1
Num=256, bytes=2
Num=4096, bytes=2
Num=65536, bytes=3
Num=1048576, bytes=4
Num=16777216, bytes=4
Num=268435456, bytes=5
Num=4294967296, bytes=5
Num=68719476736, bytes=6
Num=1099511627776, bytes=6
Num=17592186044416, bytes=7
Num=281474976710656, bytes=8
Num=4503599627370496, bytes=8
Num=72057594037927936, bytes=9

The essence of variable-length encoding is that small numbers use less bytes, but this can only be achieved if in turn big numbers may use more than 8 bytes (that would be size of int64).

Details of the specific encoding is on the linked page.

A very easy example would be: A byte is 8 bits. Use 7 bits of the output byte as the "useful" bits to encode the data/number. If the highest bit is 1, that means more bytes are required. If highest bit is 0, we're done. You can see that small numbers can be encoded using 1 output byte (e.g. n=10), while we're using 1 extra bit for every 7-bit useful data, so if the input number uses all the 64 bits, we will end up with more than 8 bytes: 10 groups are required to cover 64 bits, so we will need 10 bytes (9 groups is only 9*7=63 bits).