使用flate.NewWriterDict进行压缩很奇怪

Question

Problem

first process

compressed_1 : []byte{0xf2, 0x48, 0xcd, 0xc9, 0xc9, 0x7, 0x0}

second process(compress with dictionary)

compressed_2 : []byte{0x2, 0x13, 0x0, 0x0}

but I expected

compressed_2 : []byte{0xf2, 0x0, 0x11, 0x0, 0x0}

The basis of my expectation is from the following. https://tools.ietf.org/html/rfc7692#section-7.2.3.2

Other implementations of python have confirmed that they will be compressed with the above correct result. https://github.com/aaugustin/websockets

I do not know if the cause is go code or whether my execution method is bad. If you are familiar with zlib and flate, please advise me...

Code

executed code

package flate_dict

import (
    "bytes"
    "compress/flate"
    "fmt"
)

func compressWithDict(level int) {

    // compress
    src := []byte("Hello")

    compressed := &bytes.Buffer{}
    compressor, _ := flate.NewWriter(compressed, level)
    compressor.Write(src)
    compressor.Flush()
    compressor.Close()

    dict := compressed.Bytes()
    dict = dict[:len(dict)-9]
    fmt.Printf("compressed_1 : %#v
", dict)

    // compress
    dct := []byte{0x48, 0x65, 0x6c, 0x6c, 0x6f}
    src_2 := []byte("Hello")

    compressed_2 := &bytes.Buffer{}
    compressor_2, _ := flate.NewWriterDict(compressed_2, level, dct)
    compressor_2.Write(src_2)
    compressor_2.Flush()
    compressor_2.Close()

    dict_2 := compressed_2.Bytes()
    dict_2 = dict_2[:len(dict_2)-9]

    fmt.Printf("compressed_2 : %#v

", dict_2)
}

test code for debug

package flate_dict

import "testing"

func Test_compressWithDict(t *testing.T) {
    tests := []struct {
        name  string
        level int
    }{
        {
            name:  "01",
            level: -2,
        },
        {
            name:  "02",
            level: -1,
        },
        {
            name:  "03",
            level: 0,
        },
        {
            name:  "04",
            level: 1,
        },
        {
            name:  "05",
            level: 2,
        },
        {
            name:  "06",
            level: 3,
        },
        {
            name:  "07",
            level: 4,
        },
        {
            name:  "08",
            level: 5,
        },
        {
            name:  "09",
            level: 6,
        },
        {
            name:  "10",
            level: 7,
        },
        {
            name:  "11",
            level: 8,
        },
        {
            name:  "12",
            level: 9,
        },
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            compressWithDict(tt.level)
        })
    }
}

Answer 1

In general, compression implementations are not guaranteed to produce identical outputs, they only ensure that the output can be reversed by another implementation following the specification. You're calling flush an extra time here, which adds another sync marker, which is why you have to slice off 8 bytes at the end to match your output, rather than 4.

Without the example code for comparison, we can't tell exactly what you're comparing against, but the output appears to be what you expected:

dict := []byte{0x48, 0x65, 0x6c, 0x6c, 0x6f}

out := &bytes.Buffer{}

cmp, _ := flate.NewWriterDict(out, 9, dict)
cmp.Write([]byte("Hello"))
cmp.Flush()

fmt.Printf("deflated: %#v
", out.Bytes())

r := flate.NewReaderDict(out, dict)
orig, _ := ioutil.ReadAll(r)

fmt.Println("inflated:", string(orig))

// deflated: []byte{0x2, 0x13, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff}
// inflated: Hello

https://play.golang.org/p/Kk_mBytkbD2