duanjing7298 2017-07-04 13:27
浏览 911
已采纳

Golang：读取Gzip Reader时出现意外的EOF错误

I'm currently working on a project in which a client python script compresses the input by using gzip package, then, encrypts the compressed data according to AES, and, finally, it encodes the encrypted data in base64 encoding scheme. On the server side, which was written in golang I'm following the same steps in the reverse order to reach the original data.

As can be seen from the output, I can decode, decrypt and decompress the data received from the client on my test script. However, while reading the gzip.NewReader type it throws unexpected EOF error. (see the output at the very bottom of the question)

So far, I've done it without trimming the padding, however, then it gave me an invalid gzip header error. Then, just to verify if there are any significant differences between a valid gzip Reader and the one being created by the script, I gzipped the original data into a compressed form. Apparently, there were no notable differences I could see from outputs, but the differences in bytes which were most likely caused by the change in time that the readers are created:

Original data: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

Gzipped data: "\x1f\x8b\x08\x00\xf9w[Y\x02\xff%\x8e=\x0e\xc30\x08F\xaf\x82\x98\x91\x05\xe6\xc7\xa6c\xf7\x9e\xa0\xca\x96\xa5[\x86lQ\xee^\xdcN\xf0\xf4\xc1\x83\x0b?\xf8\x00|=\xe7D\x02< \xde\x17\xee\xab\xb85%\x82L\x02\xcb\xa6N\xa0\x7fri\xae\xd5K\xe1$\xe83\xc3\x08\x86Z\x81\xa9g-y\x88\xf6\x9a\xf5E\xde\x99\x7f\x96\xb1\xd5\x99\xb3\xfcb\x99\x121D\x1bG\xe7^.\xdcWPO\xdc\xdb\xfd\x05\x0ev\x15\x1d\x99\x00\x00\x00"

Gzip Reader created from the request by client

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

Gzip Reader created by directly reading the original data

GzReader2: &{{ [] 2017-07-04 13:11:53 +0200 CEST 255} 0xc4200741e0 0xc4200a9300 0 0 [31 139 8 0 249 119 91 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

Now, I'm quite curious about what might be causing this error and how this issue can be solved. I couldn't post the original scripts here due to some restrictions but these are the actual test scripts that I'm currently testing my code. I've tried to give as much as information I could think of at the moment, however, please feel free to ask me if you need any more explanation. I'd really appreciate if you can help me solve this!

Cheers!

Client Code (Tester): 

import time
import json
import requests
import random
import gzip
import base64

from Crypto.Cipher import AES

baseurl = 'http://0.0.0.0:80'
key = b'TfvY7I358yospfWKcoviZizOShpm5hyH'
iv = b'mb13KcoviZizvYhp'

MODE = AES.MODE_CFB
BLOCK_SIZE = 16
SEGMENT_SIZE = 128


def http_post(url, data):
    print('Going to make a request to {} with the following data: {}'.format(url, data))
    r = requests.post(url,
                      data=data,
                      headers={'Content-type': 'application/json; charset=utf-8',
                               'Connection': 'keep-alive'}, )
    if r.status_code != 200:
        print('Server returned unexpected response code {}, and content: {}'.format(r.status_code, r.content))
        return False
    else:
        data = r.json()
        return data


def post_sample_data(key, iv):
    fake_device_id = "MB88"
    Load_VA_Total_Mean = random.randint(1000, 100000)
    print('Data that should come back: {}'.format(Load_VA_Total_Mean))
    data = {"i": fake_device_id, "p": [
        {"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, Load_VA_Total_Mean, 10350, 55200, 49.7],
         "t": time.time(), "dt": "p"}]}
    url = baseurl + '/realtimedata'
    encryption_key_reference = 1
    payload = '{}
{}'.format(convert_pack(data, key, iv), encryption_key_reference)
    return http_post(url, payload)


def check_sample_request():
    fake_device_id = "MB88"
    url = baseurl + '/datapoints/powerzones/status'
    data = [{"name": "P1", "field": "Load_VA_Total_Mean", "devices": [{"id": fake_device_id, "multiplier": 1}]}]
    return http_post(url, json.dumps(data))


def convert_pack(inputdict, key, iv):
    print('input: {}'.format(inputdict))
    print('key: {}'.format(key))

    jsonpayload = json.dumps(inputdict)  # encode dict to json string
    print('json: {}'.format(jsonpayload))

    gzippayload = gzip.compress(jsonpayload.encode('utf-8'))  # compress with gzip
    print('gzip: {}'.format(gzippayload))

    encryptedpayload = base64.b64encode(encrypt(key, iv, gzippayload))
    print('encrypted: {}'.format(encryptedpayload))
    return str(encryptedpayload, encoding='utf-8')    


def _pad_string(value):
    length = len(value)
    pad_size = BLOCK_SIZE - (length % BLOCK_SIZE)
    return value.ljust(length + pad_size, b'\x00')


def encrypt(key, iv, plaintext):
    aes = AES.new(key, MODE, iv, segment_size=SEGMENT_SIZE)
    plaintext = _pad_string(plaintext)
    encrypted_text = aes.encrypt(plaintext)
    return encrypted_text


post_sample_data(key, iv)
check_sample_request()

Server Code (Tester):

package main

import (
    "crypto/aes"
    "crypto/cipher"
    "encoding/base64"
    "fmt"
    "compress/gzip"
    "io/ioutil"
    "bytes"
    "strings"
)

func main() {
    //a sample input text generated by the client script
    originalText := "xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv"
    fmt.Println("Original request: ", originalText)
    key := []byte("TfvY7I358yospfWKcoviZizOShpm5hyH")
    text := decrypt(key, originalText)

    gzReader, err1 := gzip.NewReader(bytes.NewReader(text))
    fmt.Println("GzReader: ", gzReader) 
    if err1 != nil {
        fmt.Println("error1: ", err1)
    }

    content, err2 := ioutil.ReadAll(gzReader)   
    if err2 != nil {
        fmt.Println("error2: ",err2)
    }

    fmt.Printf("
Reading: 
 -> Content in bytes: %v 
 -> Content in string: %s
", content, string(content)) 
}

func decrypt(key []byte, cryptoText string) []byte {
    ciphertext, _ := base64.StdEncoding.DecodeString(cryptoText)
    fmt.Printf("
Decoding: 
 -> Original: %v 
 -> Decoded in bytes: %v 
 -> Decoded in string: %s
", cryptoText, ciphertext, ciphertext)
    block, err := aes.NewCipher(key)
    if err != nil {
        panic(err)
    }

    iv := []byte("mb13KcoviZizvYhp")    
    stream := cipher.NewCFBDecrypter(block, iv)
    stream.XORKeyStream(ciphertext, ciphertext)
    fmt.Printf("
Decrypting: 
 -> Decrypted in bytes: %v 
 -> Decrypted in string: %s
", ciphertext, ciphertext)

    ciphertext = bytes.Trim(ciphertext, "\x00")
    fmt.Printf("
Trimming the padding: 
 -> Decrypted in bytes: %v 
 -> Decrypted in string: %s

", ciphertext, ciphertext)    
    return ciphertext
}

The output in terminal:

Original request:

xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

Decoding:

-> Original: xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

-> Decoded in bytes: [198 24 105 46 163 9 179 136 60 251 193 219 239 157 91 113 169 101 142 71 135 12 151 219 25 211 42 175 209 61 138 109 78 146 34 115 153 210 235 238 149 175 252 106 225 211 235 179 117 227 16 4 204 222 126 172 211 175 55 173 127 87 217 112 25 189 220 129 5 235 9 124 89 129 57 199 107 128 103 149 216 223 214 241 149 21 131 183 135 19 54 79 240 90 43 27 0 196 179 52 154 144 82 173 148 154 160 75 105 103 8 208 181 53 52 99 217 119 193 10 28 173 250 35 9 105 240 219 143 249 139 16 32 148 166 65 211 118 114 243 69 56 177 175]

-> Decoded in string: �i.� ��<����[q�e�G� ���*��=�mN�"s����j���u���~�ӯ7�W�p�܁� |Y�9�k�g��������6O�Z+ij4��R����Kiе54c�w� ��# i�ۏ�� ��A�vr�E8��

Decrypting:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] �q�D|7

Trimming the padding:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153] �q�D|7

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

error2: unexpected EOF

Reading:

-> Content in bytes: [123 34 105 34 58 32 34 77 66 56 56 34 44 32 34 112 34 58 32 91 123 34 100 34 58 32 91 53 52 46 51 44 32 48 44 32 57 57 44 32 52 57 46 51 53 44 32 51 44 32 57 57 44 32 53 49 46 53 51 51 44 32 49 44 32 57 56 44 32 50 56 57 54 52 44 32 55 51 52 56 44 32 52 51 53 57 48 44 32 57 55 49 48 55 44 32 49 48 51 53 48 44 32 53 53 50 48 48 44 32 52 57 46 55 93 44 32 34 116 34 58 32 49 52 57 57 48 56 57 50 49 50 46 49 51 54 55 56 57 44 32 34 100 116 34 58 32 34 112 34 125 93 125]

-> Content in string: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

  • 写回答

1条回答 默认 最新

  • dtsc14683 2017-07-04 14:00
    关注

    When you think you're "trimming the padding", you're removing three zeros that belong on the end of the gzip stream. Just because it's a zero doesn't mean it's padding. Keep the three zeros after the 153.

    When you're not trimming the padding, a gzip decoder will look for another gzip stream after it successfully decompresses the first one. Seeing zeros instead of a gzip header results in the invalid gzip header error.

    You need to keep track of the size of the gzip stream, so that you know what's padding and what isn't. Or you can decode a gzip stream at a time, and when you see following zeros, you can conclude that that's padding.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题？

悬赏问题

  • ¥15 乌班图ip地址配置及远程SSH
  • ¥15 怎么让点阵屏显示静态爱心，用keiluVision5写出让点阵屏显示静态爱心的代码，越快越好
  • ¥15 PSPICE制作一个加法器
  • ¥15 javaweb项目无法正常跳转
  • ¥15 VMBox虚拟机无法访问
  • ¥15 skd显示找不到头文件
  • ¥15 机器视觉中图片中长度与真实长度的关系
  • ¥15 fastreport table 怎么只让每页的最下面和最顶部有横线
  • ¥15 R语言卸载之后无法重装，显示电脑存在下载某些较大二进制文件行为，怎么办
  • ¥15 java 的protected权限 ，问题在注释里