duanjing7298 2017-07-04 13:27
浏览 901
已采纳

Golang：读取Gzip Reader时出现意外的EOF错误

I'm currently working on a project in which a client python script compresses the input by using gzip package, then, encrypts the compressed data according to AES, and, finally, it encodes the encrypted data in base64 encoding scheme. On the server side, which was written in golang I'm following the same steps in the reverse order to reach the original data.

As can be seen from the output, I can decode, decrypt and decompress the data received from the client on my test script. However, while reading the gzip.NewReader type it throws unexpected EOF error. (see the output at the very bottom of the question)

So far, I've done it without trimming the padding, however, then it gave me an invalid gzip header error. Then, just to verify if there are any significant differences between a valid gzip Reader and the one being created by the script, I gzipped the original data into a compressed form. Apparently, there were no notable differences I could see from outputs, but the differences in bytes which were most likely caused by the change in time that the readers are created:

Original data: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

Gzipped data: "\x1f\x8b\x08\x00\xf9w[Y\x02\xff%\x8e=\x0e\xc30\x08F\xaf\x82\x98\x91\x05\xe6\xc7\xa6c\xf7\x9e\xa0\xca\x96\xa5[\x86lQ\xee^\xdcN\xf0\xf4\xc1\x83\x0b?\xf8\x00|=\xe7D\x02< \xde\x17\xee\xab\xb85%\x82L\x02\xcb\xa6N\xa0\x7fri\xae\xd5K\xe1$\xe83\xc3\x08\x86Z\x81\xa9g-y\x88\xf6\x9a\xf5E\xde\x99\x7f\x96\xb1\xd5\x99\xb3\xfcb\x99\x121D\x1bG\xe7^.\xdcWPO\xdc\xdb\xfd\x05\x0ev\x15\x1d\x99\x00\x00\x00"

Gzip Reader created from the request by client

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

Gzip Reader created by directly reading the original data

GzReader2: &{{ [] 2017-07-04 13:11:53 +0200 CEST 255} 0xc4200741e0 0xc4200a9300 0 0 [31 139 8 0 249 119 91 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

Now, I'm quite curious about what might be causing this error and how this issue can be solved. I couldn't post the original scripts here due to some restrictions but these are the actual test scripts that I'm currently testing my code. I've tried to give as much as information I could think of at the moment, however, please feel free to ask me if you need any more explanation. I'd really appreciate if you can help me solve this!

Cheers!

Client Code (Tester): 

import time
import json
import requests
import random
import gzip
import base64

from Crypto.Cipher import AES

baseurl = 'http://0.0.0.0:80'
key = b'TfvY7I358yospfWKcoviZizOShpm5hyH'
iv = b'mb13KcoviZizvYhp'

MODE = AES.MODE_CFB
BLOCK_SIZE = 16
SEGMENT_SIZE = 128


def http_post(url, data):
    print('Going to make a request to {} with the following data: {}'.format(url, data))
    r = requests.post(url,
                      data=data,
                      headers={'Content-type': 'application/json; charset=utf-8',
                               'Connection': 'keep-alive'}, )
    if r.status_code != 200:
        print('Server returned unexpected response code {}, and content: {}'.format(r.status_code, r.content))
        return False
    else:
        data = r.json()
        return data


def post_sample_data(key, iv):
    fake_device_id = "MB88"
    Load_VA_Total_Mean = random.randint(1000, 100000)
    print('Data that should come back: {}'.format(Load_VA_Total_Mean))
    data = {"i": fake_device_id, "p": [
        {"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, Load_VA_Total_Mean, 10350, 55200, 49.7],
         "t": time.time(), "dt": "p"}]}
    url = baseurl + '/realtimedata'
    encryption_key_reference = 1
    payload = '{}
{}'.format(convert_pack(data, key, iv), encryption_key_reference)
    return http_post(url, payload)


def check_sample_request():
    fake_device_id = "MB88"
    url = baseurl + '/datapoints/powerzones/status'
    data = [{"name": "P1", "field": "Load_VA_Total_Mean", "devices": [{"id": fake_device_id, "multiplier": 1}]}]
    return http_post(url, json.dumps(data))


def convert_pack(inputdict, key, iv):
    print('input: {}'.format(inputdict))
    print('key: {}'.format(key))

    jsonpayload = json.dumps(inputdict)  # encode dict to json string
    print('json: {}'.format(jsonpayload))

    gzippayload = gzip.compress(jsonpayload.encode('utf-8'))  # compress with gzip
    print('gzip: {}'.format(gzippayload))

    encryptedpayload = base64.b64encode(encrypt(key, iv, gzippayload))
    print('encrypted: {}'.format(encryptedpayload))
    return str(encryptedpayload, encoding='utf-8')    


def _pad_string(value):
    length = len(value)
    pad_size = BLOCK_SIZE - (length % BLOCK_SIZE)
    return value.ljust(length + pad_size, b'\x00')


def encrypt(key, iv, plaintext):
    aes = AES.new(key, MODE, iv, segment_size=SEGMENT_SIZE)
    plaintext = _pad_string(plaintext)
    encrypted_text = aes.encrypt(plaintext)
    return encrypted_text


post_sample_data(key, iv)
check_sample_request()

Server Code (Tester):

package main

import (
    "crypto/aes"
    "crypto/cipher"
    "encoding/base64"
    "fmt"
    "compress/gzip"
    "io/ioutil"
    "bytes"
    "strings"
)

func main() {
    //a sample input text generated by the client script
    originalText := "xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv"
    fmt.Println("Original request: ", originalText)
    key := []byte("TfvY7I358yospfWKcoviZizOShpm5hyH")
    text := decrypt(key, originalText)

    gzReader, err1 := gzip.NewReader(bytes.NewReader(text))
    fmt.Println("GzReader: ", gzReader) 
    if err1 != nil {
        fmt.Println("error1: ", err1)
    }

    content, err2 := ioutil.ReadAll(gzReader)   
    if err2 != nil {
        fmt.Println("error2: ",err2)
    }

    fmt.Printf("
Reading: 
 -> Content in bytes: %v 
 -> Content in string: %s
", content, string(content)) 
}

func decrypt(key []byte, cryptoText string) []byte {
    ciphertext, _ := base64.StdEncoding.DecodeString(cryptoText)
    fmt.Printf("
Decoding: 
 -> Original: %v 
 -> Decoded in bytes: %v 
 -> Decoded in string: %s
", cryptoText, ciphertext, ciphertext)
    block, err := aes.NewCipher(key)
    if err != nil {
        panic(err)
    }

    iv := []byte("mb13KcoviZizvYhp")    
    stream := cipher.NewCFBDecrypter(block, iv)
    stream.XORKeyStream(ciphertext, ciphertext)
    fmt.Printf("
Decrypting: 
 -> Decrypted in bytes: %v 
 -> Decrypted in string: %s
", ciphertext, ciphertext)

    ciphertext = bytes.Trim(ciphertext, "\x00")
    fmt.Printf("
Trimming the padding: 
 -> Decrypted in bytes: %v 
 -> Decrypted in string: %s

", ciphertext, ciphertext)    
    return ciphertext
}

The output in terminal:

Original request:

xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

Decoding:

-> Original: xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

-> Decoded in bytes: [198 24 105 46 163 9 179 136 60 251 193 219 239 157 91 113 169 101 142 71 135 12 151 219 25 211 42 175 209 61 138 109 78 146 34 115 153 210 235 238 149 175 252 106 225 211 235 179 117 227 16 4 204 222 126 172 211 175 55 173 127 87 217 112 25 189 220 129 5 235 9 124 89 129 57 199 107 128 103 149 216 223 214 241 149 21 131 183 135 19 54 79 240 90 43 27 0 196 179 52 154 144 82 173 148 154 160 75 105 103 8 208 181 53 52 99 217 119 193 10 28 173 250 35 9 105 240 219 143 249 139 16 32 148 166 65 211 118 114 243 69 56 177 175]

-> Decoded in string: �i.� ��<����[q�e�G� ���*��=�mN�"s����j���u���~�ӯ7�W�p�܁� |Y�9�k�g��������6O�Z+ij4��R����Kiе54c�w� ��# i�ۏ�� ��A�vr�E8��

Decrypting:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] �q�D|7

Trimming the padding:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153] �q�D|7

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

error2: unexpected EOF

Reading:

-> Content in bytes: [123 34 105 34 58 32 34 77 66 56 56 34 44 32 34 112 34 58 32 91 123 34 100 34 58 32 91 53 52 46 51 44 32 48 44 32 57 57 44 32 52 57 46 51 53 44 32 51 44 32 57 57 44 32 53 49 46 53 51 51 44 32 49 44 32 57 56 44 32 50 56 57 54 52 44 32 55 51 52 56 44 32 52 51 53 57 48 44 32 57 55 49 48 55 44 32 49 48 51 53 48 44 32 53 53 50 48 48 44 32 52 57 46 55 93 44 32 34 116 34 58 32 49 52 57 57 48 56 57 50 49 50 46 49 51 54 55 56 57 44 32 34 100 116 34 58 32 34 112 34 125 93 125]

-> Content in string: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

  • 写回答

1条回答 默认 最新

  • dtsc14683 2017-07-04 14:00
    关注

    When you think you're "trimming the padding", you're removing three zeros that belong on the end of the gzip stream. Just because it's a zero doesn't mean it's padding. Keep the three zeros after the 153.

    When you're not trimming the padding, a gzip decoder will look for another gzip stream after it successfully decompresses the first one. Seeing zeros instead of a gzip header results in the invalid gzip header error.

    You need to keep track of the size of the gzip stream, so that you know what's padding and what isn't. Or you can decode a gzip stream at a time, and when you see following zeros, you can conclude that that's padding.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题？

悬赏问题

  • ¥60 版本过低apk如何修改可以兼容新的安卓系统
  • ¥25 由IPR导致的DRIVER_POWER_STATE_FAILURE蓝屏
  • ¥50 有数据，怎么建立模型求影响全要素生产率的因素
  • ¥50 有数据，怎么用matlab求全要素生产率
  • ¥15 TI的insta-spin例程
  • ¥15 完成下列问题完成下列问题
  • ¥15 C#算法问题, 不知道怎么处理这个数据的转换
  • ¥15 YoloV5 第三方库的版本对照问题
  • ¥15 请完成下列相关问题！
  • ¥15 drone 推送镜像时候 purge: true 推送完毕后没有删除对应的镜像,手动拷贝到服务器执行结果正确在样才能让指令自动执行成功删除对应镜像，如何解决？