问题遇到的现象和发生背景
下载大文件 >800M，程序异常报错
程序代码
import time
import requests
url = "https://api.github.com/repos/specterbike/aosp.frameworks.base/zipball/refs/tags/android-cts-5.1_r15"
s_time = time.time()
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding": "gzip, deflate, br"
}
res = requests.get(url, headers=headers, stream=True)
code = res.status_code
res.encoding = "utf-8"
file_name = "android-cts-5.1_r15.zip"
if code == 200:
with open(file_name, "wb") as f:
for text in res.iter_content(chunk_size=1024 * 4):
if text:
f.write(text)
print(f"用时：{time.time() - s_time}")
运行结果及报错内容
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 685, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 425, in _error_catcher
yield
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 752, in read_chunked
self._update_chunk_length()
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 689, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 753, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 560, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 781, in read_chunked
self._original_response.close()
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 443, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "download_.py", line 32, in <module>
for text in res.iter_content(chunk_size=1024 * 4):
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 756, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
我的解答思路和尝试过的方法
- 使用 stream=True 和 iter_content 字节流下载文件，减少内存的使用
