求助 python爬虫 requests的get出现10054错误 远程主机强迫关闭了一个现有的连接

Question

这个爬的是智联的招聘信息，在爬了上千条数据后就容易出现这个错误。

Traceback (most recent call last):
File "G:/python/zhilian_crawler/crawler3.py", line 284, in
get_more_page(1,91)
File "G:/python/zhilian_crawler/crawler3.py", line 170, in get_more_page
get_page(url+str(one))
File "G:/python/zhilian_crawler/crawler3.py", line 28, in get_page
get_more_data(soup)
File "G:/python/zhilian_crawler/crawler3.py", line 159, in get_more_data
webData=requests.get(url0)
File "D:\Python3\lib\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "D:\Python3\lib\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "D:\Python3\lib\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "D:\Python3\lib\requests\sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "D:\Python3\lib\requests\adapters.py", line 473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))

Answer 1

1、head中有一个user-agent每次都换不同的模拟代理
2、测试最大访问时间短，每秒多少次请求会出现中断，然后sleep（200）毫秒
2、每次请求中断上一次链接，keep-live=false
试试看，
不然就是换IP访问

Answer 2

应该是服务器发现你的爬虫行为了，所有强制断开链接了。

Answer 3

服务器知道你是爬虫，加headers, 模拟浏览器agent ，爬完一个暂停几秒

Answer 4

import time

Answer 5

问题解决没有，我出现了个你这个一模一样的error