python爬虫爬小说时，爬了一百多张就会报错，从新启动后爬取速度变慢了

Question

python爬虫爬小说时，爬了一百多张就会报错，从新启动后爬取速度变慢了

以下是全部代码

import requests
from lxml import etree
import time
from random import randint as rd

bookpath = r"books/"
bookname = "超级作死宝箱系统3"
bookurl = "http://www.777zw.net/30/30031/"
weburl = "http://www.777zw.net/30/30031/"
user_agent= "Mozilla/5.0 (Windows NT 5.1; WOW64) AppleWebKit/561.3 (KHTML, like Gecko) Chrome/21.0.1262.89 Safari/561.3"

headers = {
"Referer": bookurl,
"User-Agent": user_agent,
'Connection': 'close'
}

requests.DEFAULT_RETRIES = 20
s = requests.session()
s.keep_alive = False
def get_urls():
url = bookurl
response = requests.get(url, headers=headers, verify=False)
response.encoding = 'utf-8'
html = etree.HTML(response.text)

# 所有章节的url列表

url_list = [weburl + x for x in html.xpath('//li[@class="book-item"]/a/@href')]

print(f'{len(url_list)}')

    return url_list

def get_text(url):
rep = requests.get(url, headers=headers, verify=False)
rep.encoding = 'utf-8'
dom = etree.HTML(rep.text)
# 获取章节名weixin_38616018
name = dom.xpath('//h1[@class="title"]/text()')[0]
#获取章节内小说
text = dom.xpath('//div[@id="content"][@class="content"]/p/text()')
#写入txt文档
with open(bookpath + bookname + '.txt', mode='a', encoding='utf-8') as f:
f.write('\n'+ name + '\n')
for con in text:
f.write(' ' + con + '\n')
print(f'{name} ===> 下载完成')

def main():
urls = get_urls()
for url in urls:
get_text(url)
# 随机时间
c = rd(1000, 3000) / 1000
print(c)
time.sleep(c)

if name == 'main':
main()

基本会报错Max retries exceeded with url，有多次直接停止不动（不报错），加了timeout = 60 后就会报时间过长

尝试过random请求头，效果并不理想。每次重新启动有时能跑起来，有时直接停止不动。

我想要能一次性下载完

Answer 1

可能是访问次数频繁，被禁止访问，解决方法：使用代理

python 关于Max retries exceeded with url 的错误_Lucas__liu的博客-CSDN博客 在爬取boss直聘时出现这种错误，于是搜索了网上很多文章，总结如下： 1.http连接太多没有关闭导致的，解决方法：import requestsrequests.adapters.DEFAULT_RETRIES = 5 # 增加重连次数s = requests.session()s.keep_alive = False # 关闭多余连接s.get(url) # 你需要的网址2.... https://blog.csdn.net/lb245557472/article/details/80239603