请问下Scrapy加代理后显示got NoneType的错误
先放下报错信息
# 网站隐藏的以下.请求的url肯定是没出问题
2021-03-03 00:31:22 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.xxx.com/>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 45, in process_request
return (yield download_func(request=request, spider=spider))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/utils/defer.py", line 55, in mustbe_deferred
result = f(*args, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/__init__.py", line 75, in download_request
return handler.download_request(request, spider)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/http11.py", line 88, in download_request
return agent.download_request(request)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/http11.py", line 342, in download_request
agent = self._get_agent(request, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/http11.py", line 301, in _get_agent
_, _, proxyHost, proxyPort, proxyParams = _parse(proxy)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/webclient.py", line 36, in _parse
return _parsed_url_args(parsed)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/webclient.py", line 19, in _parsed_url_args
host = to_bytes(parsed.hostname, encoding="ascii")
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/utils/python.py", line 106, in to_bytes
raise TypeError('to_bytes must receive a str or bytes '
TypeError: to_bytes must receive a str or bytes object, got NoneType
(base) licongjian@licongjiandeMacBook-Pro jingdongPro $
手动添加代理的时候可以获取到数据.后来在redis的集合中拿到代理以后就出现了这个问题
def process_request(self, request, spider): proxy = str(self.redis_db.srandmember('proxy')).replace('b', '') request.meta['proxy'] = proxy 此为打印的代理信息与request.meta {'download_timeout': 3.0, 'proxy': "'https://116.115.210.140:4326'"} 'https://116.115.210.140:4326' <class 'str'>
- 点赞
- 写回答
- 关注问题
- 收藏
- 复制链接分享
- 邀请回答
为你推荐
- 为什么我的scrapy爬不到数据了
- python
- 1个回答
- scrapy + selenium 抓取不到完整的网易云页面
- 【scrapy爬虫问题】scrapy.xpath解析返回的结果有问题,求解答!!!
- python
- 1个回答
- scrapy 爬虫大量链接返回None不知道为啥
- python
- 2个回答
- 关于Scrapy 框架运行不出结果的问题,好像没有报错
- scrapy框架+formdata+ajax爬取及翻页问题
- scrapy存到mysql查询无数据
- 什么相当于SCRAPY中的CURL
- php
- scrapy
- 1个回答
- 来个大佬教下小白scrapy怎么创建多个spider
- python
- 1个回答
- 如何在scrapy中带cookie访问?
- scrapy爬虫出现 DEBUG: Crawled (404)
- python
- 1个回答
- 爬虫之scrapy报错spider 农田
- scrapy爬虫相关 关于json数据的处理
- xpath
- json
- 爬虫
- 数据
- 1个回答
- 如何利用scrapy爬取带标签的网页内容并保存到自己的服务器上?
- flask+scrapy的爬虫问题
- 运行scrapy项目报错:ImportError:DLL load failed:操作系统无法运行
- scrapy 爬虫
- 3个回答
- scrapy爬取知乎首页乱码
- 乱码
- scrapy 爬虫
- 2个回答
- 在python新建虚拟环境安装scrapy ,安装不成功,请问是怎么回事
- python
- 1个回答
- 建立scrapy工程,显示这样的错误,没找到解决方法
- python
- 命令行
- scrapy
- 1个回答
- scrapy优点是什么?比直接用正则好在哪?
- python
- scrapy
- 爬虫
- 1个回答