请问下Scrapy加代理后显示got NoneType的错误

Question

先放下报错信息

# 网站隐藏的以下.请求的url肯定是没出问题

2021-03-03 00:31:22 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.xxx.com/>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 45, in process_request
    return (yield download_func(request=request, spider=spider))
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/utils/defer.py", line 55, in mustbe_deferred
    result = f(*args, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/__init__.py", line 75, in download_request
    return handler.download_request(request, spider)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/http11.py", line 88, in download_request
    return agent.download_request(request)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/http11.py", line 342, in download_request
    agent = self._get_agent(request, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/handlers/http11.py", line 301, in _get_agent
    _, _, proxyHost, proxyPort, proxyParams = _parse(proxy)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/webclient.py", line 36, in _parse
    return _parsed_url_args(parsed)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/core/downloader/webclient.py", line 19, in _parsed_url_args
    host = to_bytes(parsed.hostname, encoding="ascii")
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/utils/python.py", line 106, in to_bytes
    raise TypeError('to_bytes must receive a str or bytes '
TypeError: to_bytes must receive a str or bytes object, got NoneType
(base) licongjian@licongjiandeMacBook-Pro jingdongPro $ 

 

手动添加代理的时候可以获取到数据.后来在redis的集合中拿到代理以后就出现了这个问题

def process_request(self, request, spider):
    proxy = str(self.redis_db.srandmember('proxy')).replace('b', '')
    request.meta['proxy'] = proxy

此为打印的代理信息与request.meta
{'download_timeout': 3.0, 'proxy': "'https://116.115.210.140:4326'"}
'https://116.115.210.140:4326' <class 'str'>

Answer 1

这是取值问题. redis默认返回的值是bytes类型的.
解决方法: 连接redis时, 添加参数decode_responses=True
例子如下:
redis = redis.Redis(host=RedisConfig.redis_host, port=RedisConfig.redis_port, db=0, decode_responses=True)