sinat_32092535 2018-07-17 09:57 采纳率: 0%
浏览 2686
已结题

scrapy-redis报错,这个真不知道什么原因,我之前写的另外一个爬虫是可以执行的

Traceback (most recent call last):
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
GeneratorExit
Exception ignored in:
RuntimeError: generator ignored GeneratorExit
Unhandled error in Deferred:
2018-07-17 17:50:00 [twisted] CRITICAL: Unhandled error in Deferred:

2018-07-17 17:50:00 [twisted] CRITICAL:
Traceback (most recent call last):
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\twisted\internet\task.py", line 517, in _oneWorkUnit
result = next(self._iterator)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\utils\defer.py", line 63, in
work = (callable(elem, *args, **named) for elem in iterable)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\core\scraper.py", line 183, in _process_spidermw_output
self.crawler.engine.crawl(request=output, spider=spider)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\core\engine.py", line 210, in crawl
self.schedule(request, spider)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy\core\engine.py", line 216, in schedule
if not self.slot.scheduler.enqueue_request(request):
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\scheduler.py", line 167, in enqueue_request
self.queue.push(request)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\queue.py", line 99, in push
data = self._encode_request(request)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\queue.py", line 43, in _encode_request
return self.serializer.dumps(obj)
File "C:\Users\xin\Desktop\spider_thief\venv\lib\site-packages\scrapy_redis\picklecompat.py", line 14, in dumps
return pickle.dumps(obj, protocol=-1)
RecursionError: maximum recursion depth exceeded while calling a Python object

  • 写回答

2条回答 默认 最新

  • oyljerry 2018-07-17 10:28
    关注

    maximum recursion depth exceeded

    你的递归调用层数太多了,超过了python预定的限制,所以异常了。
    你要设置一下你的爬虫递归调用的层数,超过一定层数就返回,不要继续递归了

    评论

报告相同问题?

悬赏问题

  • ¥15 jupyterthemes 设置完毕后没有效果
  • ¥15 matlab图像高斯低通滤波
  • ¥15 针对曲面部件的制孔路径规划,大家有什么思路吗
  • ¥15 钢筋实图交点识别,机器视觉代码
  • ¥15 如何在Linux系统中,但是在window系统上idea里面可以正常运行?(相关搜索:jar包)
  • ¥50 400g qsfp 光模块iphy方案
  • ¥15 两块ADC0804用proteus仿真时,出现异常
  • ¥15 关于风控系统,如何去选择
  • ¥15 这款软件是什么?需要能满足我的需求
  • ¥15 SpringSecurityOauth2登陆前后request不一致