用scrapy爬取图片时没有下载图片


import scrapy
from scrapy.pipelines.images import ImagesPipeline
class ImgzPipeline(ImagesPipeline):

    def get_media_requests(self, item, info):
        print(item['src'])
        yield scrapy.Request(item['src'])

    def file_path(self, request, response=None, info=None, *, item=None):
        imgName = response.url.split('/')[-1]
        print(imgName)
        return imgName

    def item_completed(self, results, item, info):
        for success, image_info in results:
            if success:
                print("Image downloaded:", image_info['path'])
            else:
                print("Image download failed for:", item['src'])
        return item

上面是pipelines.py的代码，运行后能正确输出item['src']，但是图片下载失败，因此没法执行file_path中的print(imgName)，运行结果如图片所示：

在网上搜索后发现很多都说是pillow的问题，但是我pip install pillow后依然不行，setting s.py里也设置了UA，ROBOTSTXT_OBEY也设置成了False，开启了ITEM_PLPELINES，也加了IMAGES_STORE，如下：


USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

ITEM_PIPELINES = {
   "imgz.pipelines.ImgzPipeline": 300,
}

REQUEST_FINGERPRINTER_IMPLEMENTATION = "2.7"
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
FEED_EXPORT_ENCODING = "utf-8"

#指定图片存储的目录
IMAGES_STORE = './imgs_zhanzhang'

下面是完整的日志信息（注释掉了print(item['src'])) ：


(venv) PS C:\python_learning\imgz> scrapy crawl img
2024-02-13 22:48:17 [scrapy.utils.log] INFO: Scrapy 2.11.0 started (bot: imgz)
2024-02-13 22:48:17 [scrapy.utils.log] INFO: Versions: lxml 5.1.0.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)], pyOpenSSL 24.0.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.2, Platform Windows-10-10.0.22621-SP0
2024-02-13 22:48:17 [scrapy.addons] INFO: Enabled addons:
[]
2024-02-13 22:48:17 [asyncio] DEBUG: Using selector: SelectSelector
2024-02-13 22:48:17 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2024-02-13 22:48:17 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
2024-02-13 22:48:17 [scrapy.extensions.telnet] INFO: Telnet Password: 3ca477838cf3482a
2024-02-13 22:48:17 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2024-02-13 22:48:17 [scrapy.crawler] INFO: Overridden settings:

{'BOT_NAME': 'imgz',
 'FEED_EXPORT_ENCODING': 'utf-8',
 'NEWSPIDER_MODULE': 'imgz.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['imgz.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
               '(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'}
2024-02-13 22:48:17 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2024-02-13 22:48:17 [scrapy.middleware] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-02-13 22:48:17 [scrapy.middleware] INFO: Enabled item pipelines:
['imgz.pipelines.ImgzPipeline']
2024-02-13 22:48:17 [scrapy.core.engine] INFO: Spider opened
2024-02-13 22:48:17 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-02-13 22:48:17 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-02-13 22:48:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://sc.chinaz.com/tupian/> (referer: None)
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/171e17468686ac0d_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/171e17468686ac0d_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/96c8d6f20a80b0e3_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/96c8d6f20a80b0e3_s.png'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-17/4829e81e16d4ad17_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-17/4829e81e16d4ad17_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/2b0cdfaea894bdb2_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/2b0cdfaea894bdb2_s.jpg'}

Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/0abfcdfe8dea7709_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/0abfcdfe8dea7709_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/4bbf45f797921a6b_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/4bbf45f797921a6b_s.png'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/e759eb4fe9f55311_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/e759eb4fe9f55311_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/bf1a8829c8e46c8e_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/bf1a8829c8e46c8e_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/69e7d373cd036728_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/69e7d373cd036728_s.jpg'}

Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/b25551f582e39c0f_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/b25551f582e39c0f_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/352f231647443f9b_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/352f231647443f9b_s.png'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-12/62fde35402581833_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-12/62fde35402581833_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/a2adaaf96eae9918_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/a2adaaf96eae9918_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/cc111f53ae9df480_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/cc111f53ae9df480_s.jpg'}

Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-06/2c4348afef4bffc1_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-06/2c4348afef4bffc1_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/ec2bcbc7ddd08e34_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/ec2bcbc7ddd08e34_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/680bc14fe7d0e2c0_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/680bc14fe7d0e2c0_s.png'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-17/c40fe0af00b3e0a7_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-17/c40fe0af00b3e0a7_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/960b59d5f8417dd5_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/960b59d5f8417dd5_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-17/fb7453d5ae3d19a6_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-17/fb7453d5ae3d19a6_s.jpg'}

Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/cc1e6b69b8937f9d_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/cc1e6b69b8937f9d_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/06a13e89dc4695be_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/06a13e89dc4695be_s.png'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-06/770e5109547df247_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-06/770e5109547df247_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/db4e65397ce87059_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/db4e65397ce87059_s.png'}

Image download failed for: https://scpic.chinaz.net/files/default/imgs/2023-12-26/39487e6801d9ce0f_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2023-12-26/39487e6801d9ce0f_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/c505514549ee54b2_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/c505514549ee54b2_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-19/ae0a072994903dbf_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-19/ae0a072994903dbf_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/8018846398fd445b_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/8018846398fd445b_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/c5144f1859fc84b2_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/c5144f1859fc84b2_s.jpg'}

Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/a3319b85225d1bec_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/a3319b85225d1bec_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/10b8de853a76c1c7_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/10b8de853a76c1c7_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/59a7ae0b919c519a_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/59a7ae0b919c519a_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-28/edd09697c94f3550_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-28/edd09697c94f3550_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/81608a517e857c05_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/81608a517e857c05_s.png'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/ada2003f30c81a03_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/ada2003f30c81a03_s.jpg'}

Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-19/6f61bb57f8a8e451_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-19/6f61bb57f8a8e451_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/0c40608d7490934b_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/0c40608d7490934b_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-01-12/e07ee6957e394768_s.jpg
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-01-12/e07ee6957e394768_s.jpg'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-06/7e7e72de9b39765a_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-06/7e7e72de9b39765a_s.png'}
Image download failed for: https://scpic.chinaz.net/files/default/imgs/2024-02-04/9394e09acf35cb19_s.png
2024-02-13 22:48:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://sc.chinaz.com/tupian/>
{'src': 'https://scpic.chinaz.net/files/default/imgs/2024-02-04/9394e09acf35cb19_s.png'}

2024-02-13 22:48:18 [scrapy.core.engine] INFO: Closing spider (finished)
2024-02-13 22:48:18 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 297,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 10467,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 0.545763,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2024, 2, 13, 14, 48, 18, 29026, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 53416,
 'httpcompression/response_count': 1,
 'item_scraped_count': 40,
 'log_count/DEBUG': 44,
 'log_count/INFO': 10,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2024, 2, 13, 14, 48, 17, 483263, tzinfo=datetime.timezone.utc)}
2024-02-13 22:48:18 [scrapy.core.engine] INFO: Spider closed (finished)

找了很长时间的错了但是没找到，请各位帮忙看看

写回答
好问题 0 提建议
追加酬金
关注问题
分享
邀请回答
编辑收藏删除
收藏举报

5条回答默认最新

关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
专家-郭老师 Java领域新星创作者 2024-02-13 23:00
关注
参考通义千问和郭老师的小迷弟雅思莫了-编写提供，主要是为了帮助您快速解决问题，如下回答，您可以参考下，如有问题，评论区留言。
根据您提供的信息，Scrapy 确实成功爬取了图片链接并发送了下载请求，但是图片下载失败。从日志中可以看到“Image download failed for: ...”的提示。

有几个可能的原因：

网络问题：请检查图片URL是否可以正常访问。有时候网站会设置防盗链或针对非浏览器客户端限制访问资源。

中间件干扰：确认您的 Scrapy 项目中是否有自定义的中间件阻止了图片下载。

存储路径权限问题：确保 IMAGES_STORE 指定的目录存在并且当前用户有足够的权限写入文件。

图片服务器 IP 被封禁：如果频繁快速地请求同一站点的资源，可能会被目标服务器识别为恶意请求并封禁 IP。

SSL/TLS 配置问题：如果您在下载 HTTPS 图片时遇到问题，请检查 SSL 设置。

建议您首先尝试手动访问图片URL以验证链接的有效性，并在 pipelines.py 中添加更多调试信息，例如在 file_path 方法和 item_completed 方法内都加入打印语句，以便更清楚地了解图片下载过程中的状态。

另外，查看Scrapy的日志级别，有时错误信息会被记录在 WARNING 或 ERROR 级别的日志中，您可以将日志级别调低以查看更多详细信息。
本回答被题主选为最佳回答 , 对您是否有帮助呢?

解决无用
评论打赏
分享
举报

评论

按下Enter换行，Ctrl+Enter发表内容

查看更多回答(4条)

报告相同问题？

关注问题

用scrapy爬取站长素材无法下载图片 python 爬虫
2021-08-03 18:51

回答 2 已采纳找到原因了，是要在setting中加上MEDIA_ALLOW_REDIRECTS = True，貌似是中间件的内容，我还没学到，所以不清楚什么意思，有大佬可以解释一下吗看所有日志后会发现其实有地方报错
scrapy 爬取图片时图片的url总是显示None python 有问必答
2021-12-04 01:41

回答 1 已采纳 img_url = div.xpath('./div/a/img/src').extract_first() src前面少了 @ 改成 img_url = div.xpath('./
scrapy爬取图片，爬取不到 python 有问必答
2021-05-23 20:32

回答 2 已采纳你已经爬到图片连接了，这个看到的管道文件的代码怎样写，要对图片链接发送请求访问，然后保存才行
如何利用Scrapy爬取知名技术文章网站！干货真多！
2020-09-22 14:17

爬遍天下无敌手的博客 Scrapy是一个非常优秀的爬虫框架，通过Scrapy框架，可以非常轻松地实现强大的爬虫系统，我们只需要将精力放在抓取规则以及如何处理抓取的数据上即可，本文通过实战来介绍Scrapy的入门知识以及一些高级应用。...
scrapy 爬取图片 报错 error processing python
2021-12-20 01:08

回答 1 已采纳 http: 去那里啦?
scrapy爬取百度图片时Forbid spider access python 有问必答
2021-06-13 23:12

回答 2 已采纳这个是百度反爬虫导致的，
如何利用scrapy爬取带标签的网页内容并保存到自己的服务器上？ mysql python sql
2018-02-09 09:34

回答 3 已采纳 1. 把整个爬取到的网页内容直接存储到数据库肯定是可以的，你之所以没有成功，应该是因为你的数据库中的相应字段错了，整个网页内容都比较长，一般都是要用text字段，甚至是LongText)（最大长度42
手把手教你使用Python+scrapy爬取山东各城市天气预报
2018-05-29 13:02

dongfuguo的博客 1、在命令提示符环境使用pip install scrapy命令安装Python扩展库scrapy，详见Python使用Scrapy爬虫框架爬取天涯社区小说“大宗师”全文2、使用下图中的命...
scrapy爬取知乎首页乱码
2017-12-01 03:21

回答 2 已采纳 ```python HEADERS = { 'Host': 'www.zhihu.com', 'Accept': 'text/html,application/xhtml+xml
scrapy 爬取商品视频url,详情失败 python
2022-02-17 10:28

回答 3 已采纳 js加载的，先打印看下获取的整个网页。我试了下，在下图这个位置可以找到视频链接，可以用正则或者别的方法取出来
scrapy-爬取京东笔记本电脑信息问题 chrome python selenium 开发语言
2020-09-01 19:12

回答 2 已采纳 ``` browser.quit() return HtmlResponse(url=request.url, body=browser.page_source, re
python Scrapy爬取天气预报，零基础的你也可以快速上手
2021-07-12 18:03

Java进阶营菌的博客写一个真正意义上一个爬虫，并将他爬取到的数据分别保存到txt、json、已经存在的mysql数据库中。目标分析： [图片上传失败…(image-cf5522-1626084108588)] 初学者有什么不懂的可以私信我——我刚整理了一套2018...
在以瀑布流方式翻页的网站,使用scrapy网络爬虫,但是只爬取了第一页数据,没有爬取第二页. python 爬虫
2021-09-05 19:18

回答 2 已采纳那叫ajax，
Python3网络爬虫开发实战，Scrapy 爬取新浪微博
2020-08-11 16:09

Python新世界的博客前面讲解了 Scrapy 中各个模块基本使用方法以及代理池、Cookies 池。接下来我们以一个反爬比较强的网站新浪微博为例，来实现一下 Scrapy 的大规模爬取。很多人学习python，不知道从何学起。很多人学习python，...
「Python爬虫系列讲解」十三、用 Scrapy 技术爬取网络数据
2020-07-08 00:06

荣仔！最靓的仔！的博客本专栏是以杨秀璋老师爬虫著作《Python网络数据爬取及分析「从入门到精通」》为主线、个人学习理解为主要内容，以学习笔记形式编写的。本专栏不光是自己的一个学习分享，也希望能给您普及一些关于爬虫的相关知识...
没有解决我的问题, 去提问

问题事件

关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
系统已结题 3月5日
关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
已采纳回答 2月26日
关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
创建了问题 2月13日

悬赏问题

¥15 python的qt5界面
¥15 无线电能传输系统MATLAB仿真问题
¥50 如何用脚本实现输入法的热键设置
¥20 我想使用一些网络协议或者部分协议也行，主要想实现类似于traceroute的一定步长内的路由拓扑功能
¥30 深度学习，前后端连接
¥15 孟德尔随机化结果不一致
¥15 apm2.8飞控罗盘bad health，加速度计校准失败
¥15 求解O-S方程的特征值问题给出边界层布拉休斯平行流的中性曲线
¥15 谁有desed数据集呀
¥20 手写数字识别运行c仿真时，程序报错错误代码sim211-100

用scrapy爬取图片时没有下载图片

5条回答 默认 最新

问题事件

悬赏问题

5条回答默认最新