Python爬取招股说明书PDF，运行成功但是下载文件失败，是有什么问题吗

Question

scrapy crawl letter运行下面的代码，但是总是能运行但是不能下载文件。

import scrapy,json,requests,os
import pandas as pd
from numpy import random
from datetime import datetime
from report_downloader.items import ReportDownloaderItem

class LetterSpider(scrapy.Spider):
    name = 'letter'
    allowed_domains = ['eastmoney.com']

    def start_requests(self):
        # 读取股票列表
        index_path = os.path.join(os.path.dirname(os.path.dirname(__file__)),'股票代码.xlsx')
        df = pd.read_excel(index_path,dtype={'代码':str})
        code_list = df['代码']
        for code in code_list:
            # 设置api网址
            baseurl = 'https://np-anotice-stock.eastmoney.com/api/security/ann?'
            # 获取当前时间戳
            ts = int(datetime.now().timestamp()*1000)
            cb = 'jQuery11230{}{}_{}'.format(random.randint(100000000,499999999),random.randint(10000000,99999999),ts)
            # 设置请求表单
            formdata = {
                'cb':cb,
                'sr':'-1',
                'page_size':'50',
                'page_index':1,
                'ann_type':'A',
                'client_source':'web',
                'stock_list':code,
                'f_node':'0',
                's_node':'0',
            }
            # 发起一次请求，获取总页数
            url = '{}{}'.format(baseurl,'&'.join(['{}={}'.format(x[0],x[1]) for x in formdata.items()]))
            r = requests.get(url,data=formdata)
            r = json.loads(r.text.replace('{}('.format(cb),'').replace(')',''))
            finalpage = r['data']['total_hits']//50

            # 只遍历最后三页，每页查看50条文件列表
            for page in range(finalpage-2,finalpage+2):
                # 重设时间戳
                ts = int(datetime.now().timestamp()*1000)
                cb = 'jQuery11230{}{}_{}'.format(random.randint(100000000,499999999),random.randint(10000000,99999999),ts)
                formdata = {
                    'cb':cb,
                    'sr':'-1',
                    'page_size':'50',
                    'page_index':page,
                    'ann_type':'A',
                    'client_source':'web',
                    'stock_list': code,
                    'f_node':'0',
                    's_node':'0',
                }
                # 发起请求
                url = '{}{}'.format(baseurl,'&'.join(['{}={}'.format(x[0],x[1]) for x in formdata.items()]))
                yield scrapy.Request(url,meta={'cb':cb},callback=self.get_list,dont_filter=True)


    def get_list(self, response):
        li = json.loads(response.text.replace('{}('.format(response.meta['cb']),'').replace(')',''))['data']['list']
        for one in li:
            name = one['title'].replace('\\',' ').replace('/',' ')
            # 回复意见
            # if ('发行人' in one['title']) and ('回复意见' in one['title']) and ('落实' not in one['title']):
            #     title = '{}_{}_{}.pdf'.format(one['codes'][0]['stock_code'],one['codes'][0]['short_name'],name)
            #     durl = 'https://pdf.dfcfw.com/pdf/H2_{}_1.pdf'.format(one['art_code'])
            #     item = ReportDownloaderItem(title=title,durl=durl)
            #     yield item

            # 招股说明书
            if ('招股说明书' in one['title']) and ('首次公开' in one['title'])and ('摘要' not in one['title']) and ('上会' not in one['title'])and ('申报' not in one['title'])and ('注册' not in one['title']):
                title = '{}_{}_{}.pdf'.format(one['codes'][0]['stock_code'],one['codes'][0]['short_name'],name)
                durl = 'https://pdf.dfcfw.com/pdf/H2_{}_1.pdf'.format(one['art_code'])
                item = ReportDownloaderItem(title=title,durl=durl)
                yield item

这就是运行的结果，各位给看看

PS F:\IPO_crawler-main\report_downloader> scrapy crawl letter
2024-07-14 19:51:35 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: report_downloader)
2024-07-14 19:51:36 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.11.7, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.1, Twisted 24.3.0, Python 3.12.3 | packaged by conda-forge | (main, 
Apr 15 2024, 18:20:11) [MSC v.1938 64 bit (AMD64)], pyOpenSSL 24.1.0 (OpenSSL 3.2.2 4 Jun 2024), 
cryptography 42.0.8, Platform Windows-10-10.0.19045-SP0
2024-07-14 19:51:36 [scrapy.addons] INFO: Enabled addons:
[]
2024-07-14 19:51:36 [py.warnings] WARNING: D:\anaconda\envs\fraud_env\Lib\site-packages\scrapy\utils\request.py:254: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.

It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.

See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
  return cls(crawler)

2024-07-14 19:51:36 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2024-07-14 19:51:36 [scrapy.extensions.telnet] INFO: Telnet Password: 77d3334bb7b13664
2024-07-14 19:51:36 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2024-07-14 19:51:36 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'report_downloader',
 'NEWSPIDER_MODULE': 'report_downloader.spiders',
 'SPIDER_MODULES': ['report_downloader.spiders']}
2024-07-14 19:51:36 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2024-07-14 19:51:36 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-07-14 19:51:36 [py.warnings] WARNING: D:\anaconda\envs\fraud_env\Lib\site-packages\scrapy\pipelines\media.py:148: ScrapyDeprecationWarning: file_path(self, request, response=None, info=None) is deprecated, please use file_path(self, request, response=None, info=None, *, item=None)     
  self._check_signature(func)

2024-07-14 19:51:36 [scrapy.middleware] INFO: Enabled item pipelines:
['report_downloader.pipelines.ReportDownloaderPipeline']
2024-07-14 19:51:36 [scrapy.core.engine] INFO: Spider opened
2024-07-14 19:51:36 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 
0 items (at 0 items/min)
2024-07-14 19:51:36 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-07-14 19:51:37 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): np-anotice-stock.eastmoney.com:443
2024-07-14 19:51:37 [urllib3.connectionpool] DEBUG: https://np-anotice-stock.eastmoney.com:443 "GET /api/security/ann?cb=jQuery1123048249952422348865_1720957897289&sr=-1&page_size=50&page_index=1&ann_type=A&client_source=web&stock_list=001308&f_node=0&s_node=0 HTTP/1.1" 200 34582
2024-07-14 19:51:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://np-anotice-stock.eastmoney.com/api/security/ann?cb=jQuery1123021207829864337257_1720957897621&sr=-1&page_size=50&page_index=8&ann_type=A&client_source=web&stock_list=001308&f_node=0&s_node=0> (referer: None)
2024-07-14 19:51:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://np-anotice-stock.eastmoney.com/api/security/ann?cb=jQuery1123038682287218392616_1720957897614&sr=-1&page_size=50&page_index=7&ann_type=A&client_source=web&stock_list=001308&f_node=0&s_node=0> (referer: None)
2024-07-14 19:51:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://np-anotice-stock.eastmoney.com/api/security/ann?cb=jQuery1123034284541950785597_1720957897625&sr=-1&page_size=50&page_index=10&ann_type=A&client_source=web&stock_list=001308&f_node=0&s_node=0> (referer: None)        
2024-07-14 19:51:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://np-anotice-stock.eastmoney.com/api/security/ann?cb=jQuery1123040443715152602585_1720957897623&sr=-1&page_size=50&page_index=9&ann_type=A&client_source=web&stock_list=001308&f_node=0&s_node=0> (referer: None)
2024-07-14 19:51:38 [scrapy.downloadermiddlewares.offsite] DEBUG: Filtered offsite request to 'pdf.dfcfw.com': <GET https://pdf.dfcfw.com/pdf/H2_AN202203061551011071_1.pdf>
2024-07-14 19:51:38 [scrapy.pipelines.media] ERROR: [Failure instance: Traceback: <class 'scrapy.pipelines.files.FileException'>:
D:\anaconda\envs\fraud_env\Lib\site-packages\twisted\internet\defer.py:536:addCallbacks
D:\anaconda\envs\fraud_env\Lib\site-packages\twisted\internet\defer.py:1078:_runCallbacks        
D:\anaconda\envs\fraud_env\Lib\site-packages\scrapy\pipelines\media.py:197:_check_media_to_download
D:\anaconda\envs\fraud_env\Lib\site-packages\twisted\internet\defer.py:536:addCallbacks
--- <exception caught here> ---
D:\anaconda\envs\fraud_env\Lib\site-packages\twisted\internet\defer.py:1078:_runCallbacks        
D:\anaconda\envs\fraud_env\Lib\site-packages\scrapy\pipelines\files.py:459:media_failed
]
NoneType: None
2024-07-14 19:51:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://np-anotice-stock.eastmoney.com/api/security/ann?cb=jQuery1123040443715152602585_1720957897623&sr=-1&page_size=50&page_index=9&ann_type=A&client_source=web&stock_list=001308&f_node=0&s_node=0>
{'durl': 'https://pdf.dfcfw.com/pdf/H2_AN202203061551011071_1.pdf',
 'title': '001308_康冠科技_康冠科技:首次公开发行股票招股说明书.pdf'}
2024-07-14 19:51:38 [scrapy.core.engine] INFO: Closing spider (finished)
2024-07-14 19:51:38 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
 'downloader/exception_type_count/scrapy.exceptions.IgnoreRequest': 1,
 'downloader/request_bytes': 1541,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 4,
 'downloader/response_bytes': 108593,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 4,
 'elapsed_time_seconds': 1.250309,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2024, 7, 14, 11, 51, 38, 199251, tzinfo=datetime.timezone.utc), 'item_scraped_count': 1,
 'log_count/DEBUG': 9,
 'log_count/ERROR': 1,
 'log_count/INFO': 10,
 'log_count/WARNING': 2,
 'offsite/domains': 1,
 'offsite/filtered': 1,
 'response_received_count': 4,
 'scheduler/dequeued': 4,
 'scheduler/dequeued/memory': 4,
 'scheduler/enqueued': 4,
 'scheduler/enqueued/memory': 4,
 'start_time': datetime.datetime(2024, 7, 14, 11, 51, 36, 948942, tzinfo=datetime.timezone.utc)} 
2024-07-14 19:51:38 [scrapy.core.engine] INFO: Spider closed (finished)

Answer 1

以下内容由CHATGPT及阿里嘎多学长共同生成、有用望采纳：

---

根据您提供的日志和代码，以下是一些可能的原因和解决方案：

1. FileException错误：这个错误通常表示文件下载过程中出现了问题。根据日志，错误发生在scrapy.pipelines.media模块中，但没有具体的错误信息。您可能需要在ReportDownloaderPipeline中添加更多的错误处理和日志记录，以确定问题所在。

2. 检查Pipeline配置：确保ReportDownloaderPipeline类中的方法如process_item和media_to_download正确实现，并且media_downloaded方法能够正确处理下载的文件。

3. 检查文件下载逻辑：在get_list方法中生成的下载链接durl需要确保是正确的。您可以打印出来检查是否与实际的PDF文件URL匹配。

4. 检查文件存储路径：确保您的项目目录中有一个可以写入的文件夹，用于存储下载的PDF文件。在ReportDownloaderPipeline中设置正确的文件路径。

5. 检查请求头和Cookies：如果服务器需要特定的请求头或Cookies来允许下载，您需要在请求中包含这些信息。这可能需要您分析正常的浏览器请求来获取必要的头信息。

6. 检查重试机制：在RetryMiddleware中设置合适的重试规则，例如最大重试次数和重试间隔。

7. 检查Scrapy设置：确保您的settings.py文件中与下载相关的设置是合理的，比如CONCURRENT_REQUESTS_PER_DOMAIN和DOWNLOAD_DELAY。

8. 检查网络问题：如果网络不稳定，可能会导致下载失败。确保您的网络连接是稳定的。

9. 检查文件大小限制：如果PDF文件很大，可能需要调整FILES_STORE_SPOOL_DIR和FILES_STORE_FULL_PATH等设置，以确保有足够的空间存储下载的文件。

10. 日志记录：增加更多的日志记录，特别是在ReportDownloaderPipeline中，这样可以帮助您更好地追踪问题。

此外，您的代码中有一些未使用的导入，如from numpy import random和from datetime import datetime，这些可以被移除以避免混淆。

最后，您可能需要检查ReportDownloaderItem的定义，确保它包含了所有必要的字段，并且这些字段在Pipeline中被正确处理。

如果问题仍然存在，您可能需要逐步调试Pipeline中的代码，或者在Pipeline中添加更多的错误处理来捕获并记录异常。