scrapy出错twisted.python.failure.Failure twisted.internet.error

我是跟这网上视频写的

import scrapy
class QsbkSpider(scrapy.Spider):
    name = 'qsbk'
    allowed_domains = ['www.qiushibaike.com/']
    start_urls = ['https://www.qiushibaike.com/text/']

    def parse(self, response):
        print('='*10)
        print(response)
        print('*'*10)

出现
图片说明

请求头改了还是不行,用requests库爬取又可以

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
python scrapy 爬虫图片新手求助
求问大神 我这个data她怎么了 报错: 2020-02-07 09:24:55 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: meizitu) 2020-02-07 09:24:55 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Windows-10-10.0.17763-SP0 2020-02-07 09:24:55 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'meizitu', 'NEWSPIDER_MODULE': 'meizitu.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['meizitu.spiders']} 2020-02-07 09:24:55 [scrapy.extensions.telnet] INFO: Telnet Password: 0936097982b9bcc8 2020-02-07 09:24:55 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] 2020-02-07 09:24:56 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-02-07 09:24:56 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] Unhandled error in Deferred: 2020-02-07 09:24:56 [twisted] CRITICAL: Unhandled error in Deferred: Traceback (most recent call last): File "e:\python3.7\lib\site-packages\scrapy\crawler.py", line 184, in crawl return self._crawl(crawler, *args, **kwargs) File "e:\python3.7\lib\site-packages\scrapy\crawler.py", line 188, in _crawl d = crawler.crawl(*args, **kwargs) File "e:\python3.7\lib\site-packages\twisted\internet\defer.py", line 1613, in unwindGenerator return _cancellableInlineCallbacks(gen) File "e:\python3.7\lib\site-packages\twisted\internet\defer.py", line 1529, in _cancellableInlineCallbacks _inlineCallbacks(None, g, status) --- <exception caught here> --- File "e:\python3.7\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "e:\python3.7\lib\site-packages\scrapy\crawler.py", line 86, in crawl self.engine = self._create_engine() File "e:\python3.7\lib\site-packages\scrapy\crawler.py", line 111, in _create_engine return ExecutionEngine(self, lambda _: self.stop()) File "e:\python3.7\lib\site-packages\scrapy\core\engine.py", line 70, in __init__ self.scraper = Scraper(crawler) File "e:\python3.7\lib\site-packages\scrapy\core\scraper.py", line 71, in __init__ self.itemproc = itemproc_cls.from_crawler(crawler) File "e:\python3.7\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler return cls.from_settings(crawler.settings, crawler) File "e:\python3.7\lib\site-packages\scrapy\middleware.py", line 34, in from_settings mwcls = load_object(clspath) File "e:\python3.7\lib\site-packages\scrapy\utils\misc.py", line 46, in load_object mod = import_module(module) File "e:\python3.7\lib\importlib\__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 724, in exec_module File "<frozen importlib._bootstrap_external>", line 860, in get_code File "<frozen importlib._bootstrap_external>", line 791, in source_to_code File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed builtins.SyntaxError: unexpected EOF while parsing (pipelines.py, line 22) 2020-02-07 09:24:56 [twisted] CRITICAL: Traceback (most recent call last): File "e:\python3.7\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "e:\python3.7\lib\site-packages\scrapy\crawler.py", line 86, in crawl self.engine = self._create_engine() File "e:\python3.7\lib\site-packages\scrapy\crawler.py", line 111, in _create_engine return ExecutionEngine(self, lambda _: self.stop()) File "e:\python3.7\lib\site-packages\scrapy\core\engine.py", line 70, in __init__ self.scraper = Scraper(crawler) File "e:\python3.7\lib\site-packages\scrapy\core\scraper.py", line 71, in __init__ self.itemproc = itemproc_cls.from_crawler(crawler) File "e:\python3.7\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler return cls.from_settings(crawler.settings, crawler) File "e:\python3.7\lib\site-packages\scrapy\middleware.py", line 34, in from_settings mwcls = load_object(clspath) File "e:\python3.7\lib\site-packages\scrapy\utils\misc.py", line 46, in load_object mod = import_module(module) File "e:\python3.7\lib\importlib\__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 724, in exec_module File "<frozen importlib._bootstrap_external>", line 860, in get_code File "<frozen importlib._bootstrap_external>", line 791, in source_to_code File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "E:\python_work\爬虫\meizitu\meizitu\pipelines.py", line 22 f.write(data) ^ SyntaxError: unexpected EOF while parsing 代码如下: pipeline ``` import requests class MeizituPipeline(object): def process_item(self, item, spider): print("main_title:",item['main_title']) print("main_image:", item['main_image']) print("main_tags:", item['main_tags']) print("main_meta:", item['main_meta']) print("page:", item['main_pagenavi']) url = requests.get(item['main_image']) print(url) try: with open(item['main_pagenavi'] +'.jpg','wb') as f: data = url.read() f.write(data) ``` image.py ``` import scrapy from scrapy.http import response from ..items import MeizituItem class ImageSpider(scrapy.Spider): #定义Spider的名字scrapy crawl meiaitu name = 'SpiderMain' #允许爬虫的域名 allowed_domains = ['www.mzitu.com/203554'] #爬取的首页列表 start_urls = ['https://www.mzitu.com/203554'] #负责提取response的信息 #response代表下载器从start_urls中的url的到的回应 #提取的信息 def parse(self,response): #遍历所有节点 for Main in response.xpath('//div[@class = "main"]'): item = MeizituItem() #匹配所有节点元素/html/body/div[2]/div[1]/div[3]/p/a content = Main.xpath('//div[@class = "content"]') item['main_title'] = content.xpath('./h2/text()') item['main_image'] = content.xpath('./div[@class="main-image"]/p/a/img') item['main_meta'] = content.xpath('./div[@class="main-meta"]/span/text()').extract() item['main_tags'] = content.xpath('./div[@class="main-tags"]/a/text()').extract() item['main_pagenavi'] = content.xpath('./div[@class="main_pagenavi"]/span/text()').extract_first() yield item new_links = response.xpath('.//div[@class="pagenavi"]/a/@href').extract() new_link =new_links[-1] yield scrapy.Request(new_link,callback=self.parse) ``` setting ``` BOT_NAME = 'meizitu' SPIDER_MODULES = ['meizitu.spiders'] NEWSPIDER_MODULE = 'meizitu.spiders' ROBOTSTXT_OBEY = True #配置默认请求头 DEFAULT_REQUEST_HEADERS = { "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36", 'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' } ITEM_PIPELINES = { 'meizitu.pipelines.MeizituPipeline':300, } IMAGES_STORE = 'E:\python_work\爬虫\meizitu' IMAGES_MIN_HEIGHT = 1050 IMAGES_MIN_WIDTH = 700 ```
请问scrapy为什么会爬取失败
C:\Users\Administrator\Desktop\新建文件夹\xiaozhu>python -m scrapy crawl xiaozhu 2019-10-26 11:43:11 [scrapy.utils.log] INFO: Scrapy 1.7.3 started (bot: xiaozhu) 2019-10-26 11:43:11 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9 .5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.7.0, Python 3.5.3 (v 3.5.3:1880cb95a742, Jan 16 2017, 15:51:26) [MSC v.1900 32 bit (Intel)], pyOpenSS L 19.0.0 (OpenSSL 1.1.1c 28 May 2019), cryptography 2.7, Platform Windows-7-6.1 .7601-SP1 2019-10-26 11:43:11 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'xi aozhu', 'SPIDER_MODULES': ['xiaozhu.spiders'], 'NEWSPIDER_MODULE': 'xiaozhu.spid ers'} 2019-10-26 11:43:11 [scrapy.extensions.telnet] INFO: Telnet Password: c61bda45d6 3b8138 2019-10-26 11:43:11 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats'] 2019-10-26 11:43:12 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2019-10-26 11:43:12 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2019-10-26 11:43:12 [scrapy.middleware] INFO: Enabled item pipelines: [] 2019-10-26 11:43:12 [scrapy.core.engine] INFO: Spider opened 2019-10-26 11:43:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pag es/min), scraped 0 items (at 0 items/min) 2019-10-26 11:43:12 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2019-10-26 11:43:12 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting ( 307) to <GET https://bizverify.xiaozhu.com?slideRedirect=https%3A%2F%2Fbj.xiaozh u.com%2Ffangzi%2F125535477903.html> from <GET http://bj.xiaozhu.com/fangzi/12553 5477903.html> 2019-10-26 11:43:12 [scrapy.core.engine] DEBUG: Crawled (400) <GET https://bizve rify.xiaozhu.com?slideRedirect=https%3A%2F%2Fbj.xiaozhu.com%2Ffangzi%2F125535477 903.html> (referer: None) 2019-10-26 11:43:12 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <400 https://bizverify.xiaozhu.com?slideRedirect=https%3A%2F%2Fbj.xiaozhu.com%2 Ffangzi%2F125535477903.html>: HTTP status code is not handled or not allowed 2019-10-26 11:43:12 [scrapy.core.engine] INFO: Closing spider (finished) 2019-10-26 11:43:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 529, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 725, 'downloader/response_count': 2, 'downloader/response_status_count/307': 1, 'downloader/response_status_count/400': 1, 'elapsed_time_seconds': 0.427734, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2019, 10, 26, 3, 43, 12, 889648), 'httperror/response_ignored_count': 1, 'httperror/response_ignored_status_count/400': 1, 'log_count/DEBUG': 2, 'log_count/INFO': 11, 'response_received_count': 1, 'scheduler/dequeued': 2, 'scheduler/dequeued/memory': 2, 'scheduler/enqueued': 2, 'scheduler/enqueued/memory': 2, 'start_time': datetime.datetime(2019, 10, 26, 3, 43, 12, 461914)} 2019-10-26 11:43:12 [scrapy.core.engine] INFO: Spider closed (finished)
scrapy在settdings.py中已经设置好了DEFAULT_REQUEST_HEADERS,在发起请求的时候应该怎么写headers?
scrapy在settdings.py中已经设置好了DEFAULT_REQUEST_HEADERS,在发起请求的时候应该怎么写headers?
scrapy 运行抛出NotImplementedError,请问一般什么原因造成呢?
/usr/bin/python3.5 /home/pzs/PycharmProjects/News/main.py 2017-04-08 11:00:12 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: News) 2017-04-08 11:00:12 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'News', 'SPIDER_MODULES': ['News.spiders'], 'NEWSPIDER_MODULE': 'News.spiders'} 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.telnet.TelnetConsole',  'scrapy.extensions.corestats.CoreStats',  'scrapy.extensions.logstats.LogStats'] 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',  'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',  'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',  'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',  'scrapy.downloadermiddlewares.retry.RetryMiddleware',  'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',  'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',  'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',  'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',  'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',  'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',  'scrapy.spidermiddlewares.referer.RefererMiddleware',  'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',  'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled item pipelines: ['News.pipelines.MysqlPipeline'] 2017-04-08 11:00:12 [scrapy.core.engine] INFO: Spider opened 2017-04-08 11:00:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-04-08 11:00:12 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-04-08 11:00:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://18.92.0.1/contents/7/121174.html> (referer: None) 2017-04-08 11:00:13 [scrapy.core.scraper] ERROR: Spider error processing <GET http://18.92.0.1/contents/7/121174.html> (referer: None) Traceback (most recent call last):   File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks     current.result = callback(current.result, *args, **kw)   File "/usr/local/lib/python3.5/dist-packages/scrapy/spiders/__init__.py", line 76, in parse     raise NotImplementedError NotImplementedError 2017-04-08 11:00:13 [scrapy.core.engine] INFO: Closing spider (finished) 2017-04-08 11:00:13 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 229,  'downloader/request_count': 1,  'downloader/request_method_count/GET': 1,  'downloader/response_bytes': 16609,  'downloader/response_count': 1,  'downloader/response_status_count/200': 1,  'finish_reason': 'finished',  'finish_time': datetime.datetime(2017, 4, 8, 18, 0, 13, 938637),  'log_count/DEBUG': 2,  'log_count/ERROR': 1,  'log_count/INFO': 7,  'response_received_count': 1,  'scheduler/dequeued': 1,  'scheduler/dequeued/memory': 1,  'scheduler/enqueued': 1,  'scheduler/enqueued/memory': 1,  'spider_exceptions/NotImplementedError': 1,  'start_time': datetime.datetime(2017, 4, 8, 18, 0, 12, 917719)} 2017-04-08 11:00:13 [scrapy.core.engine] INFO: Spider closed (finished) Process finished with exit code 0 直接运行会弹出NotImplementedError错误,单步调试也看不出到底哪里出了问题
用anaconda的scrapy爬取数据,按照步骤设置好了,却爬不到数据,求助大神救救菜鸟
这是运行的全部结果: (D:\Anaconda2) C:\Users\luyue>cd C:\Users\luyue\movie250 (D:\Anaconda2) C:\Users\luyue\movie250>scrapy crawl movie250 -o items.json 2017-05-12 19:24:26 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: movie250) 2017-05-12 19:24:26 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'movie250.spiders', 'FEED_URI': 'items.json', 'SPIDER_MODULES': ['movie250.spiders'], 'BOT_NAME': 'movie250', 'ROBOTSTXT_OBEY': True, 'FEED_FORMAT': 'json'} 2017-05-12 19:24:26 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.corestats.CoreStats'] 2017-05-12 19:24:26 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2017-05-12 19:24:26 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-05-12 19:24:26 [scrapy.middleware] INFO: Enabled item pipelines: [] 2017-05-12 19:24:26 [scrapy.core.engine] INFO: Spider opened 2017-05-12 19:24:26 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-05-12 19:24:26 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-05-12 19:24:26 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://movie.douban.com/robots.txt> (referer: None) 2017-05-12 19:24:26 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://movie.douban.com/top250/> (referer: None) 2017-05-12 19:24:27 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://movie.douban.com/top250/>: HTTP status code is not handled or not allowed 2017-05-12 19:24:27 [scrapy.core.engine] INFO: Closing spider (finished) 2017-05-12 19:24:27 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 445, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 496, 'downloader/response_count': 2, 'downloader/response_status_count/403': 2, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 5, 12, 11, 24, 27, 13000), 'log_count/DEBUG': 3, 'log_count/INFO': 8, 'response_received_count': 2, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'start_time': datetime.datetime(2017, 5, 12, 11, 24, 26, 675000)} 2017-05-12 19:24:27 [scrapy.core.engine] INFO: Spider closed (finished)
为什么我用scrapy爬取谷歌应用市场却爬取不到内容?
我想用scrapy爬取谷歌应用市场,代码没有报错,但是却爬取不到内容,这是为什么? ``` # -*- coding: utf-8 -*- import scrapy # from scrapy.spiders import CrawlSpider, Rule # from scrapy.linkextractors import LinkExtractor from gp.items import GpItem # from html.parser import HTMLParser as SGMLParser import requests class GoogleSpider(scrapy.Spider): name = 'google' allowed_domains = ['https://play.google.com/'] start_urls = ['https://play.google.com/store/apps/'] ''' rules = [ Rule(LinkExtractor(allow=("https://play\.google\.com/store/apps/details",)), callback='parse_app', follow=True), ] ''' def parse(self, response): selector = scrapy.Selector(response) urls = selector.xpath('//a[@class="LkLjZd ScJHi U8Ww7d xjAeve nMZKrb id-track-click"]/@href').extract() link_flag = 0 links = [] for link in urls: links.append(link) for each in urls: yield scrapy.Request(links[link_flag], callback=self.parse_next, dont_filter=True) link_flag += 1 def parse_next(self, response): selector = scrapy.Selector(response) app_urls = selector.xpath('//div[@class="details"]/a[@class="title"]/@href').extract() print(app_urls) urls = [] for url in app_urls: url = "http://play.google.com" + url print(url) urls.append(url) link_flag = 0 for each in app_urls: yield scrapy.Request(urls[link_flag], callback=self.parse_app, dont_filter=True) link_flag += 1 def parse_app(self, response): item = GpItem() item['app_url'] = response.url item['app_name'] = response.xpath('//div[@itemprop="name"]').xpath('text()').extract() item['app_icon'] = response.xpath('//img[@itempro="image"]/@src') item['app_developer'] = response.xpath('//') print(response.text) yield item ``` terminal运行信息如下: ``` BettyMacbookPro-764:gp zhanjinyang$ scrapy crawl google 2019-11-12 08:46:45 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: gp) 2019-11-12 08:46:45 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 19.2.1, Python 3.7.1 (default, Dec 14 2018, 13:28:58) - [Clang 4.0.1 (tags/RELEASE_401/final)], pyOpenSSL 18.0.0 (OpenSSL 1.1.1a 20 Nov 2018), cryptography 2.4.2, Platform Darwin-18.5.0-x86_64-i386-64bit 2019-11-12 08:46:45 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'gp', 'NEWSPIDER_MODULE': 'gp.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['gp.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36'} 2019-11-12 08:46:45 [scrapy.extensions.telnet] INFO: Telnet Password: b2d7dedf1f4a91eb 2019-11-12 08:46:45 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats'] 2019-11-12 08:46:45 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2019-11-12 08:46:45 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2019-11-12 08:46:45 [scrapy.middleware] INFO: Enabled item pipelines: ['gp.pipelines.GpPipeline'] 2019-11-12 08:46:45 [scrapy.core.engine] INFO: Spider opened 2019-11-12 08:46:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2019-11-12 08:46:45 [py.warnings] WARNING: /anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py:61: URLWarning: allowed_domains accepts only domains, not URLs. Ignoring URL entry https://play.google.com/ in allowed_domains. warnings.warn(message, URLWarning) 2019-11-12 08:46:45 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2019-11-12 08:46:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://play.google.com/robots.txt> (referer: None) 2019-11-12 08:46:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://play.google.com/store/apps/> (referer: None) 2019-11-12 08:46:46 [scrapy.core.engine] INFO: Closing spider (finished) 2019-11-12 08:46:46 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 810, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 232419, 'downloader/response_count': 2, 'downloader/response_status_count/200': 2, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2019, 11, 12, 8, 46, 46, 474543), 'log_count/DEBUG': 2, 'log_count/INFO': 9, 'log_count/WARNING': 1, 'memusage/max': 58175488, 'memusage/startup': 58175488, 'response_received_count': 2, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/200': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'start_time': datetime.datetime(2019, 11, 12, 8, 46, 45, 562775)} 2019-11-12 08:46:46 [scrapy.core.engine] INFO: Spider closed (finished) ``` 求助!!!
python Scrapy创建项目出错,执行scrapy startproject test 出错;
Traceback (most recent call last): File "d:\users\july_whj\lib\runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "d:\users\july_whj\lib\runpy.py", line 72, in _run_code exec code in run_globals File "D:\Users\July_whj\Scripts\scrapy.exe\__main__.py", line 5, in <module> File "d:\users\july_whj\lib\site-packages\scrapy\cmdline.py", line 9, in <module> from scrapy.crawler import CrawlerProcess File "d:\users\july_whj\lib\site-packages\scrapy\crawler.py", line 7, in <module> from twisted.internet import reactor, defer File "d:\users\july_whj\lib\site-packages\twisted\internet\reactor.py", line 38, in <module> from twisted.internet import default File "d:\users\july_whj\lib\site-packages\twisted\internet\default.py", line 56, in <module> install = _getInstallFunction(platform) File "d:\users\july_whj\lib\site-packages\twisted\internet\default.py", line 50, in _getInstallFunction from twisted.internet.selectreactor import install File "d:\users\july_whj\lib\site-packages\twisted\internet\selectreactor.py", line 18, in <module> from twisted.internet import posixbase File "d:\users\july_whj\lib\site-packages\twisted\internet\posixbase.py", line 18, in <module> from twisted.internet import error, udp, tcp File "d:\users\july_whj\lib\site-packages\twisted\internet\tcp.py", line 28, in <module> from twisted.internet._newtls import ( File "d:\users\july_whj\lib\site-packages\twisted\internet\_newtls.py", line 21, in <module> from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol File "d:\users\july_whj\lib\site-packages\twisted\protocols\tls.py", line 63, in <module> from twisted.internet._sslverify import _setAcceptableProtocols File "d:\users\july_whj\lib\site-packages\twisted\internet\_sslverify.py", line 38, in <module> TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1, AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1' ![图片说明](https://img-ask.csdn.net/upload/201703/14/1489472404_160732.png)
scrapy1.4.0版本保存数据为JSON格式的疑问
spider的代码 ``` from textsc.items import TextscItem from scrapy.selector import Selector from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors import LinkExtractor class Baispider(CrawlSpider): name = "Baidu" allowed_domains = ["baidu.com"] start_urls = [ "https://zhidao.baidu.com/list" ] rules = ( Rule(LinkExtractor(allow=('/shop', ), deny=('fr', )), callback='parse_item'), ) def parse_item(self, response): sel= Selector(response) items=[] item=TextscItem() title=sel.xpath('//div[@class="shop-menu"]/ul/li/a/text()').extract() for i in title: items.append(i) item['TitleName'] = items print (item['TitleName']) return item ``` item.py的代码 ``` import scrapy import json class TextscItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() TitleName = scrapy.Field() pass ``` 运行时输入了 ``` scrapy crawl Bai -o items.json -t json ``` 运行时很正常 没报错 但是运行后点击查看了 items.json文件 什么都没有 求解决方法 谢过.
scrapy配置问题,求大家帮忙啊
配置scrapy 我是按照http://blog.csdn.net/wukaibo1986/article/details/8167590配置的 创建项目可以 但是运行项目的时候报错,做的demo是按照 http://www.oschina.net/translate/scrapy-demo做的 求解释: E:\爬虫\tutorial>scrapy crawl dmoz 2013-11-20 11:09:50+0800 [scrapy] INFO: Scrapy 0.20.0 started (bot: tutorial) 2013-11-20 11:09:50+0800 [scrapy] DEBUG: Optional features available: ssl, http11 2013-11-20 11:09:50+0800 [scrapy] DEBUG: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial'} 2013-11-20 11:09:50+0800 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState Traceback (most recent call last): File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "C:\Python27\lib\runpy.py", line 72, in _run_code exec code in run_globals File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\cmdline.py", line 168, in <module> execute() File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\cmdline.py", line 143, in execute _run_print_help(parser, _run_command, cmd, args, opts) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\cmdline.py", line 89, in _run_print_help func(*a, **kw) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\cmdline.py", line 150, in _run_command cmd.run(args, opts) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\commands\crawl.py", line 50, in run self.crawler_process.start() File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\crawler.py", line 92, in start if self.start_crawling(): File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\crawler.py", line 124, in start_crawling return self._start_crawler() is not None File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\crawler.py", line 139, in _start_crawler crawler.configure() File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\crawler.py", line 47, in configure self.engine = ExecutionEngine(self, self._spider_closed) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\core\engine.py", line 63, in __init__ self.downloader = Downloader(crawler) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\core\downloader\__init__.py", line 73, in __init__ self.handlers = DownloadHandlers(crawler) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\core\downloader\handlers\__init__.py", line 18, in __init__ cls = load_object(clspath) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\utils\misc.py", line 40, in load_object mod = import_module(module) File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module __import__(name) File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\core\downloader\handlers\s3.py", line 4, in <module> from .http import HTTPDownloadHandler File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\core\downloader\handlers\http.py", line 5, in <module> from .http11 import HTTP11DownloadHandler as HTTPDownloadHandler File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\core\downloader\handlers\http11.py", line 17, in <module> from scrapy.responsetypes import responsetypes File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\responsetypes.py", line 113, in <module> responsetypes = ResponseTypes() File "C:\Python27\lib\site-packages\scrapy-0.20.0-py2.7.egg\scrapy\responsetypes.py", line 34, in __init__ self.mimetypes = MimeTypes() File "C:\Python27\lib\mimetypes.py", line 66, in __init__ init() File "C:\Python27\lib\mimetypes.py", line 358, in init db.read_windows_registry() File "C:\Python27\lib\mimetypes.py", line 258, in read_windows_registry for subkeyname in enum_types(hkcr): File "C:\Python27\lib\mimetypes.py", line 249, in enum_types ctype = ctype.encode(default_encoding) # omit in 3.x! UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 9: ordinal not in range(128)
scrapy爬取知乎首页乱码
爬取知乎首页,返回的response.text是乱码,尝试解码response.body,得到的还是乱码,不知道为什么,代码如下: ``` import scrapy HEADERS = { 'Host': 'www.zhihu.com', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'zh-CN,zh;q=0.9', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive', 'Origin': 'https://www.zhihu.com', 'Referer': 'https://www.zhihu.com/', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36' } class ZhihuSpider(scrapy.Spider): name = 'zhihu' allowed_domains = ['www.zhihu.com'] start_urls = ['https://www.zhihu.com/'] def start_requests(self): for url in self.start_urls: yield scrapy.Request(url, headers=HEADERS) def parse(self, response): print('========== parse ==========') print(response.text[:100]) body = response.body encodings = ['utf-8', 'gbk', 'gb2312', 'iso-8859-1', 'latin1'] for encoding in encodings: try: print('========== decode ' + encoding) print(body.decode(encoding)[:100]) print('========== decode end\n') except Exception as e: print('########## decode {0}, error: {1}\n'.format(encoding, e)) pass ``` 输出的log如下: D:\workspace_python\ZhihuSpider>scrapy crawl zhihu 2017-12-01 11:12:03 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: ZhihuSpider) 2017-12-01 11:12:03 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'ZhihuSpider', 'FEED_EXPORT_ENCODING': 'utf-8', 'NEWSPIDER_MODULE': 'ZhihuSpider.spiders', 'SPIDER_MODULES': ['ZhihuSpider.spiders']} 2017-12-01 11:12:03 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] 2017-12-01 11:12:04 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2017-12-01 11:12:04 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-12-01 11:12:04 [scrapy.middleware] INFO: Enabled item pipelines: [] 2017-12-01 11:12:04 [scrapy.core.engine] INFO: Spider opened 2017-12-01 11:12:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-01 11:12:04 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-12-01 11:12:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.zhihu.com/> (referer: https://www.zhihu.com/) ========== parse ========== ��~!���#5���=B���_��^��ˆ� ═4�� 1���J�╗%Xi��/{�vH�"�� z�I�zLgü^�1� Q)Ա�_k}�䄍���/T����U�3���l��� ========== decode utf-8 ########## decode utf-8, error: 'utf-8' codec can't decode byte 0xe1 in position 0: invalid continuation byte ========== decode gbk ########## decode gbk, error: 'gbk' codec can't decode byte 0xa2 in position 4: illegal multibyte sequence ========== decode gb2312 ########## decode gb2312, error: 'gb2312' codec can't decode byte 0xa2 in position 4: illegal multibyte sequence ========== decode iso-8859-1 áø~!¢ 同样的代码,如果将爬取的网站换成douban,就一点问题都没有,百度找遍了都没找到办法,只能来这里提问了,请各位大神帮帮忙,如果爬虫搞不定,我仿的知乎后台就没数据展示了,真的很着急,。剩下不到5C币,没法悬赏,但真的需要大神的帮助。
在Linux上跑scrapy项目 [scrapy.core.engine] DEBUG
![图片说明](https://img-ask.csdn.net/upload/201906/12/1560308593_973487.png) 困扰我很久了,想爬入下来之后存入本地数据库,可是这个问题一直困扰我,ROBOTSTXT_OBEY = False,useragent也设置了,不知道问题是什么,网上试过了很多方法了,希望大佬 ``` ```
为什么我直接用requests爬网页可以,但用scrapy不行?
``` class job51(): def __init__(self): self.headers={ 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Encoding':'gzip, deflate, sdch', 'Accept-Language': 'zh-CN,zh;q=0.8', 'Cache-Control': 'max-age=0', 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36', 'Cookie':'' } def start(self): html=session.get("http://my.51job.com/cv/CResume/CV_CResumeManage.php",headers=self.headers) self.parse(html) def parse(self,response): tree=lxml.etree.HTML(response.text) resume_url=tree.xpath('//tbody/tr[@class="resumeName"]/td[1]/a/@href') print (resume_url[0] ``` 能爬到我想要的结果,就是简历的url,但是用scrapy,同样的headers,页面好像停留在登录页面? ``` class job51(Spider): name = "job51" #allowed_domains = ["my.51job.com"] start_urls = ["http://my.51job.com/cv/CResume/CV_CResumeManage.php"] headers={ 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Encoding':'gzip, deflate, sdch', 'Accept-Language': 'zh-CN,zh;q=0.8', 'Cache-Control': 'max-age=0', 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36', 'Cookie':'' } def start_requests(self): yield Request(url=self.start_urls[0],headers=self.headers,callback=self.parse) def parse(self,response): #tree=lxml.etree.HTML(text) selector=Selector(response) print ("<<<<<<<<<<<<<<<<<<<<<",response.text) resume_url=selector.xpath('//tr[@class="resumeName"]/td[1]/a/@href') print (">>>>>>>>>>>>",resume_url) ``` 输出的结果: scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'job51', 'SPIDER_MODULES': ['job51.spiders'], 'ROBOTSTXT_OBEY': True, 'NEWSPIDER_MODULE': 'job51.spiders'} 2017-04-11 10:58:31 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.logstats.LogStats', 'scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole'] 2017-04-11 10:58:32 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2017-04-11 10:58:32 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-04-11 10:58:32 [scrapy.middleware] INFO: Enabled item pipelines: [] 2017-04-11 10:58:32 [scrapy.core.engine] INFO: Spider opened 2017-04-11 10:58:32 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-04-11 10:58:32 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-04-11 10:58:33 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://my.51job.com/robots.txt> (referer: None) 2017-04-11 10:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://my.51job.com/cv/CResume/CV_CResumeManage.php> (referer: None) <<<<<<<<<<<<<<<<<<<<< <script>window.location='https://login.51job.com/login.php?url=http://my.51job.com%2Fcv%2FCResume%2FCV_CResumeManage.php%3F7087';</script> >>>>>>>>>>>> [] 2017-04-11 10:58:33 [scrapy.core.scraper] ERROR: Spider error processing <GET http://my.51job.com/cv/CResume/CV_CResumeManage.php> (referer: None) Traceback (most recent call last): File "d:\python35\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback yield next(it) File "d:\python35\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output for x in result: File "d:\python35\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 22, in <genexpr> return (_set_referer(r) for r in result or ()) File "d:\python35\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr> return (r for r in result or () if _filter(r)) File "d:\python35\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr> return (r for r in result or () if _filter(r)) File "E:\WorkGitResp\spider\job51\job51\spiders\51job_resume.py", line 43, in parse yield Request(resume_url[0],headers=self.headers,callback=self.getResume) File "d:\python35\lib\site-packages\parsel\selector.py", line 58, in __getitem__ o = super(SelectorList, self).__getitem__(pos) IndexError: list index out of range 2017-04-11 10:58:33 [scrapy.core.engine] INFO: Closing spider (finished) 2017-04-11 10:58:33 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 628, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 5743, 'downloader/response_count': 2, 'downloader/response_status_count/200': 1, 'downloader/response_status_count/404': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 4, 11, 2, 58, 33, 275634), 'log_count/DEBUG': 3, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'response_received_count': 2, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'spider_exceptions/IndexError': 1, 'start_time': datetime.datetime(2017, 4, 11, 2, 58, 32, 731603)} 2017-04-11 10:58:33 [scrapy.core.engine] INFO: Spider closed (finished)
非常简单的scrapy代码但就是不清楚到底哪里出问题了,高手帮忙看看吧!
News_spider文件 # -*- coding: utf-8 -*- import scrapy import re from scrapy import Selector from News.items import NewsItem class NewsSpiderSpider(scrapy.Spider): name = "news_spider" allowed_domains = ["http://18.92.0.1"] start_urls = ['http://18.92.0.1/contents/7/121174.html'] def parse_detail(self, response): sel = Selector(response) items = [] item = NewsItem() item['title'] = sel.css('.div_bt::text').extract()[0] characters = sel.css('.div_zz::text').extract()[0].replace("\xa0","") pattern = re.compile('[:].*[ ]') result = pattern.search(characters) item['post'] = result.group().replace(":","").strip() pattern = re.compile('[ ][^发]*') result = pattern.search(characters) item['approver'] = result.group() pattern = re.compile('[201].{9}') result = pattern.search(characters) item['date_of_publication'] = result.group() pattern = re.compile('([0-9]+)$') result = pattern.search(characters) item['browse_times'] = result.group() content = sel.css('.xwnr').extract()[0] pattern = re.compile('[\u4e00-\u9fa5]|[,、。“”]') result = pattern.findall(content) item['content'] = ''.join(result).replace("仿宋"," ").replace("宋体"," ").replace("楷体"," ") item['img1_url'] = sel.xpath('//*[@id="newpic"]/div[1]/div[1]/img/@src').extract()[0] item['img1_name'] = sel.xpath('//*[@id="newpic"]/div[1]/div[2]/text()').extract()[0] item['img2_url'] = sel.xpath('//*[@id="newpic"]/div[2]/div[1]/img/@src').extract()[0] item['img2_name'] = sel.xpath('//*[@id="newpic"]/div[2]/div[2]').extract()[0] item['img3_url'] = sel.xpath('//*[@id="newpic"]/div[3]/div[1]/img/@src').extract()[0] item['img3_name'] = sel.xpath('//*[@id="newpic"]/div[3]/div[2]/text()').extract()[0] item['img4_url'] = sel.xpath('//*[@id="newpic"]/div[4]/div[1]/img/@src').extract()[0] item['img4_name'] = sel.xpath('//*[@id="newpic"]/div[4]/div[2]/text()').extract()[0] item['img5_url'] = sel.xpath('//*[@id="newpic"]/div[5]/div[1]/img/@src').extract()[0] item['img5_name'] = sel.xpath('//*[@id="newpic"]/div[5]/div[2]/text()').extract()[0] item['img6_url'] = sel.xpath('//*[@id="newpic"]/div[6]/div[1]/img/@src').extract()[0] item['img6_name'] = sel.xpath('//*[@id="newpic"]/div[6]/div[2]/text()').extract()[0] characters = sel.xpath('/html/body/div/div[2]/div[4]/div[4]/text()').extract()[0].replace("\xa0","") pattern = re.compile('[:].*?[ ]') result = pattern.search(characters) item['company'] = result.group().replace(":", "").strip() pattern = re.compile('[ ][^联]*') result = pattern.search(characters) item['writer_photography'] = result.group() pattern = re.compile('(([0-9]|[-])+)$') result = pattern.search(characters) item['tel'] = result.group() items.append(item) items文件 return items import scrapy class NewsItem(scrapy.Item): title = scrapy.Field() post = scrapy.Field() approver = scrapy.Field() date_of_publication = scrapy.Field() browse_times = scrapy.Field() content = scrapy.Field() img1_url = scrapy.Field() img1_name = scrapy.Field() img2_url = scrapy.Field() img2_name = scrapy.Field() img3_url = scrapy.Field() img3_name = scrapy.Field() img4_url = scrapy.Field() img4_name = scrapy.Field() img5_url = scrapy.Field() img5_name = scrapy.Field() img6_url = scrapy.Field() img6_name = scrapy.Field() company = scrapy.Field() writer_photography = scrapy.Field() tel = scrapy.Field() pipelines文件 import MySQLdb import MySQLdb.cursors class NewsPipeline(object): def process_item(self, item, spider): return item class MysqlPipeline(object): def __init__(self): self.conn = MySQLdb.connect('192.168.254.129','root','root','news',charset="utf8",use_unicode=True) self.cursor = self.conn.cursor() def process_item(self, item, spider): insert_sql = "insert into news_table(title,post,approver,date_of_publication,browse_times,content,img1_url,img1_name,img2_url,img2_name,img3_url,img3_name,img4_url,img4_name,img5_url,img5_name,img6_url,img6_name,company,writer_photography,tel)VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)" self.cursor.execute(insert_sql,(item['title'],item['post'],item['approver'],item['date_of_publication'],item['browse_times'],item['content'],item['img1_url'],item['img1_name'],item['img1_url'],item['img1_name'],item['img2_url'],item['img2_name'],item['img3_url'],item['img3_name'],item['img4_url'],item['img4_name'],item['img5_url'],item['img5_name'],item['img6_url'],item['img6_name'],item['company'],item['writer_photography'],item['tel'])) self.conn.commit() setting文件 BOT_NAME = 'News' SPIDER_MODULES = ['News.spiders'] NEWSPIDER_MODULE = 'News.spiders' ROBOTSTXT_OBEY = False COOKIES_ENABLED = True ITEM_PIPELINES = { #'News.pipelines.NewsPipeline': 300, 'News.pipelines.MysqlPipeline': 300, } /usr/bin/python3.5 /home/pzs/PycharmProjects/News/main.py 2017-04-08 11:00:12 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: News) 2017-04-08 11:00:12 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'News', 'SPIDER_MODULES': ['News.spiders'], 'NEWSPIDER_MODULE': 'News.spiders'} 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats'] 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-04-08 11:00:12 [scrapy.middleware] INFO: Enabled item pipelines: ['News.pipelines.MysqlPipeline'] 2017-04-08 11:00:12 [scrapy.core.engine] INFO: Spider opened 2017-04-08 11:00:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-04-08 11:00:12 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-04-08 11:00:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://18.92.0.1/contents/7/121174.html> (referer: None) 2017-04-08 11:00:13 [scrapy.core.scraper] ERROR: Spider error processing <GET http://18.92.0.1/contents/7/121174.html> (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python3.5/dist-packages/scrapy/spiders/__init__.py", line 76, in parse raise NotImplementedError NotImplementedError 2017-04-08 11:00:13 [scrapy.core.engine] INFO: Closing spider (finished) 2017-04-08 11:00:13 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 229, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'downloader/response_bytes': 16609, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 4, 8, 18, 0, 13, 938637), 'log_count/DEBUG': 2, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'response_received_count': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'spider_exceptions/NotImplementedError': 1, 'start_time': datetime.datetime(2017, 4, 8, 18, 0, 12, 917719)} 2017-04-08 11:00:13 [scrapy.core.engine] INFO: Spider closed (finished) Process finished with exit code 0 直接运行会弹出NotImplementedError错误,单步调试也看不出到底哪里出了问题
scrapy运行爬虫时报错Missing scheme in request url
scrapy刚入门小白一枚。用网上的案例代码来玩一玩,案例是http://blog.csdn.net/czl389/article/details/77278166 中的爬取嘻哈歌词。这个案例下有三只爬虫,分别是songurls,lyrics和songinfo。我用songurls爬虫能从虾米音乐上爬取了url并保存在SongUrls.csv中,但是在用lyrics爬虫的时候会报错。信息如下 **D:\xiami2\xiami2>scrapy crawl lyrics -o Lyrics.csv 2017-10-21 21:13:29 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: xiami2) 2017-10-21 21:13:29 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'xiami2.spiders', 'USER_AGENT': 'Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 4.0; Trident/3.0)', 'FEED_URI': 'Lyrics.csv', 'FEED_FORMAT': 'csv', 'DOWNLOAD_DELAY': 0.2, 'SPIDER_MODULES': ['xiami2.spiders'], 'BOT_NAME': 'xiami2'} 2017-10-21 21:13:29 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2017-10-21 21:13:31 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2017-10-21 21:13:31 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-10-21 21:13:31 [scrapy.middleware] INFO: Enabled item pipelines: ['xiami2.pipelines.Xiami2Pipeline'] 2017-10-21 21:13:31 [scrapy.core.engine] INFO: Spider opened 2017-10-21 21:13:31 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-10-21 21:13:31 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-10-21 21:13:31 [scrapy.core.engine] ERROR: Error while obtaining start requests Traceback (most recent call last): File "d:\python3.5\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request request = next(slot.start_requests) File "d:\python3.5\lib\site-packages\scrapy\spiders\__init__.py", line 83, in start_requests yield Request(url, dont_filter=True) File "d:\python3.5\lib\site-packages\scrapy\http\request\__init__.py", line 25, in __init__ self._set_url(url) File "d:\python3.5\lib\site-packages\scrapy\http\request\__init__.py", line 58, in _set_url raise ValueError('Missing scheme in request url: %s' % self._url) ValueError: Missing scheme in request url: 2017-10-21 21:13:31 [scrapy.core.engine] INFO: Closing spider (finished) 2017-10-21 21:13:31 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 10, 21, 13, 13, 31, 567323), 'log_count/DEBUG': 1, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'start_time': datetime.datetime(2017, 10, 21, 13, 13, 31, 536236)} 2017-10-21 21:13:31 [scrapy.core.engine] INFO: Spider closed (finished) _------------------------------分割线--------------------------------------_ 我去查看了一下_init_.py,发现如下语句。 if ':' not in self._url: raise ValueError('Missing scheme in request url: %s' % self._url) 网上的解决方法看了一些,都没有能解决我的问题的,因此在此讨教,望大家指点一二(真没C币了)。提问次数不多,若有格式方面缺陷还请包含。 另附上代码。 #songurls.py import scrapy import re from scrapy.spiders import CrawlSpider, Rule from ..items import SongUrlItem class SongurlsSpider(scrapy.Spider): name = 'songurls' allowed_domains = ['xiami.com'] #将page/1到page/401,这些链接放进start_urls start_url_list=[] url_fixed='http://www.xiami.com/song/tag/Hip-Hop/page/' #将range范围扩大为1-401,获得所有页面 for i in range(1,402): start_url_list.extend([url_fixed+str(i)]) start_urls=start_url_list def parse(self,response): urls=response.xpath('//*[@id="wrapper"]/div[2]/div/div/div[2]/table/tbody/tr/td[2]/a[1]/@href').extract() for url in urls: song_url=response.urljoin(url) url_item=SongUrlItem() url_item['song_url']=song_url yield url_item ------------------------------分割线-------------------------------------- #lyrics.py import scrapy import re class LyricsSpider(scrapy.Spider): name = 'lyrics' allowed_domains = ['xiami.com'] song_url_file='SongUrls.csv' def __init__(self, *args, **kwargs): #从song_url.csv 文件中读取得到所有歌曲url f = open(self.song_url_file,"r") lines = f.readlines() #这里line[:-1]的含义是每行末尾都是一个换行符,要去掉 #这里in lines[1:]的含义是csv第一行是字段名称,要去掉 song_url_list=[line[:-1] for line in lines[1:]] f.close() while '\n' in song_url_list: song_url_list.remove('\n') self.start_urls = song_url_list#[:100]#删除[:100]之后爬取全部数据 def parse(self,response): lyric_lines=response.xpath('//*[@id="lrc"]/div[1]/text()').extract() lyric='' for lyric_line in lyric_lines: lyric+=lyric_line #print lyric lyricItem=LyricItem() lyricItem['lyric']=lyric lyricItem['song_url']=response.url yield lyricItem songinfo因为还没有用到所以不重要。 ------------------------------分割线-------------------------------------- #items.py import scrapy class SongUrlItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() song_url=scrapy.Field() #歌曲链接 class LyricItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() lyric=scrapy.Field() #歌词 song_url=scrapy.Field() #歌曲链接 class SongInfoItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() song_url=scrapy.Field() #歌曲链接 song_title=scrapy.Field() #歌名 album=scrapy.Field() #专辑 #singer=scrapy.Field() #歌手 language=scrapy.Field() #语种 ------------------------------分割线-------------------------------------- 在middleware下加了几行: sleep_seconds = 0.2 # 模拟点击后休眠3秒,给出浏览器取得响应内容的时间 default_sleep_seconds = 1 # 无动作请求休眠的时间 def process_request(self, request, spider): spider.logger.info('--------Spider request processed: %s' % spider.name) page = None driver = webdriver.PhantomJS() spider.logger.info('--------request.url: %s' % request.url) driver.get(request.url) driver.implicitly_wait(0.2) # 仅休眠数秒加载页面后返回内容 time.sleep(self.sleep_seconds) page = driver.page_source driver.close() return HtmlResponse(request.url, body=page, encoding='utf-8', request=request) ------------------------------分割线-------------------------------------- setting中加了几行也改了几行: from faker import Factory f = Factory.create() USER_AGENT = f.user_agent() DOWNLOAD_DELAY = 0.2 DEFAULT_REQUEST_HEADERS = { 'Host': 'www.xiami.com', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'zh-CN,zh;q=0.8', 'Cache-Control': 'no-cache', 'Connection': 'Keep-Alive', } ITEM_PIPELINES = { 'xiami2.pipelines.Xiami2Pipeline': 300, }
新手学scrapy ,sys.exit(execute()) 以泪崩,求指导
![图片说明](https://img-ask.csdn.net/upload/201707/12/1499849181_65780.png) os系统,python3.6,根据这个15年的学习贴 http://blog.csdn.net/u012150179/article/details/32911511 一字不差的输入,然后死活不能正常运行 小白求大神
用Python安装scrapy第三方库时安装失败怎么办
ERROR: Command errored out with exit status 1: command: 'c:\program files\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\22348\\AppData\\Local\\Temp\\pip-install-s_hynvin\\Twisted\\setup.py'"'"'; __file__='"'"'C:\\Users\\22348\\AppData\\Local\\Temp\\pip-install-s_hynvin\\Twisted\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\22348\AppData\Local\Temp\pip-wheel-4c5rcjky' --python-tag cp37 cwd: C:\Users\22348\AppData\Local\Temp\pip-install-s_hynvin\Twisted\ Complete output (943 lines): running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-3.7 creating build\lib.win-amd64-3.7\twisted copying src\twisted\copyright.py -> build\lib.win-amd64-3.7\twisted copying src\twisted\plugin.py -> build\lib.win-amd64-3.7\twisted copying src\twisted\_version.py -> build\lib.win-amd64-3.7\twisted copying src\twisted\__init__.py -> build\lib.win-amd64-3.7\twisted copying src\twisted\__main__.py -> build\lib.win-amd64-3.7\twisted creating build\lib.win-amd64-3.7\twisted\application copying src\twisted\application\app.py -> build\lib.win-amd64-3.7\twisted\application copying src\twisted\application\internet.py -> build\lib.win-amd64-3.7\twisted\application copying src\twisted\application\reactors.py -> build\lib.win-amd64-3.7\twisted\application copying src\twisted\application\service.py -> build\lib.win-amd64-3.7\twisted\application copying src\twisted\application\strports.py -> build\lib.win-amd64-3.7\twisted\application copying src\twisted\application\__init__.py -> build\lib.win-amd64-3.7\twisted\application creating build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\avatar.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\checkers.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\endpoints.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\error.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\interfaces.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\ls.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\manhole.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\manhole_ssh.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\manhole_tap.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\mixin.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\recvline.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\stdio.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\tap.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\telnet.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\ttymodes.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\unix.py -> build\lib.win-amd64-3.7\twisted\conch copying src\twisted\conch\__init__.py -> build\lib.win-amd64-3.7\twisted\conch creating build\lib.win-amd64-3.7\twisted\cred copying src\twisted\cred\checkers.py -> build\lib.win-amd64-3.7\twisted\cred copying src\twisted\cred\credentials.py -> build\lib.win-amd64-3.7\twisted\cred copying src\twisted\cred\error.py -> build\lib.win-amd64-3.7\twisted\cred copying src\twisted\cred\portal.py -> build\lib.win-amd64-3.7\twisted\cred copying src\twisted\cred\strcred.py -> build\lib.win-amd64-3.7\twisted\cred copying src\twisted\cred\_digest.py -> build\lib.win-amd64-3.7\twisted\cred copying src\twisted\cred\__init__.py -> build\lib.win-amd64-3.7\twisted\cred creating build\lib.win-amd64-3.7\twisted\enterprise copying src\twisted\enterprise\adbapi.py -> build\lib.win-amd64-3.7\twisted\enterprise copying src\twisted\enterprise\__init__.py -> build\lib.win-amd64-3.7\twisted\enterprise creating build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\abstract.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\address.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\asyncioreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\base.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\cfreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\default.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\defer.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\endpoints.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\epollreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\error.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\fdesc.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\gireactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\glib2reactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\gtk2reactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\gtk3reactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\inotify.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\interfaces.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\kqreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\main.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\pollreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\posixbase.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\process.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\protocol.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\pyuisupport.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\reactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\selectreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\serialport.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\ssl.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\stdio.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\task.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\tcp.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\threads.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\tksupport.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\udp.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\unix.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\utils.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\win32eventreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\wxreactor.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\wxsupport.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_baseprocess.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_dumbwin32proc.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_glibbase.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_idna.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_newtls.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_pollingfile.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_posixserialport.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_posixstdio.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_producer_helpers.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_resolver.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_signals.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_sslverify.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_threadedselect.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_win32serialport.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\_win32stdio.py -> build\lib.win-amd64-3.7\twisted\internet copying src\twisted\internet\__init__.py -> build\lib.win-amd64-3.7\twisted\internet creating build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_buffer.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_file.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_filter.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_flatten.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_format.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_global.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_io.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_json.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_legacy.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_levels.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_logger.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_observer.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_stdlib.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\_util.py -> build\lib.win-amd64-3.7\twisted\logger copying src\twisted\logger\__init__.py -> build\lib.win-amd64-3.7\twisted\logger creating build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\imap4.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\interfaces.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\pop3.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\pop3client.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\protocols.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\relay.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\smtp.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\_cred.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\_except.py -> build\lib.win-amd64-3.7\twisted\mail copying src\twisted\mail\__init__.py -> build\lib.win-amd64-3.7\twisted\mail creating build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\authority.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\cache.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\client.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\common.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\dns.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\error.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\hosts.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\resolve.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\root.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\secondary.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\server.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\srvconnect.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\tap.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\_rfc1982.py -> build\lib.win-amd64-3.7\twisted\names copying src\twisted\names\__init__.py -> build\lib.win-amd64-3.7\twisted\names creating build\lib.win-amd64-3.7\twisted\pair copying src\twisted\pair\ethernet.py -> build\lib.win-amd64-3.7\twisted\pair copying src\twisted\pair\ip.py -> build\lib.win-amd64-3.7\twisted\pair copying src\twisted\pair\raw.py -> build\lib.win-amd64-3.7\twisted\pair copying src\twisted\pair\rawudp.py -> build\lib.win-amd64-3.7\twisted\pair copying src\twisted\pair\testing.py -> build\lib.win-amd64-3.7\twisted\pair copying src\twisted\pair\tuntap.py -> build\lib.win-amd64-3.7\twisted\pair copying src\twisted\pair\__init__.py -> build\lib.win-amd64-3.7\twisted\pair creating build\lib.win-amd64-3.7\twisted\persisted copying src\twisted\persisted\aot.py -> build\lib.win-amd64-3.7\twisted\persisted copying src\twisted\persisted\crefutil.py -> build\lib.win-amd64-3.7\twisted\persisted copying src\twisted\persisted\dirdbm.py -> build\lib.win-amd64-3.7\twisted\persisted copying src\twisted\persisted\sob.py -> build\lib.win-amd64-3.7\twisted\persisted copying src\twisted\persisted\styles.py -> build\lib.win-amd64-3.7\twisted\persisted copying src\twisted\persisted\__init__.py -> build\lib.win-amd64-3.7\twisted\persisted creating build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\cred_anonymous.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\cred_file.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\cred_memory.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\cred_sshkeys.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\cred_unix.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_conch.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_core.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_ftp.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_inet.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_names.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_portforward.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_reactors.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_runner.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_socks.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_trial.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_web.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\twisted_words.py -> build\lib.win-amd64-3.7\twisted\plugins copying src\twisted\plugins\__init__.py -> build\lib.win-amd64-3.7\twisted\plugins creating build\lib.win-amd64-3.7\twisted\positioning copying src\twisted\positioning\base.py -> build\lib.win-amd64-3.7\twisted\positioning copying src\twisted\positioning\ipositioning.py -> build\lib.win-amd64-3.7\twisted\positioning copying src\twisted\positioning\nmea.py -> build\lib.win-amd64-3.7\twisted\positioning copying src\twisted\positioning\_sentence.py -> build\lib.win-amd64-3.7\twisted\positioning copying src\twisted\positioning\__init__.py -> build\lib.win-amd64-3.7\twisted\positioning creating build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\amp.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\basic.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\dict.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\finger.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\ftp.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\htb.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\ident.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\loopback.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\memcache.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\pcp.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\policies.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\portforward.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\postfix.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\sip.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\socks.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\stateful.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\tls.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\wire.py -> build\lib.win-amd64-3.7\twisted\protocols copying src\twisted\protocols\__init__.py -> build\lib.win-amd64-3.7\twisted\protocols creating build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\compat.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\components.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\constants.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\context.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\deprecate.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\failure.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\fakepwd.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\filepath.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\formmethod.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\htmlizer.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\lockfile.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\log.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\logfile.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\modules.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\monkey.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\procutils.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\randbytes.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\rebuild.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\reflect.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\release.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\roots.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\runtime.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\sendmsg.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\shortcut.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\syslog.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\systemd.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\text.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\threadable.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\threadpool.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\url.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\urlpath.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\usage.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\util.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\versions.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\win32.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\zippath.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\zipstream.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_appdirs.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_inotify.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_oldstyle.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_release.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_setup.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_shellcomp.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_textattributes.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_tzhelper.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\_url.py -> build\lib.win-amd64-3.7\twisted\python copying src\twisted\python\__init__.py -> build\lib.win-amd64-3.7\twisted\python creating build\lib.win-amd64-3.7\twisted\runner copying src\twisted\runner\inetd.py -> build\lib.win-amd64-3.7\twisted\runner copying src\twisted\runner\inetdconf.py -> build\lib.win-amd64-3.7\twisted\runner copying src\twisted\runner\inetdtap.py -> build\lib.win-amd64-3.7\twisted\runner copying src\twisted\runner\procmon.py -> build\lib.win-amd64-3.7\twisted\runner copying src\twisted\runner\procmontap.py -> build\lib.win-amd64-3.7\twisted\runner copying src\twisted\runner\__init__.py -> build\lib.win-amd64-3.7\twisted\runner creating build\lib.win-amd64-3.7\twisted\scripts copying src\twisted\scripts\htmlizer.py -> build\lib.win-amd64-3.7\twisted\scripts copying src\twisted\scripts\trial.py -> build\lib.win-amd64-3.7\twisted\scripts copying src\twisted\scripts\twistd.py -> build\lib.win-amd64-3.7\twisted\scripts copying src\twisted\scripts\_twistd_unix.py -> build\lib.win-amd64-3.7\twisted\scripts copying src\twisted\scripts\_twistw.py -> build\lib.win-amd64-3.7\twisted\scripts copying src\twisted\scripts\__init__.py -> build\lib.win-amd64-3.7\twisted\scripts creating build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\banana.py -> build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\flavors.py -> build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\interfaces.py -> build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\jelly.py -> build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\pb.py -> build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\publish.py -> build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\util.py -> build\lib.win-amd64-3.7\twisted\spread copying src\twisted\spread\__init__.py -> build\lib.win-amd64-3.7\twisted\spread creating build\lib.win-amd64-3.7\twisted\tap copying src\twisted\tap\ftp.py -> build\lib.win-amd64-3.7\twisted\tap copying src\twisted\tap\portforward.py -> build\lib.win-amd64-3.7\twisted\tap copying src\twisted\tap\socks.py -> build\lib.win-amd64-3.7\twisted\tap copying src\twisted\tap\__init__.py -> build\lib.win-amd64-3.7\twisted\tap creating build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\crash_test_dummy.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\iosim.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\mock_win32process.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\myrebuilder1.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\myrebuilder2.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\plugin_basic.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\plugin_extra1.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\plugin_extra2.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\process_cmdline.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\process_echoer.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\process_fds.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\process_getargv.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\process_getenv.py -> build\lib.win-amd64-3.7\twisted\test copying src\twisted\test\process_linger.py -> build\lib.win-amd64-3.7\twisted\test
一个百度拇指医生爬虫,想要先实现爬取某个问题的所有链接,但是爬不出来东西。求各位大神帮忙看一下这是为什么?
#写在前面的话 在这个爬虫里我想实现把百度拇指医生里关于“咳嗽”的链接全部爬取下来,下一步要进行的是把爬取到的每个链接里的items里面的内容爬取下来,但是我在第一步就卡住了,求各位大神帮我看一下吧。之前刚刚发了一篇问答,但是不知道怎么回事儿,现在找不到了,(貌似是被删了...?)救救小白吧!感激不尽! 这个是我的爬虫的结构 ![图片说明](https://img-ask.csdn.net/upload/201911/27/1574787999_274479.png) ##ks: ``` # -*- coding: utf-8 -*- import scrapy from kesou.items import KesouItem from scrapy.selector import Selector from scrapy.spiders import Spider from scrapy.http import Request ,FormRequest import pymongo class KsSpider(scrapy.Spider): name = 'ks' allowed_domains = ['kesou,baidu.com'] start_urls = ['https://www.baidu.com/s?wd=%E5%92%B3%E5%97%BD&pn=0&oq=%E5%92%B3%E5%97%BD&ct=2097152&ie=utf-8&si=muzhi.baidu.com&rsv_pq=980e0c55000e2402&rsv_t=ed3f0i5yeefxTMskgzim00cCUyVujMRnw0Vs4o1%2Bo%2Bohf9rFXJvk%2FSYX%2B1M'] def parse(self, response): item = KesouItem() contents = response.xpath('.//h3[@class="t"]') for content in contents: url = content.xpath('.//a/@href').extract()[0] item['url'] = url yield item if self.offset < 760: self.offset += 10 yield scrapy.Request(url = "https://www.baidu.com/s?wd=%E5%92%B3%E5%97%BD&pn=" + str(self.offset) + "&oq=%E5%92%B3%E5%97%BD&ct=2097152&ie=utf-8&si=muzhi.baidu.com&rsv_pq=980e0c55000e2402&rsv_t=ed3f0i5yeefxTMskgzim00cCUyVujMRnw0Vs4o1%2Bo%2Bohf9rFXJvk%2FSYX%2B1M",callback=self.parse,dont_filter=True) ``` ##items: ``` # -*- coding: utf-8 -*- # Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class KesouItem(scrapy.Item): # 问题ID question_ID = scrapy.Field() # 问题描述 question = scrapy.Field() # 医生回答发表时间 answer_pubtime = scrapy.Field() # 问题详情 description = scrapy.Field() # 医生姓名 doctor_name = scrapy.Field() # 医生职位 doctor_title = scrapy.Field() # 医生所在医院 hospital = scrapy.Field() ``` ##middlewares: ``` # -*- coding: utf-8 -*- # Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals class KesouSpiderMiddleware(object): # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, dict or Item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request, dict # or Item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) class KesouDownloaderMiddleware(object): # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called return None def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) ``` ##piplines: ``` # -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html import pymongo from scrapy.utils.project import get_project_settings settings = get_project_settings() class KesouPipeline(object): def __init__(self): host = settings["MONGODB_HOST"] port = settings["MONGODB_PORT"] dbname = settings["MONGODB_DBNAME"] sheetname= settings["MONGODB_SHEETNAME"] # 创建MONGODB数据库链接 client = pymongo.MongoClient(host = host, port = port) # 指定数据库 mydb = client[dbname] # 存放数据的数据库表名 self.sheet = mydb[sheetname] def process_item(self, item, spider): data = dict(item) self.sheet.insert(data) return item ``` ##settings: ``` # -*- coding: utf-8 -*- # Scrapy settings for kesou project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = 'kesou' SPIDER_MODULES = ['kesou.spiders'] NEWSPIDER_MODULE = 'kesou.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'kesou (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False USER_AGENT="Mozilla/5.0 (Windows NT 10.0; WOW64; rv:67.0) Gecko/20100101 Firefox/67.0" # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'kesou.middlewares.KesouSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = { # 'kesou.middlewares.KesouDownloaderMiddleware': 543, #} # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { 'kesou.pipelines.KesouPipeline': 300, } # MONGODB 主机名 MONGODB_HOST = "127.0.0.1" # MONGODB 端口号 MONGODB_PORT = 27017 # 数据库名称 MONGODB_DBNAME = "ks" # 存放数据的表名称 MONGODB_SHEETNAME = "ks_urls" # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' ``` ##run.py: ``` # -*- coding: utf-8 -*- from scrapy import cmdline cmdline.execute("scrapy crawl ks".split()) ``` ##这个是运行出来的结果: ``` PS D:\scrapy_project\kesou> scrapy crawl ks 2019-11-27 00:14:17 [scrapy.utils.log] INFO: Scrapy 1.7.3 started (bot: kesou) 2019-11-27 00:14:17 [scrapy.utils.log] INFO: Versions: lxml 4.3.2.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twis.7.0, Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1b 26 Feb 2019), cryphy 2.6.1, Platform Windows-10-10.0.18362-SP0 2019-11-27 00:14:17 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'kesou', 'COOKIES_ENABLED': False, 'NEWSPIDER_MODULE': 'spiders', 'SPIDER_MODULES': ['kesou.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:67.0) Gecko/20100101 Firefox/67 2019-11-27 00:14:17 [scrapy.extensions.telnet] INFO: Telnet Password: 051629c46f34abdf 2019-11-27 00:14:17 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] 2019-11-27 00:14:19 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2019-11-27 00:14:19 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2019-11-27 00:14:19 [scrapy.middleware] INFO: Enabled item pipelines: ['kesou.pipelines.KesouPipeline'] 2019-11-27 00:14:19 [scrapy.core.engine] INFO: Spider opened 2019-11-27 00:14:19 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2019-11-27 00:14:19 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2019-11-27 00:14:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.baidu.com/s?wd=%E5%92%B3%E5%97%BD&pn=0&oq=%E5%92%B3%E5&ct=2097152&ie=utf-8&si=muzhi.baidu.com&rsv_pq=980e0c55000e2402&rsv_t=ed3f0i5yeefxTMskgzim00cCUyVujMRnw0Vs4o1%2Bo%2Bohf9rFXJvk%2FSYX% (referer: None) 2019-11-27 00:14:20 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.baidu.com/s?wd=%E5%92%B3%E5%97%BD&pn=0&oq=%B3%E5%97%BD&ct=2097152&ie=utf-8&si=muzhi.baidu.com&rsv_pq=980e0c55000e2402&rsv_t=ed3f0i5yeefxTMskgzim00cCUyVujMRnw0Vs4o1%2Bo%2Bohf9rFFSYX%2B1M> (referer: None) Traceback (most recent call last): File "d:\anaconda3\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback yield next(it) File "d:\anaconda3\lib\site-packages\scrapy\core\spidermw.py", line 84, in evaluate_iterable for r in iterable: File "d:\anaconda3\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output for x in result: File "d:\anaconda3\lib\site-packages\scrapy\core\spidermw.py", line 84, in evaluate_iterable for r in iterable: File "d:\anaconda3\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr> return (_set_referer(r) for r in result or ()) File "d:\anaconda3\lib\site-packages\scrapy\core\spidermw.py", line 84, in evaluate_iterable for r in iterable: File "d:\anaconda3\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr> return (r for r in result or () if _filter(r)) File "d:\anaconda3\lib\site-packages\scrapy\core\spidermw.py", line 84, in evaluate_iterable for r in iterable: File "d:\anaconda3\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr> return (r for r in result or () if _filter(r)) File "D:\scrapy_project\kesou\kesou\spiders\ks.py", line 19, in parse item['url'] = url File "d:\anaconda3\lib\site-packages\scrapy\item.py", line 73, in __setitem__ (self.__class__.__name__, key)) KeyError: 'KesouItem does not support field: url' 2019-11-27 00:14:20 [scrapy.core.engine] INFO: Closing spider (finished) 2019-11-27 00:14:20 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 438, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'downloader/response_bytes': 68368, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'elapsed_time_seconds': 0.992207, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2019, 11, 26, 16, 14, 20, 855804), 'log_count/DEBUG': 1, 2019-11-27 00:14:20 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 438, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'downloader/response_bytes': 68368, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'elapsed_time_seconds': 0.992207, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2019, 11, 26, 16, 14, 20, 855804), 'log_count/DEBUG': 1, 'log_count/ERROR': 1, 'log_count/INFO': 10, 'response_received_count': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'spider_exceptions/KeyError': 1, 'start_time': datetime.datetime(2019, 11, 26, 16, 14, 19, 863597)} 2019-11-27 00:14:21 [scrapy.core.engine] INFO: Spider closed (finished) ```
scrapy爬虫出现 DEBUG: Crawled (404)
为什么会出现解析页面错误呢? ``` 2019-04-17 16:14:46 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://www.xxx.xx/xxgk/xwzx/zwdt/None> (referer: http://www.xxx.xx/xxgk/xwzx/zwdt/index_1.htm) 2019-04-17 16:14:46 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://www.xxx.xx/xxgk/xwzx/zwdt/None> (referer: http://www.xxx.xx/xxgk/xwzx/zwdt/index_1.htm) 2019-04-17 16:14:46 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://www.xxx.xx/xxgk/xwzx/zwdt/None> (referer: http://www.xxx.xx/xxgk/xwzx/zwdt/index_1.htm) 2019-04-17 16:14:46 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://www.xxx.xx/xxgk/xwzx/zwdt/None> (referer: http://www.xxx.xx/xxgk/xwzx/zwdt/index_1.htm) 2019-04-17 16:14:46 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://www.xxx.xx/xxgk/xwzx/zwdt/None> (referer: http://www.xxx.xx/xxgk/xwzx/zwdt/index_1.htm) ``` 代码在上边,解析一个页面为什么返回了 页面的None,?是网站监测到爬虫了吗?scrapy需要再哪更改headers呢 我尝试了在主程序里更改 然后yield 然而失败了 谢谢各位
python scrapy 爬取的数据保存不了
# python scrapy 爬取的数据保存不了 上代码 spider代码: ``` from textsc.items import TextscItem from scrapy.selector import Selector from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors import LinkExtractor class Baispider(CrawlSpider): name = "Baidu" allowed_domains = ["baidu.com"] start_urls = [ "https://zhidao.baidu.com/list" ] rules = ( Rule(LinkExtractor(allow=('/shop', ), deny=('fr', )), callback='parse_item'), ) def parse_item(self, response): sel= Selector(response) items=[] item=TextscItem() title=sel.xpath('//div[@class="shop-menu"]/ul/li/a/text()').extract() for i in title: items.append(i) item['TitleName'] = items print (item['TitleName']) return item ``` items.py代码 ``` import scrapy import json class TextscItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() TitleName = scrapy.Field() pass ``` scrapy的版本是1.4.0 运行没有报错 但是json文件里面什么都没有 求解答 谢过!..
终于明白阿里百度这样的大公司,为什么面试经常拿ThreadLocal考验求职者了
点击上面↑「爱开发」关注我们每晚10点,捕获技术思考和创业资源洞察什么是ThreadLocalThreadLocal是一个本地线程副本变量工具类,各个线程都拥有一份线程私有的数
win10系统安装教程(U盘PE+UEFI安装)
一、准备工作 u盘,电脑一台,win10原版镜像(msdn官网) 二、下载wepe工具箱  极力推荐微pe(微pe官方下载) 下载64位的win10 pe,使用工具箱制作启动U盘打开软件,   选择安装到U盘(按照操作无需更改) 三、重启进入pe系统   1、关机后,将U盘插入电脑 2、按下电源后,按住F12进入启动项选择(技嘉主板是F12)     选择需要启
程序员必须掌握的核心算法有哪些?
由于我之前一直强调数据结构以及算法学习的重要性,所以就有一些读者经常问我,数据结构与算法应该要学习到哪个程度呢?,说实话,这个问题我不知道要怎么回答你,主要取决于你想学习到哪些程度,不过针对这个问题,我稍微总结一下我学过的算法知识点,以及我觉得值得学习的算法。这些算法与数据结构的学习大多数是零散的,并没有一本把他们全部覆盖的书籍。下面是我觉得值得学习的一些算法以及数据结构,当然,我也会整理一些看过
《奇巧淫技》系列-python!!每天早上八点自动发送天气预报邮件到QQ邮箱
将代码部署服务器,每日早上定时获取到天气数据,并发送到邮箱。 也可以说是一个小人工智障。 思路可以运用在不同地方,主要介绍的是思路。
Nginx 软件层面加强Nginx性能优化的面试问答和解决方案
Nginx 软件层面加强Nginx性能优化的面试问答和解决方案 去年我去爱卡汽车面试PHP,一轮和二轮面的都不错,在三轮面到Nginx的时候很多问题当时不知道怎么回答,确实没有深入学习过,花了一段时间的学习,终于能解答Nginx高性能优化的问题了,10月24号为了获得程序员勋章,发布了半个优化笔记,浏览到了1000+,受到这个鼓舞,我抽时间在仔细整理下关于Nginx性能优化的问题,我们从软件说起。...
【管理系统课程设计】美少女手把手教你后台管理
【文章后台管理系统】URL设计与建模分析+项目源码+运行界面 栏目管理、文章列表、用户管理、角色管理、权限管理模块(文章最后附有源码) 1. 这是一个什么系统? 1.1 学习后台管理系统的原因 随着时代的变迁,现如今各大云服务平台横空出世,市面上有许多如学生信息系统、图书阅读系统、停车场管理系统等的管理系统,而本人家里就有人在用烟草销售系统,直接在网上完成挑选、购买与提交收货点,方便又快捷。 试想,若没有烟草销售系统,本人家人想要购买烟草,还要独自前往药...
11月19日科技资讯|华为明日发布鸿蒙整体战略;京东宣告全面向技术转型;Kotlin 1.3.60 发布
「极客头条」—— 技术人员的新闻圈! CSDN 的读者朋友们早上好哇,「极客头条」来啦,快来看今天都有哪些值得我们技术人关注的重要新闻吧。扫描上方二维码进入 CSDN App 可以收听御姐萌妹 Style 的人工版音频哟。 一分钟速览新闻点! 6G 专家组成员:速率是 5G 的 10 至 100 倍,预计 2030 年商用 雷军:很多人多次劝我放弃WPS,能坚持下来并不是纯粹的商业决定 ...
C语言魔塔游戏
很早就很想写这个,今天终于写完了。 游戏截图: 编译环境: VS2017 游戏需要一些图片,如果有想要的或者对游戏有什么看法的可以加我的QQ 2985486630 讨论,如果暂时没有回应,可以在博客下方留言,到时候我会看到。 下面我来介绍一下游戏的主要功能和实现方式 首先是玩家的定义,使用结构体,这个名字是可以自己改变的 struct gamerole { char n
化繁为简 - 腾讯计费高一致TDXA的实践之路
导语:腾讯计费是孵化于支撑腾讯内部业务千亿级营收的互联网计费平台,在如此庞大的业务体量下,腾讯计费要支撑业务的快速增长,同时还要保证每笔交易不错账。采用最终一致性或离线补...
Python爬虫爬取淘宝,京东商品信息
小编是一个理科生,不善长说一些废话。简单介绍下原理然后直接上代码。 使用的工具(Python+pycharm2019.3+selenium+xpath+chromedriver)其中要使用pycharm也可以私聊我selenium是一个框架可以通过pip下载 pip install selenium -i https://pypi.tuna.tsinghua.edu.cn/simple/ 
Java学习笔记(七十二)—— Cookie
概述 会话技术: 会话:一次会话中包含多次请求和响应 一次会话:浏览器第一次给服务器发送资源请求,会话建立,直到有一方断开为止 功能:在一次会话的范围内的多次请求间,共享数据 方式: 客户端会话技术:Cookie,把数据存储到客户端 服务器端会话技术:Session,把数据存储到服务器端 Cookie 概念:客户端会话技术,将数据存储到客户端 快速入门: 使用步骤: 创建C
阿里程序员写了一个新手都写不出的低级bug,被骂惨了。
这种新手都不会范的错,居然被一个工作好几年的小伙子写出来,差点被当场开除了。
Java工作4年来应聘要16K最后没要,细节如下。。。
前奏: 今天2B哥和大家分享一位前几天面试的一位应聘者,工作4年26岁,统招本科。 以下就是他的简历和面试情况。 基本情况: 专业技能: 1、&nbsp;熟悉Sping了解SpringMVC、SpringBoot、Mybatis等框架、了解SpringCloud微服务 2、&nbsp;熟悉常用项目管理工具:SVN、GIT、MAVEN、Jenkins 3、&nbsp;熟悉Nginx、tomca
2020年,冯唐49岁:我给20、30岁IT职场年轻人的建议
点击“技术领导力”关注∆  每天早上8:30推送 作者| Mr.K   编辑| Emma 来源| 技术领导力(ID:jishulingdaoli) 前天的推文《冯唐:职场人35岁以后,方法论比经验重要》,收到了不少读者的反馈,觉得挺受启发。其实,冯唐写了不少关于职场方面的文章,都挺不错的。可惜大家只记住了“春风十里不如你”、“如何避免成为油腻腻的中年人”等不那么正经的文章。 本文整理了冯
程序员该看的几部电影
##1、骇客帝国(1999) 概念:在线/离线,递归,循环,矩阵等 剧情简介: 不久的将来,网络黑客尼奥对这个看似正常的现实世界产生了怀疑。 他结识了黑客崔妮蒂,并见到了黑客组织的首领墨菲斯。 墨菲斯告诉他,现实世界其实是由一个名叫“母体”的计算机人工智能系统控制,人们就像他们饲养的动物,没有自由和思想,而尼奥就是能够拯救人类的救世主。 可是,救赎之路从来都不会一帆风顺,到底哪里才是真实的世界?
作为一个程序员,CPU的这些硬核知识你必须会!
CPU对每个程序员来说,是个既熟悉又陌生的东西? 如果你只知道CPU是中央处理器的话,那可能对你并没有什么用,那么作为程序员的我们,必须要搞懂的就是CPU这家伙是如何运行的,尤其要搞懂它里面的寄存器是怎么一回事,因为这将让你从底层明白程序的运行机制。 随我一起,来好好认识下CPU这货吧 把CPU掰开来看 对于CPU来说,我们首先就要搞明白它是怎么回事,也就是它的内部构造,当然,CPU那么牛的一个东
@程序员,如何花式构建线程?
作者 |曾建责编 | 郭芮出品 | CSDN(ID:CSDNnews)在项目和业务的开发中,我们难免要经常使用线程来进行业务处理,使用线程可以保证我们的业务在相互处理之间可以保证原子性...
破14亿,Python分析我国存在哪些人口危机!
2020年1月17日,国家统计局发布了2019年国民经济报告,报告中指出我国人口突破14亿。 猪哥的朋友圈被14亿人口刷屏,但是很多人并没有看到我国复杂的人口问题:老龄化、男女比例失衡、生育率下降、人口红利下降等。 今天我们就来分析一下我们国家的人口数据吧! 一、背景 1.人口突破14亿 2020年1月17日,国家统计局发布了 2019年国民经济报告 ,报告中指出:年末中国大陆总人口(包括31个
强烈推荐10本程序员在家读的书
很遗憾,这个鼠年春节注定是刻骨铭心的,新型冠状病毒让每个人的神经都是紧绷的。那些处在武汉的白衣天使们,尤其值得我们的尊敬。而我们这些窝在家里的程序员,能不外出就不外出,就是对社会做出的最大的贡献。 有些读者私下问我,窝了几天,有点颓丧,能否推荐几本书在家里看看。我花了一天的时间,挑选了 10 本我最喜欢的书,你可以挑选感兴趣的来读一读。读书不仅可以平复恐惧的压力,还可以对未来充满希望,毕竟苦难终
Linux自学篇——linux命令英文全称及解释
man: Manual 意思是手册,可以用这个命令查询其他命令的用法。 pwd:Print working directory 显示当前目录 su:Swith user 切换用户,切换到root用户 cd:Change directory 切换目录 ls:List files 列出目录下的文件 ps:Process Status 进程状态 mk
Python实战:抓肺炎疫情实时数据,画2019-nCoV疫情地图
文章目录1. 前言2. 数据下载3. 数据处理4. 数据可视化 1. 前言 今天,群里白垩老师问如何用python画武汉肺炎疫情地图。白垩老师是研究海洋生态与地球生物的学者,国家重点实验室成员,于不惑之年学习python,实为我等学习楷模。先前我并没有关注武汉肺炎的具体数据,也没有画过类似的数据分布图。于是就拿了两个小时,专门研究了一下,遂成此文。 2月6日追记:本文发布后,腾讯的数据源多次变更u
智力题(程序员面试经典)
NO.1  有20瓶药丸,其中19瓶装有1克/粒的药丸,余下一瓶装有1.1克/粒的药丸。给你一台称重精准的天平,怎么找出比较重的那瓶药丸?天平只能用一次。 解法 有时候,严格的限制条件有可能反倒是解题的线索。在这个问题中,限制条件是天平只能用一次。 因为天平只能用一次,我们也得以知道一个有趣的事实:一次必须同时称很多药丸,其实更准确地说,是必须从19瓶拿出药丸进行称重。否则,如果跳过两瓶或更多瓶药
在家远程办公效率低?那你一定要收好这个「在家办公」神器!
相信大家都已经收到国务院延长春节假期的消息,接下来,在家远程办公可能将会持续一段时间。 但是问题来了。远程办公不是人在电脑前就当坐班了,相反,对于沟通效率,文件协作,以及信息安全都有着极高的要求。有着非常多的挑战,比如: 1在异地互相不见面的会议上,如何提高沟通效率? 2文件之间的来往反馈如何做到及时性?如何保证信息安全? 3如何规划安排每天工作,以及如何进行成果验收? ......
作为一个程序员,内存和磁盘的这些事情,你不得不知道啊!!!
截止目前,我已经分享了如下几篇文章: 一个程序在计算机中是如何运行的?超级干货!!! 作为一个程序员,CPU的这些硬核知识你必须会! 作为一个程序员,内存的这些硬核知识你必须懂! 这些知识可以说是我们之前都不太重视的基础知识,可能大家在上大学的时候都学习过了,但是嘞,当时由于老师讲解的没那么有趣,又加上这些知识本身就比较枯燥,所以嘞,大家当初几乎等于没学。 再说啦,学习这些,也看不出来有什么用啊!
2020年的1月,我辞掉了我的第一份工作
其实,这篇文章,我应该早点写的,毕竟现在已经2月份了。不过一些其它原因,或者是我的惰性、还有一些迷茫的念头,让自己迟迟没有试着写一点东西,记录下,或者说是总结下自己前3年的工作上的经历、学习的过程。 我自己知道的,在写自己的博客方面,我的文笔很一般,非技术类的文章不想去写;另外我又是一个还比较热衷于技术的人,而平常复杂一点的东西,如果想写文章写的清楚点,是需要足够...
别低估自己的直觉,也别高估自己的智商
所有群全部吵翻天,朋友圈全部沦陷,公众号疯狂转发。这两周没怎么发原创,只发新闻,可能有人注意到了。我不是懒,是文章写了却没发,因为大家的关注力始终在这次的疫情上面,发了也没人看。当然,我...
Java坑人面试题系列: 包装类(中级难度)
Java Magazine上面有一个专门坑人的面试题系列: https://blogs.oracle.com/javamagazine/quiz-2。 这些问题的设计宗旨,主要是测试面试者对Java语言的了解程度,而不是为了用弯弯绕绕的手段把面试者搞蒙。 如果你看过往期的问题,就会发现每一个都不简单。 这些试题模拟了认证考试中的一些难题。 而 “中级(intermediate)” 和 “高级(ad
Spring Boot 2.x基础教程:使用JdbcTemplate访问MySQL数据库
在第2章节中,我们介绍了如何通过Spring Boot来实现HTTP接口,以及围绕HTTP接口相关的单元测试、文档生成等实用技能。但是,这些内容还不足以帮助我们构建一个动态应用的服务端程序。不论我们是要做App、小程序、还是传统的Web站点,对于用户的信息、相关业务的内容,通常都需要对其进行存储,而不是像第2章节中那样,把用户信息存储在内存中(重启就丢了!)。 对于信息的存储,现在已经有非常非常多...
基于Python的人脸自动戴口罩系统
目录 1、项目背景 2、页面设计 3、器官识别 4、退出系统 1、项目背景 2019年新型冠状病毒感染的肺炎疫情发生以来,牵动人心,举国哀痛,口罩、酒精、消毒液奇货可居。 抢不到口罩,怎么办?作为技术人今天分享如何使用Python实现自动戴口罩系统,来安慰自己,系统效果如下所示: 本系统的实现原理是借助 Dlib模块的Landmark人脸68个关键点检测库轻松识别出人脸五官
这个世界上人真的分三六九等,你信吗?
偶然间,在知乎上看到一个问题 一时间,勾起了我深深的回忆。 以前在厂里打过两次工,做过家教,干过辅导班,做过中介。零下几度的晚上,贴过广告,满脸、满手地长冻疮。   再回首那段岁月,虽然苦,但让我学会了坚持和忍耐。让我明白了,在这个世界上,无论环境多么的恶劣,只要心存希望,星星之火,亦可燎原。   下文是原回答,希望能对你能有所启发。   如果我说,这个世界上人真的分三六九等,
相关热词 c#导入fbx c#中屏蔽键盘某个键 c#正态概率密度 c#和数据库登陆界面设计 c# 高斯消去法 c# codedom c#读取cad文件文本 c# 控制全局鼠标移动 c# temp 目录 bytes初始化 c#
立即提问