scrapy刚入门小白一枚。用网上的案例代码来玩一玩，案例是http://blog.csdn.net/czl389/article/details/77278166 中的爬取嘻哈歌词。这个案例下有三只爬虫，分别是songurls，lyrics和songinfo。我用songurls爬虫能从虾米音乐上爬取了url并保存在SongUrls.csv中，但是在用lyrics爬虫的时候会报错。信息如下
**D:\xiami2\xiami2>scrapy crawl lyrics -o Lyrics.csv
2017-10-21 21:13:29 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: xiami2)
2017-10-21 21:13:29 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'xiami2.spiders', 'USER_AGENT': 'Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 4.0; Trident/3.0)', 'FEED_URI': 'Lyrics.csv', 'FEED_FORMAT': 'csv', 'DOWNLOAD_DELAY': 0.2, 'SPIDER_MODULES': ['xiami2.spiders'], 'BOT_NAME': 'xiami2'}
2017-10-21 21:13:29 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2017-10-21 21:13:31 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-10-21 21:13:31 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-10-21 21:13:31 [scrapy.middleware] INFO: Enabled item pipelines:
['xiami2.pipelines.Xiami2Pipeline']
2017-10-21 21:13:31 [scrapy.core.engine] INFO: Spider opened
2017-10-21 21:13:31 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-10-21 21:13:31 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-10-21 21:13:31 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "d:\python3.5\lib\site-packages\scrapy\core\engine.py", line 127, in next_request
request = next(slot.start_requests)
File "d:\python3.5\lib\site-packages\scrapy\spiders\_init__.py", line 83, in start_requests
yield Request(url, dont_filter=True)
File "d:\python3.5\lib\site-packages\scrapy\http\request__init__.py", line 25, in init
self._set_url(url)
File "d:\python3.5\lib\site-packages\scrapy\http\request__init__.py", line 58, in set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url:
2017-10-21 21:13:31 [scrapy.core.engine] INFO: Closing spider (finished)
2017-10-21 21:13:31 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 10, 21, 13, 13, 31, 567323),
'log_count/DEBUG': 1,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'start_time': datetime.datetime(2017, 10, 21, 13, 13, 31, 536236)}
2017-10-21 21:13:31 [scrapy.core.engine] INFO: Spider closed (finished)
_------------------------------分割线--------------------------------------
我去查看了一下_init_.py，发现如下语句。
if ':' not in self._url:
raise ValueError('Missing scheme in request url: %s' % self._url)
网上的解决方法看了一些，都没有能解决我的问题的，因此在此讨教，望大家指点一二(真没C币了)。提问次数不多，若有格式方面缺陷还请包含。
另附上代码。
#songurls.py
import scrapy
import re
from scrapy.spiders import CrawlSpider, Rule
from ..items import SongUrlItem
class SongurlsSpider(scrapy.Spider):
name = 'songurls'
allowed_domains = ['xiami.com']
#将page/1到page/401，这些链接放进start_urls
start_url_list=[]
url_fixed='http://www.xiami.com/song/tag/Hip-Hop/page/'
#将range范围扩大为1-401，获得所有页面
for i in range(1,402):
start_url_list.extend([url_fixed+str(i)])
start_urls=start_url_list
def parse(self,response):
urls=response.xpath('//*[@id="wrapper"]/div[2]/div/div/div[2]/table/tbody/tr/td[2]/a[1]/@href').extract()
for url in urls:
song_url=response.urljoin(url)
url_item=SongUrlItem()
url_item['song_url']=song_url
yield url_item
------------------------------分割线--------------------------------------
#lyrics.py
import scrapy
import re
class LyricsSpider(scrapy.Spider):
name = 'lyrics'
allowed_domains = ['xiami.com']
song_url_file='SongUrls.csv'
def __init__(self, *args, **kwargs):
#从song_url.csv 文件中读取得到所有歌曲url
f = open(self.song_url_file,"r")
lines = f.readlines()
#这里line[:-1]的含义是每行末尾都是一个换行符，要去掉
#这里in lines[1:]的含义是csv第一行是字段名称，要去掉
song_url_list=[line[:-1] for line in lines[1:]]
f.close()
while '\n' in song_url_list:
song_url_list.remove('\n')
self.start_urls = song_url_list#[:100]#删除[:100]之后爬取全部数据
def parse(self,response):
lyric_lines=response.xpath('//*[@id="lrc"]/div[1]/text()').extract()
lyric=''
for lyric_line in lyric_lines:
lyric+=lyric_line
#print lyric
lyricItem=LyricItem()
lyricItem['lyric']=lyric
lyricItem['song_url']=response.url
yield lyricItem
songinfo因为还没有用到所以不重要。
------------------------------分割线--------------------------------------
#items.py
import scrapy
class SongUrlItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
song_url=scrapy.Field() #歌曲链接
class LyricItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
lyric=scrapy.Field() #歌词
song_url=scrapy.Field() #歌曲链接
class SongInfoItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
song_url=scrapy.Field() #歌曲链接
song_title=scrapy.Field() #歌名
album=scrapy.Field() #专辑
#singer=scrapy.Field() #歌手
language=scrapy.Field() #语种
------------------------------分割线--------------------------------------
在middleware下加了几行：
sleep_seconds = 0.2 # 模拟点击后休眠3秒，给出浏览器取得响应内容的时间
default_sleep_seconds = 1 # 无动作请求休眠的时间
def process_request(self, request, spider):
spider.logger.info('--------Spider request processed: %s' % spider.name)
page = None
driver = webdriver.PhantomJS()
spider.logger.info('--------request.url: %s' % request.url)
driver.get(request.url)
driver.implicitly_wait(0.2)
# 仅休眠数秒加载页面后返回内容
time.sleep(self.sleep_seconds)
page = driver.page_source
driver.close()
return HtmlResponse(request.url, body=page, encoding='utf-8', request=request)
------------------------------分割线--------------------------------------
setting中加了几行也改了几行：
from faker import Factory
f = Factory.create()
USER_AGENT = f.user_agent()
DOWNLOAD_DELAY = 0.2
DEFAULT_REQUEST_HEADERS = {
'Host': 'www.xiami.com',
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'zh-CN,zh;q=0.8',
'Cache-Control': 'no-cache',
'Connection': 'Keep-Alive',
}
ITEM_PIPELINES = {
'xiami2.pipelines.Xiami2Pipeline': 300,
}
