刚学习用scrapy + selenium爬动态网页,但是不知道为什么就是不行,下面是代码,求大神指点!!!

MySpider里面是这样的:

class MySpider(scrapy.Spider):
    name = 'BAIScrapy'

    def start_requests(self):
        print('开始')
        url = 'https://www.bilibili.com/'
        request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
        request.meta['PhantomJS'] = True
        yield request

    def parse(self, response):
        print('Emmm...')
        item = BilibiliAnimeInfoScrapyItem()
        item['links'] = response.css('a::attr("href")').re("www.bilibili.com/bangumi/play/")

middlewares里面是这样的:

    def process_reqeust(self, request, spider):
        print('进入selenium')
        driver = webdriver.PhantomJS()
        driver.get(request.url)
        element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID,'bili_bangumi')))
        driver.quit()
        yield HtmlResponse(url=request.url, encoding='utf-8', body=driver.page_source, request=request)

settings里面是这样的:

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.163 Safari/535.1'


ROBOTSTXT_OBEY = False

DOWNLOADER_MIDDLEWARES = {
    'bilibili_anime_info_scrapy.middlewares.BilibiliAnimeInfoScrapyDownloaderMiddleware': 543,
}

1个回答

sinat_38677939
sinat_38677939 感谢大佬!!!
一年多之前 回复
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问