非常简单的爬虫,静态网站没问题,但是在动态加载的网站中,请求每次超市报错。如下:
2020-08-03 13:19:45 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying (failed 1 times): 504 Gateway Time-out
代码如下
import scrapy
import time
import json
from movies.items import MoviesItem
class YiqingSpider(scrapy.Spider):
name = 'yiqing'
allowed_domains = ['view.inews.qq.com']
start_urls = ['https://view.inews.qq.com/g2/getOnsInfo?name=disease_h5']
def parse(self, response):
text=response.text
print(text)