怎么实现简单的python翻页并且保存数据？

Question

import requests
from lxml import etree
url='https://top.chinaz.com/hangye/index_yule_xiaoshuo.html'
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36'}
rqg=requests.get(url=url,headers=headers)
rqg_text=rqg.text
rqg.encoding='utf8'
html=etree.HTML(rqg_text)
node_list=html.xpath('//*[@id="content"]/div[4]/div[3]/div[2]')
alexa=[]
fls=[]
for node in node_list:
    name=node.xpath('./ul/li/div[2]/h3/a/text()')
    website=node.xpath('./ul/li/div[2]/h3/span/text()')
    alexa1=node.xpath('./ul/li/div[2]/div/p[1]/span/text()')
    for i in range(len(alexa1)):
        alexa12=node.xpath('./ul/li/div[2]/div/p[1]/a/text()')
        for i in range(len(alexa12)):
            alexa.append(alexa1[i]+alexa12[i])
    fls1=node.xpath('./ul/li/div[2]/div/p[4]/span/text()')
    for i in range(len(fls1)):
        fls2=node.xpath('./ul/li/div[2]/div/p[4]/a/text()')
        for i in range(len(fls2)):
            fls.append(fls1[i]+fls2[i])
    pm=node.xpath('./ul/li/div[3]/div/strong/text()')
    score=node.xpath('./ul/li/div[3]/div/span/text()')
    text=node.xpath('./ul/li/div[2]/p/text()')
print(name)

Answer 1

实现翻页可以有2种:

第1种

• 找网页的规律, 可以通过如下方法构造你需要爬取的网址

lis1 = ['' if i ==1 else '_' +str(i) for i in range(1,10)]

url = ['https://top.chinaz.com/hangye/index_yule_xiaoshuo{}.html'.format(i) for i in lis1]

• 通过for循环爬取每个网页的信息
• 保存数据创建一个空的dataframe, 通过标签节点获取数据,添加到dataframe中

第2种 

• 用selenium 模拟浏览器, 点击翻页

 

简单的爬虫我也习惯用for循环, 这个你可以参考下:

https://blog.csdn.net/muyashui/article/details/114298666