python scrapy爬虫如果想要下一页但是没有href

Question

scrapy 想实现爬取下一页的功能，但是”下一页“按钮并没有指向一个href。想请教如何实现爬取到下一页的功能
网址：https://chl.cn/?lishi

<input value="4" name="page" type="hidden" /> <input value="" name="str" type="hidden" /> <input value="1.." type="submit" class="pg" /> <input value="上一页" type="submit" class="pg" /> <input value="4/148" type="button" class="pg pg0" /> <input value="下一页" type="submit" class="pg" />

Answer 1

你要模拟参数，具体代码如下：

import http.client

conn = http.client.HTTPSConnection("chl.cn")
#page 5
#submit 下一页
payload = 'page=5&submit=%E4%B8%8B%E4%B8%80%E9%A1%B5&str='
headers = {
   'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
   'Host': 'chl.cn',
   'Connection': 'keep-alive',
   'Content-Type': 'application/x-www-form-urlencoded'
}
conn.request("GET", "/?lishi", payload, headers)
res = conn.getresponse()
data = res.read()
print(data)
print(data.decode("utf-8"))