python3爬虫页面404

Question

用python3的requests做网络测试，发现有一个页面，headers不管怎么调，爬到的页面都是404
代码片段：

import requests
url = "https://www.luogu.com.cn/"
headers = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
    "Alt-Used": "www.luogu.com.cn",
    "Connection": "keep-alive",
    "Cookie": "__client_id=09c8dc7671a1aa7d2af3eedb956d7ba30bf9cb85; _uid=511454",
    "Host": "www.luogu.com.cn",
    "Referer": "https://www.luogu.com.cn/",
    "Sec-Fetch-Dest": "script",
    "Sec-Fetch-Mode": "no-cors",
    "Sec-Fetch-Site": "same-origin",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
}
response = requests.get(url, headers)
print(response.text)

仲么办？

Answer 1

用下面的方式可以请求到数据

from requests import Session

url = "https://www.luogu.com.cn/"
headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'zh-CN,zh;q=0.9',
    'cache-control': 'max-age=0',
    'cookie': '__client_id=09c8dc7671a1aa7d2af3eedb956d7ba30bf9cb85; _uid=511454',
    'sec-ch-ua': '"Google Chrome";v="111", "Not(A:Brand";v="8", "Chromium";v="111"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': "Windows",
    'sec-fetch-dest': 'document',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-site': 'none',
    'sec-fetch-user': '?1',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0'
}
session = Session()
response = session.get(url,headers=headers)
html = response.content.decode('utf-8')
print(html)