在pycharm中get到网页的源代码和右键查看的源代码不一样,在爬取拉钩网发现了这个问题,代码如下,应该没问题,但是和网页的源代码出入很大,不知道是怎么回事
import requests
from concurrent.futures import ThreadPoolExecutor
def download_one_page(url):
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/97.0.4692.99 Safari/537.36 Edg/97.0.1072.69"
}
resp = requests.post(url, headers=headers)
resp.encoding = "utf-8"
print(resp.text)
if __name__ == "__main__":
with ThreadPoolExecutor(50) as t:
t.submit(download_one_page, f"https://www.lagou.com/zhaopin/")