使用requests 获取博客园每页内容时，为什么一直重复爬取第一页内容？感谢大佬

Question

import requests
from bs4 import BeautifulSoup
urls = [
    f"https://cnblogs.com/#p{page}"  
    for page in range(1,5+1)
]
def parse(url):
    print(url)
    r = requests.get(url)
    r.encoding = 'uft-8'
    html= r.text
    soup = BeautifulSoup(html,'lxml')
    links = soup.find_all('a',class_ = 'post-item-title')
    return [
            (link.get('href'),link.text) 
                for link in links ]
def main():
    for url in urls:
        for result in parse(url):
            print(result)
        print('\n')
            
if __name__ == "__main__":
    main()

 

Answer 1

问题出在urls构造上，第一页和后续网页的地址是不一样的，将这行改为：urls = [f'https://cnblogs.com/#p{page}' if page == 1 else f'https://cnblogs.com/sitehome/p/{page}' for page in range(1, 6)]即可。