想给家里老人爬取些广场舞的视频，遇到问题，求大神指导

Question

打算先爬取视频内容合集页大概100左右数量的单个视频页的地址，再转到单个视频页爬取视频，但是返回的结果是空值，试过其他几种定位方式，都不行

import requests

from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
    'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding' : 'gzip',
    'DNT' : '1', 
    'Connection' : 'close',
    'Accept-Language':'zh-CN'
}
f
url_address = 'https://haokan.baidu.com/author/1622522456467855'  
item_collection_responde = requests.get(url = url_address, headers = headers)
item_collection_soup = BeautifulSoup(item_collection_responde.content,  'lxml')
item_collection =  item_collection_soup.find_all('a', class_= 'card-item-link skeleton')

print(item_collection)

也试过第二步，直接在单个视频页抓取 <video>标签里的下载地址，也是返回空值

Answer 1

我又研究了一下，发现可以直接爬取好看视频网站的json数据

 

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
    'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding' : 'gzip',
    'DNT' : '1',
    'Connection' : 'close',
    'Accept-Language':'zh-CN'
}

url_address = 'https://haokan.baidu.com/author/1622522456467855'

ctime = ""
while True:
    resjson = requests.get(url = f'{url_address}?_format=json&rn=16&ctime={ctime}&_api=1', headers = headers).json()
    response = resjson['data']['response']
    for it in response['results']:
        print("视频名称：",it['content']['title'])
        print("标清视频地址：",it['content']['video_list'].get('sd','无'))
        print("高清视频地址：",it['content']['video_list'].get('hd','无'))
        print("超清视频地址：",it['content']['video_list'].get('sc','无'))
        print()
    if response['has_more'] == 0:
        print("爬取结束")
        break
    ctime = response['ctime']