python爬虫有时出现keyerror问题

Question

报错信息如下：

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_49980\3608093107.py in <module>
      1 spider = BilibiliSpider()
      2 tv_total, reply_total = 1000* 1, 1  # 设置视频条数, 回复记录数
----> 3 spider.update_data_by_online("20221101", "20221231", tv_total, reply_total)
      4 spider.load_from_file()
      5 df = spider.data

~\AppData\Local\Temp\ipykernel_49980\3437160765.py in update_data_by_online(self, start_date, end_date, tv_total, reply_total, interval_days)
    199     b2 = time.time()
    200     for index, video in enumerate(video_list):
--> 201       self.__update_video_detail(video, reply_total)
    202       if (index + 1) % 100 == 0:
    203         print(f"进度: {index + 1} / {len(video_list)} = {(index + 1) / len(video_list) * 100:.2f}%, Cost Time : {time.time() - b2:.2f}")

~\AppData\Local\Temp\ipykernel_49980\3437160765.py in __update_video_detail(self, video, reply_total)
    239     data_item["up主粉丝量"] = upData["fans"]
    240     # 获取了所有的视频, 尝试更新评论等其他信息
--> 241     reply_list = self.__load_all_reply(detail["aid"], reply_total)
    242     if len(reply_list) == 0:
    243       reply_list = [{}]

~\AppData\Local\Temp\ipykernel_49980\3437160765.py in __load_all_reply(self, aid, reply_total)
    183     reply_list = []
    184     while True:
--> 185       is_end, replies = self.reply(aid, pn)
    186       reply_list += replies
    187       if len(reply_list) >= reply_total or is_end:

~\AppData\Local\Temp\ipykernel_49980\3437160765.py in reply(self, aid, pn)
    128     logger.debug(f"访问URL: {url}")
    129     data = requests.get(url, headers=headers, proxies=proxies).json()
--> 130     is_end = data[u"data"]["cursor"]["is_end"]
    131     data_list = []
    132     for reply in data[u"data"]["replies"]:

KeyError: 'data'

但是并不是每次运行都出问题的，爬取不同时间段，有的可以正常运行，有的又会出现相关报错
之前是没有u的，我加了u发现代码与报错都不受影响

Answer 1

不知道你这个问题是否已经解决, 如果还没有解决的话:

• 帮你找了个相似的问题, 你可以看下: https://ask.csdn.net/questions/7423349
• 这篇博客也不错, 你可以看下python keyerror解决方案
• 除此之外, 这篇博客: Python的KeyError的解决办法中的 解决办法 部分也许能够解决你的问题, 你可以仔细阅读以下内容或者直接跳转源博客中阅读:
  

  2.1 方法1通过in判断key是否存在

  my = {"nick":"福哥","age":"35","languages":["c/c++","java","php","python","javascript"]}
  
  if "province" in my:
      print(my["province"])
  

  2.2 方法2使用get方法判断key是否存在

  my = {"nick":"福哥","age":"35","languages":["c/c++","java","php","python","javascript"]}
  
  if my.get("province") is not None:
      print(my["province"])
  

  2.3 方法3使用get方法的第二个参数，如果目标key不存在则返回第二个参数

  my = {"nick":"福哥","age":"35","languages":["c/c++","java","php","python","javascript"]}
  
  print(my.get("province", "北京"))
  print(my)
  

  2.4 方法4使用setdefault方法的第二个参数，如果目标key不存在则返回第二个参数同时讲第二个参数写入key位置

  my = {"nick":"福哥","age":"35","languages":["c/c++","java","php","python","javascript"]}
  
  print(my.setdefault("province", "北京"))
  print(my)
  

如果你已经解决了该问题, 非常希望你能够分享一下解决方案, 写成博客, 将相关链接放在评论区, 以帮助更多的人 ^-^