m0_52277682
2021-07-02 22:54
采纳率: 50%
浏览 22

想请教一下python爬虫中文乱码问题

小白最近在学爬虫,请问各位大哥们我想爬取的这句话“第一回·宴桃园豪杰三结义 斩黄巾英雄首立功”为啥是乱码呢
我用利用正则解析下面这个
第一回·宴桃园豪杰三结义 斩黄巾英雄首立功
代码如下,问题可能有些弱智,但恳请各位不吝赐教

import requests
import re

url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
headers = {"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"}
page_text = requests.get(url=url, headers=headers).text

ex = '<a href="/book/sanguoyanyi/1.html">(.*?)</a>'
text_data = re.findall(ex, page_text, re.S)
print(text_data)

这个是乱码情况:['第ä¸\x80å\x9b\x9e·宴æ¡\x83å\x9b\xad豪æ\x9d°ä¸\x89ç»\x93ä¹\x89 æ\x96©é»\x84å·¾è\x8b±é\x9b\x84é¦\x96ç«\x8bå\x8a\x9f']

  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

2条回答 默认 最新

  • 冰履踏青云 2021-07-02 23:09
    已采纳

    教你一招,请求数据之前先右键查看网页源代码,看看最上面的charset是啥,一般都是utf-8或者gbk的,要按它给的写程序,你这个网页是utf8编码,所以可以这样写,参考:

    import requests
    import re
    url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
    headers = {"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"}
    res = requests.get(url=url, headers=headers)
    res.encoding='utf-8'
    page_text = res.text
    ex = '<a href="/book/sanguoyanyi/1.html">(.*?)</a>'
    text_data = re.findall(ex, page_text, re.S)
    print(text_data)
    
    已采纳该答案
    1 打赏 评论
  • 获取
    requests.get(url=url, headers=headers)

    之后要设置编码
    res.encoding = "utf-8"

    再取.text

    代码如下:(如有帮助,望采纳!谢谢! 点击我这个回答右上方的【采纳】按钮)

    
    import requests
    import re
    url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
    headers = {"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"}
    res = requests.get(url=url, headers=headers)
    res.encoding = "utf-8"
    page_text = res.text
    ex = '<a href="/book/sanguoyanyi/1.html">(.*?)</a>'
    text_data = re.findall(ex, page_text, re.S)
    print(text_data)
    

    img

    1 打赏 评论