努力才能被爱慕 2018-03-25 07:07 采纳率: 100%
浏览 5498
已采纳

Python爬虫、requests库的get方法,带上headers后出现编码错误

目标网站:知乎
Pyrhon版本:3.6


agent = 'Mozilla/5.0 (Windows NT 6.1; W…) Gecko/20100101 Firefox/59.0'
header = {
    'HOST': 'www.zhihu.com',
    'Referer': 'https://www.zhihu.com',
    'User-Agent': agent
}

response = requests.get('https://www.zhihu.com', headers=header)

报错内容:

 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\lib\site-packages\requests\api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Anaconda3\lib\site-packages\requests\api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Anaconda3\lib\site-packages\requests\sessions.py", line 518, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Anaconda3\lib\site-packages\requests\sessions.py", line 639, in send
    r = adapter.send(request, **kwargs)
  File "C:\Anaconda3\lib\site-packages\requests\adapters.py", line 438, in send
    timeout=timeout
  File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 356, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Anaconda3\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Anaconda3\lib\http\client.py", line 1280, in _send_request
    self.putheader(hdr, value)
  File "C:\Anaconda3\lib\http\client.py", line 1212, in putheader
    values[i] = one_value.encode('latin-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026' in position 30: ordinal not in range(256)

\u2026好像是空格,看了下源码,好像是空格要转latin-1这种编码失败了,
我上面有空格的是agent,要用来伪装浏览器,这可怎么办,
而且在我看的那个知乎爬虫视频中,他也是有空格的,却不出错,这是我的环境有问题吗,
请大佬们能指点一下

  • 写回答

4条回答

  • 不知了先生 2018-03-25 10:29
    关注

    头有问题啊!'Mozilla/5.0 (Windows NT 6.1; W…) Gecko/20100101 Firefox/59.0'不知道怎么会出现...这种符号!自己去复制浏览器的user-agent

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(3条)

报告相同问题?

悬赏问题

  • ¥15 孟德尔随机化结果不一致
  • ¥15 apm2.8飞控罗盘bad health,加速度计校准失败
  • ¥15 求解O-S方程的特征值问题给出边界层布拉休斯平行流的中性曲线
  • ¥15 谁有desed数据集呀
  • ¥20 手写数字识别运行c仿真时,程序报错错误代码sim211-100
  • ¥15 关于#hadoop#的问题
  • ¥15 (标签-Python|关键词-socket)
  • ¥15 keil里为什么main.c定义的函数在it.c调用不了
  • ¥50 切换TabTip键盘的输入法
  • ¥15 可否在不同线程中调用封装数据库操作的类