weixin_40825517
罗家马德里球迷
采纳率22.2%
2018-11-25 09:05 阅读 2.3k

请问我的Python爬虫代码 哪里出现了问题?(要求:爬取猫眼电影top100榜单的信息)

代码如下:

 import requests
from requests.exceptions import RequestException
import time
from bs4 import BeautifulSoup

def get_one_page(url):
    try:
        headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}
        response = requests.get(url,headers=headers)
        if response.status_code==200:
            return response.text
        return None
    except RequestException:
        return None

def page(offset):
    url='http://maoyan.com/board/6?offset='+str(offset)
    return url

for j in range(10):
    html_doc = get_one_page(page(j*10))
    soup = BeautifulSoup(html_doc,'lxml')
    i = 1
    for dd in soup.select("dd"):
        print(dd.find("i","board-index board-index-"+str(i+j*10)).get_text()
              +dd.find("p","name").get_text()
              +dd.find("p","star").get_text().strip()
              +dd.find("p","releasetime").string
              +dd.find("p","score").get_text()+'\n')
        i = i + 1
    time.sleep(1)

运行反馈结果为:

 Traceback (most recent call last):

  File "<ipython-input-8-95f75b1c7bd0>", line 1, in <module>
    runfile('H:/程序语言学习用文件夹/Spider/beautifulSoup.py', wdir='H:/程序语言学习用文件夹/Spider')

  File "C:\Users\pc1\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "C:\Users\pc1\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "H:/程序语言学习用文件夹/Spider/beautifulSoup.py", line 29, in <module>
    soup = BeautifulSoup(html_doc,'lxml')

  File "C:\Users\pc1\Anaconda3\lib\site-packages\bs4\__init__.py", line 192, in __init__
    elif len(markup) <= 256 and (

TypeError: object of type 'NoneType' has no len()
  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

1条回答 默认 最新

  • weixin_39416561 lyhsdy 2018-11-26 03:05

    我这里测试没有出现你这边的错误,应该是'lxml'没有安装好,试下print(soup)的结果看看是什么,或者换成
    soup= BeautifulSoup(html_doc,'html.parser')

    我这边测试的dd.find("p","score").get_text()出错,没找到对应 的值,建议查看下值的获取方式,或者删掉

    点赞 评论 复制链接分享

相关推荐