爬虫实战-豆瓣Top250爬取实战 ip被禁

Question

问题遇到的现象和发生背景

实战练习导致IP被禁，尝试代理ip无法解决

问题相关代码，请勿粘贴截图 #使用代理

    proxies = [{'https':'115.75.5.17:38351'},{'https':'14.215.212.37:9168'}]
    proxy = random.choice(proxies)
    print(proxy)
    resp = requests.get(url, headers=headers,proxies=proxy)

运行结果及报错内容

{'https': '115.75.5.17:38351'}
Traceback (most recent call last):
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
    raise err
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    sock.connect(sa)
TimeoutError: [WinError 10060] 由于连接方在一段时间后没有正确答复或连接的主机没有反应，连接尝试失败。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 994, in _prepare_proxy
    conn.connect()
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 179, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x000002712D2D5A20>, 'Connection to 115.75.5.17 timed out. (connect timeout=None)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 440, in send
    resp = conn.urlopen(
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='movie.douban.com', port=443): Max retries exceeded with url: /top250?start=0&filter= (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000002712D2D5A20>, 'Connection to 115.75.5.17 timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\Users\18081\PycharmProjects\数据解析\实战演练——爬取豆瓣TOP250.py", line 66, in <module>
    main()
  File "E:\Users\18081\PycharmProjects\数据解析\实战演练——爬取豆瓣TOP250.py", line 61, in main
    detail_urls = get_detail_uel(url)
  File "E:\Users\18081\PycharmProjects\数据解析\实战演练——爬取豆瓣TOP250.py", line 15, in get_detail_uel
    resp = requests.get(url, headers=headers,proxies=proxy)#headers=headers)
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\18081\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='movie.douban.com', port=443): Max retries exceeded with url: /top250?start=0&filter= (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000002712D2D5A20>, 'Connection to 115.75.5.17 timed out. (connect timeout=None)'))

我的解答思路和尝试过的方法

使用开放代理ip的方法，但是一直报错

我想要达到的结果

使用代理IP进行实战课程演练

Answer 1

随便找的免费代理ip质量不行，十个未必能有一个有用的，需要花钱买