weixin_39925350
weixin_39925350
2020-12-08 18:05

Catalyst should not crash when timouts occur

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

  • Operating System: Linux trading-bot 4.13.0-39-generic #44-Ubuntu SMP Thu Apr 5 14:25:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Python Version: Python 3.6.4 :: Anaconda, Inc.
  • Python Bitness: 64
  • How did you install Catalyst: conda
  • Python packages: aiodns 1.1.1
    aiohttp 3.1.3
    alembic 0.9.9
    async-timeout 2.0.1
    attrdict 2.0.0
    attrs 17.4.0
    bcolz 0.12.1
    bcolz 1.1.2 py36h00f5784_0
    bokeh 0.12.14 py36_0
    boto3 1.7.4
    botocore 1.10.4
    Bottleneck 1.2.1
    bzip2 1.0.6 h9a117a8_4
    ca-certificates 2017.08.26 h1d4fec5_0
    cchardet 2.1.1
    ccxt 1.13.1 certifi 2018.1.18 py36_0
    certifi 2018.4.16
    chardet 3.0.4
    click 6.7 py36h5253387_0
    cloudpickle 0.5.2 py36_1
    configparser 3.5.0
    contextlib2 0.5.5
    cycler 0.10.0
    cyordereddict 1.0.0
    Cython 0.28.2
    cython 0.27.3 py36h1860423_0
    cytoolz 0.9.0.1 py36h14c3975_0
    dask 0.17.2 py36_0
    dask-core 0.17.2 py36_0
    decorator 4.3.0
    distributed 1.21.4 py36_0
    docutils 0.14
    empyrical 0.2.1
    enigma-catalyst 0.5.9
    eth-abi 1.0.0
    eth-account 0.1.0a2
    eth-hash 0.1.2
    eth-keyfile 0.5.1
    eth-keys 0.2.0b3
    eth-rlp 0.1.0
    eth-utils 1.0.2
    hdf5 1.10.1 h9caa474_1
    heapdict 1.0.0 py36_2
    hexbytes 0.1.0
    idna 2.6
    idna-ssl 1.0.1
    intel-openmp 2018.0.0 8
    intervaltree 2.1.0
    jinja2 2.10 py36ha16c418_0
    jmespath 0.9.3
    kiwisolver 1.0.1
    libedit 3.1 heed3624_0
    libffi 3.2.1 hd88cf55_4 libgcc-ng 7.2.0 hdf63c60_3 [38/577] libgfortran-ng 7.2.0 hdf63c60_3
    libstdcxx-ng 7.2.0 hdf63c60_3
    locket 0.2.0 py36h787c0ad_1
    Logbook 1.3.3
    lru-dict 1.1.6
    lxml 4.2.1
    lzo 2.10 h49e0be7_2
    Mako 1.0.7
    markupsafe 1.0 py36hd9260cd_1
    matplotlib 2.2.2
    mkl 2018.0.2 1
    mpld3 0.3
    msgpack-python 0.5.5 py36h6bb024c_0
    multidict 4.2.0
    multipledispatch 0.5.0
    ncurses 6.0 h9df7e31_2
    networkx 2.1
    numexpr 2.6.4 py36hc4a3f9a_0
    numpy 1.14.2
    numpy 1.14.2 py36hdbf6ddf_0
    openssl 1.0.2n hb7f436b_0
    packaging 17.1 py36_0
    pandas 0.22.0 py36hf484d3e_0
    pandas 0.19.2
    pandas-datareader 0.6.0
    partd 0.3.8 py36h36fd896_0
    patsy 0.5.0
    patsy 0.5.0 py36_0
    pip 10.0.0
    pip 9.0.1 py36_5
    psutil 5.4.3 py36h14c3975_0
    pycares 2.3.0
    pycryptodome 3.6.1
    pyparsing 2.2.0 py36hee85983_1
    pytables 3.4.2 py36h3b5282a_2
    python 3.6.4 hc3d631a_3
    python-dateutil 2.7.2 python-dateutil 2.7.0 py36_0
    python-editor 1.0.3 pytz 2018.3 py36_0
    pytz 2018.4 pyyaml 3.12 py36hafb9ca4_1
    readline 7.0 ha6073c6_4
    redo 1.6 requests 2.18.4 requests-file 1.4.3 requests-ftp 0.3.1 requests-toolbelt 0.8.0 rlp 0.6.0 s3transfer 0.1.13 scipy 1.0.0 py36hbf646e7_0
    scipy 1.0.1 setuptools 38.5.1 py36_0
    setuptools 39.0.1 six 1.10.0 six 1.11.0 py36h372c433_1
    sortedcontainers 1.5.9 py36_0
    SQLAlchemy 1.2.6 sqlite 3.22.0 h1bed415_0
    statsmodels 0.8.0 statsmodels 0.8.0 py36h8533d0b_0
    tabulate 0.8.2 tblib 1.3.2 py36h34cf8b6_0
    tk 8.6.7 hc745277_3
    toolz 0.9.0 py36_0
    tornado 5.0 py36_0
    urllib3 1.22 web3 4.1.0 websockets 4.0.1 wheel 0.30.0 py36hfd4bba0_1
    wrapt 1.10.11 xz 5.2.3 h55aa19d_2
    yaml 0.1.7 had09818_2
    yarl 1.1.0 zict 0.1.3 py36h3a3bf81_0
    zlib 1.2.11 ha838bed_2

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

There was a timeout while Catalyst was trying to fetch the balances on Poloniex. Poloniex is apparently under maintenance right now. This caused the algorithm to crash.


[2018-05-06 11:37:02.432304] WARNING: CCXT: unable to fetch balance poloniex: poloniex {"error":"Internal error. Please try again."}        
[2018-05-06 11:37:11.389414] WARNING: CCXT: unable to fetch balance poloniex: poloniex {"error":"Internal error. Please try again."}                           
[2018-05-06 11:37:28.910313] WARNING: CCXT: unable to fetch balance poloniex: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.
com', port=443): Read timed out. (read timeout=10)
[2018-05-06 11:37:49.242851] WARNING: CCXT: unable to fetch balance poloniex: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.
com', port=443): Read timed out. (read timeout=10)
[2018-05-06 11:38:17.206662] WARNING: CCXT: unable to fetch balance poloniex: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.
com', port=443): Read timed out. (read timeout=10)

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='poloniex.com', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 356, in fetch
    proxies=self.proxies
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='poloniex.com', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 628, in get_balances
    balances = self.api.fetch_balance()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/poloniex.py", line 227, in fetch_balance
    }, params))
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 305, in request
    return self.fetch2(path, api, method, params, headers, body)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 302, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 366, in fetch
    self.raise_error(RequestTimeout, method, url, e)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 287, in raise_error
    raise exception_type(output)
ccxt.base.errors.RequestTimeout: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.com', port=443): Read timed out. (read timeou
t=10)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "algo.py", line 855, in <module>
    simulate_orders = PAPER_TRADE,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 584, in run_algorithm
    stats_output=stats_output
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 348, in _run
    overwrite_sim_params=False,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 1044, in run
    data, overwrite_sim_params
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 358, in run
    data, overwrite_sim_params
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py", line 724, in run
    for perf in self.get_generator():
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 224, in transform
    for capital_change_packet in every_bar(dt):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 216, in handle_data
    dt,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 235, in handle_data
    self.callback(context, data)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 890, in handle_data
    cleanup=lambda: log.warn('Ordering again.')
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/redo/__init__.py", line 162, in retry
    return action(*args, **kwargs)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 713, in synchronize_portfolio
    cash=required_cash,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange.py", line 702, in sync_positions
    balances = self.get_balances()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 640, in get_balances
    raise ExchangeRequestError(error=e)
catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.com',
port=443): Read timed out. (read timeout=10)
</module></string>

I feel that in the case of a timeout, Catalyst should not crash, but rather just continue trying to connect while providing warnings, maybe with a decreasing frequency, until the problem disappears. You don't want a trading algorithm to just crash, leaving positions open.

I had similar issues when fetching price data, which I solved by explicitly catching the exceptions and taking appropriate action in my algorithm. But the problem above lie outside the scope of the algorithm.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. Wait for a timeout... Or maybe cut off your internet connection for a while.

Sincerely, Samuel

该提问来源于开源项目:enigmampc/catalyst

  • 点赞
  • 回答
  • 收藏
  • 复制链接分享

4条回答

为你推荐

换一换