Users should be able to set the timeout and the chunk_size for elastic search bulk requests.

Question

While dumping huge data to elasticsearch, the mongo-connector can't work normally because it often crashes due to connection timeout. The default timeout is 10, and can't be changed. It should be an option of the mongo-connector command so that user can change it when necessary.

2015-10-20 01:48:04,992 [CRITICAL] mongo_connector.oplog_manager:543 - Exception during collection dump
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/mongo_connector/oplog_manager.py", line 495, in do_dump
    upsert_all(dm)
  File "/usr/lib/python2.6/site-packages/mongo_connector/oplog_manager.py", line 479, in upsert_all
    dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)
  File "/usr/lib/python2.6/site-packages/mongo_connector/util.py", line 32, in wrapped
    return f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/mongo_connector/doc_managers/elastic_doc_manager.py", line 190, in bulk_upsert
    for ok, resp in responses:
  File "/usr/lib/python2.6/site-packages/elasticsearch/helpers/__init__.py", line 138, in streaming_bulk
    raise e
ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host=u'172.31.1.254', port=9200): Read timed out. (read timeout=10))

Thanks,

该提问来源于开源项目：yougov/mongo-connector

Answer 1

Please note that the line number of the python file in the log above may be incorrect because I modified the python to add more log locally.