elasticsearch中的bulk使用问题

使用es的bulk往数据库中添加1万条数据,
其中9千条插入成功,1千条插入失败,
那么失败的那1千条应该怎么处理呢?

1个回答

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
Elasticsearch PHP批量索引性能与索引

<div class="post-text" itemprop="text"> <p>I run a benchmark on elasticsearch using elasticsearch-php. I compare the time taken by 10 000 index one by one vs 10 000 with bulk of 1 000 documents.</p> <p>On my vpn server 3 cores 2 Gb mem the performance is quite the same with or without bulk index. </p> <p>My php code (inspired by à post): </p> <pre><code>&lt;?php set_time_limit(0); // no timeout require 'vendor/autoload.php'; $es = new Elasticsearch\Client([ 'hosts'=&gt;['127.0.0.1:9200'] ]); $max = 10000; // ELASTICSEARCH BULK INDEX $temps_debut = microtime(true); for ($i = 0; $i &lt;= $max; $i++) { $params['body'][] = array( 'index' =&gt; array( '_index' =&gt; 'articles', '_type' =&gt; 'article', '_id' =&gt; 'cle' . $i ) ); $params['body'][] = array( 'my_field' =&gt; 'my_value' . $i ); if ($i % 1000) { // Every 1000 documents stop and send the bulk request $responses = $es-&gt;bulk($params); $params = array(); // erase the old bulk request unset($responses); // unset to save memory } } $temps_fin = microtime(true); echo 'Elasticsearch bulk: ' . round($i / round($temps_fin - $temps_debut, 4)) . ' per sec &lt;br&gt;'; // ELASTICSEARCH WITHOUT BULK INDEX $temps_debut = microtime(true); for ($i = 1; $i &lt;= $max; $i++) { $params = array(); $params['index'] = 'my_index'; $params['type'] = 'my_type'; $params['id'] = "key".$i; $params['body'] = array('testField' =&gt; 'valeur'.$i); $ret = $es-&gt;index($params); } $temps_fin = microtime(true); echo 'Elasticsearch One by one : ' . round($i / round($temps_fin - $temps_debut, 4)) . 'per sec &lt;br&gt;'; ?&gt; </code></pre> <p>Elasticsearch bulk: 1209 per sec Elasticsearch One by one : 1197per sec </p> <p>Is there something wrong on my bulk index to obtain better performance ?</p> <p>Thank's</p> </div>

如何使用bulkDelete实现多次删除

<div class="post-text" itemprop="text"> <p>I have array of Profile Ids (uid) and need to delete all these profiles with 1 request.</p> <p>Here is my code.</p> <pre><code>func MultipleDeleteFromElastic(index string, inType string, uid string, ct interface{}) error { client, err := GetElasticCon() if err != nil { ElasticConnectError.DeveloperMessage = err.Error() return ElasticConnectError } deleteReq := elastic.NewBulkDeleteRequest().Index(index).Type(inType).Id(uid) _, err1 := client.Bulk().Add(deleteReq).Do(context.Background()) if err1 != nil { ElasticConnectError.DeveloperMessage = err1.Error() return ElasticConnectError } return err1 } </code></pre> <p>What does the bulkDelete need? How I can pass an array in BulkDelete?</p> <p>I have no idea if I doing this right (obviously I am not).</p> </div>

elasticsearch超时问题

elasticsearch scroll Java 中设置的超时时间是针对查询出来的结果数据,还是结果数据和scrollId超过时间都会失效

spark 写入elasticsearch报错Could not write all entries

我在使用Spark将Rdd写入到elasticsearch集群的时候报出异常 ``` Could not write all entries [199/161664] (maybe ES was overloaded?). Bailing out... at org.elasticsearch.hadoop.rest.RestRepository.flush(RestRepository.java:250) at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:201) at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:163) at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:49) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:84) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:84) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ``` RDD大概是5000W行数据,es集群有两个节点 ``` EsSpark.saveToEs(result, "userindex/users", Map("es.mapping.id" -> "uid")) ```

Go中Elasticsearch的大量更新

<div class="post-text" itemprop="text"> <p>I try to update my database of elasticsearch with golang.I have two functions :</p> <pre><code>func UpdateAllByUserID(client *elastic.Client, id string, requestName string, requestNick string) error { ctx := context.Background() query := elastic.NewMatchQuery("user_id", id) out_name, err := client.UpdateByQuery().Index("test").Type("test").Query(query).Script(elastic.NewScriptInline("ctx._source.user_name = '" + requestName + "'")).Do(ctx) if nil != err { log.Println(err) } fmt.Println("update all name: ", out_name.Updated) return nil } func UpdateAllNicksByUserIdInFeed(client *elastic.Client, id string, requestNick string) error { ctx := context.Background() query := elastic.NewMatchQuery("user_id", id) out_nick, err := client.UpdateByQuery().Index("test").Type("test").Query(query).Script(elastic.NewScriptInline("ctx._source.user_nick = '" + requestNick + "'")).Do(ctx) if nil != err { log.Println(err) } fmt.Println("update all nick: ", out_nick.Updated) return nil } </code></pre> <p>The POST in elastic:</p> <pre><code>POST {index}/{type}/_update_by_query { "script": { "inline": "ctx._source.user_name = 'test'", "inline": "ctx._source.user_nick = 'test test'" }, "query": { "match": { "user_id": "mtJZngDOy6Qj22Qv9MEf1MhSLVb2" } } } </code></pre> <p>I'm using the library github.com/olivere/elastic. The versión of elasticsearch is 5.6 This functions per separate it works well, but I have two problems:</p> <p>How can update in the same function? Why then use the two functions at the same time I have this error:</p> <p>elastic: Error 409 (Conflict)</p> </div>

elasticsearch插入数据报错

之前用的elasticsearch是1.3.2版本的,装了ik插件,用起来很正常,后面elasticsearch换成2.0.0的了,还装了logstash2.0.0,kibana4.2.0, ik分词器是1.5.0的,用Java代码批量插入的时候就报错了,在elasticsearch的控制台手动插入是没问题的 [INFO][2016-11-16 18:22:34] org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:151) main [Agatha Harkness] loaded [analysis-jcseg], sites [] [INFO][2016-11-16 18:22:35] org.elasticsearch.client.transport.TransportClientNodesService$SniffNodesSampler$1$1.handleException(TransportClientNodesService.java:443) elasticsearch[Agatha Harkness][transport_client_worker][T#1]{New I/O worker #28} [Agatha Harkness] failed to get local cluster state for [#transport#-1][USER-20150529VW][inet[localhost/127.0.0.1:9300]], disconnecting... org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:173) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.StreamCorruptedException: Unsupported version: 1 at org.elasticsearch.common.io.ThrowableObjectInputStream.readStreamHeader(ThrowableObjectInputStream.java:46) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at org.elasticsearch.common.io.ThrowableObjectInputStream.<init>(ThrowableObjectInputStream.java:38) at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:170) ... 23 more [WARN][2016-11-16 18:22:35] org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:135) elasticsearch[Agatha Harkness][transport_client_worker][T#1]{New I/O worker #28} [Agatha Harkness] Message not fully read (response) for [0] handler org.elasticsearch.client.transport.TransportClientNodesService$SniffNodesSampler$1$1@7e8c9412, error [true], resetting [INFO][2016-11-16 18:22:39] org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:862) Thread-1 Closing org.springframework.context.support.GenericApplicationContext@fbd1f6: startup date [Wed Nov 16 18:21:55 CST 2016]; root of context hierarchy [INFO][2016-11-16 18:22:39] org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:272) Thread-1 [Fer-de-Lance] stopping ... [INFO][2016-11-16 18:22:39] org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:310) Thread-1 [Fer-de-Lance] stopped

从logstash向elasticsearch导入数据出错

从logstash向elasticsearch写数据的时候,如果是单节点的ES,就可以写入成功,如果是多个节点的ES集群,就会出现以下错误: ``` 警告: [logstash-h1s5-4437-11306] waited for 30s and no initial state was set by the discovery 六月 19, 2015 10:48:49 上午 org.elasticsearch.node.internal.InternalNode start 信息: [logstash-h1s5-4437-11306] started Failed to install template: waited for [30s] {:level=>:error} Logstash startup completed yhao 2015-06-19T02:57:25.046Z h1s5 yhao word Got error to send bulk of actions: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master]; {:level=>:error} Failed to flush outgoing items {:outgoing_count=>1, :exception=>org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];, :backtrace=>["org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(org/elasticsearch/cluster/block/ClusterBlocks.java:151)"...... ``` ES只对cluster的名字进行了改动,logstash也没有进行改动

Elasticsearch索引数据丢失

项目已经运行一年了之前从未出现过这种问题最近半个月突然出现索引丢失数据的情况 通过es打印的日志也没有发现问题 ## 服务器配置 系统:contos7 CPU:1 内核:1核 内存:16G 环境:jdk1.8、es2.4** ## 部分日志 [2020-05-13 09:02:54,107][INFO ][discovery ] [chain_store_1] chain_store/i-TSPqKyRR6F-od14etdJQ [2020-05-13 09:02:57,148][INFO ][cluster.service ] [chain_store_1] new_master {chain_store_1}{i-TSPqKyRR6F-od14etdJQ}{172.17.0.1}{172.17.0.1:9301}, reason: zen-disco-join(elected_as_master, [0] joins received) [2020-05-13 09:02:57,189][INFO ][http ] [chain_store_1] publish_address {172.17.0.1:9300}, bound_addresses {0.0.0.0:9300} [2020-05-13 09:02:57,189][INFO ][node ] [chain_store_1] started [2020-05-13 09:02:57,197][INFO ][gateway ] [chain_store_1] recovered [0] indices into cluster_state [2020-05-13 09:03:22,780][INFO ][cluster.metadata ] [chain_store_1] [activity_es_index] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [] [2020-05-13 09:03:23,247][INFO ][cluster.metadata ] [chain_store_1] [activity_es_index] create_mapping [activity_es_type] [2020-05-13 09:03:23,266][INFO ][cluster.routing.allocation] [chain_store_1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[activity_es_index][4], [activity_es_index][4]] ...]). [2020-05-13 09:03:32,556][INFO ][cluster.metadata ] [chain_store_1] [apk_es_index] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [] [2020-05-13 09:03:32,694][INFO ][cluster.routing.allocation] [chain_store_1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[apk_es_index][4]] ...]). ## es配置文件 ![图片说明](https://img-ask.csdn.net/upload/202005/13/1589338034_25651.png)

关于Elasticsearch中Index与type、id、routing数据结构设计合理性的疑惑,有两套方案麻烦各位了

各位大神: 我先简单讲述一下项目的业务逻辑。一个物联网的项目,设备连接网关(可以理解为中间介质),经由网关进行数据推送到服务器,我们做的事情需要在服务上接收各地项目网关上报的数据,进行不同时间维度的数据分析。 几个约定: 1、同一个项目,网关编号不会重复; 2、同一个网关下的设备编号不会重复,但是有可能与其他网关下的设备编号重复 3、设备拥有多种不同的参数编号及参数值 初步存放数据设计的格式有两种方案(基于es 5.x的版本) 方案一: 1、Index=存储项目编号+日期(yyyy-MM) 2、type=存储网关编号 3、id =存储设备编号 4、routing=设备参数 基于方案一,好处在数据分类上可能比较直观,坏处就是会产生多个type、id,而且id不唯一,需要id+routing才能表达唯一。但是不知道es对多个type的支持是否效率高。 方案二: 1、index=项目编号+日期(yyyy-MM) 2、type=data(仅仅说明这个是原始的数据) 基于方案二、好处就是type较少,检索数据直接通过字段查询。 上诉两种方案,可能是我理解的不够透彻,所以不懂那种方式属于es支持的方案

ES环境搭建,使用head插件访问,在es的log文件中的记录如下,这个怎么解决?

[2017-12-20T04:53:41,220][WARN ][o.e.n.Node ] [node-1] timed out while waiting for initial discovery state - timeout: 30s [2017-12-20T04:54:11,513][WARN ][r.suppressed ] path: /_all, params: {index=_all} org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:211) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:307) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:237) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:1157) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-5.2.0.jar:5.2.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] [2017-12-20T04:54:11,513][WARN ][r.suppressed ] path: /_all, params: {index=_all} org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:211) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:307) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:237) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:1157) [elasticsearch-5.2.0.jar:5.2.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-5.2.0.jar:5.2.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

ELK日志收集错误,elasticsearch日志抛错

![图片说明](https://img-ask.csdn.net/upload/201603/30/1459332638_672977.png) 问题描述: 我用logstash收集日志,保存到elasticsearch 每天都会按日期建立新的索引。但是elasticsearch日志显示,建立索引之后,有错误日志, [2016-03-30 08:00:03,002][INFO ][cluster.metadata ] [log_master] [union_user-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [union_user] [2016-03-30 08:00:08,369][INFO ][cluster.metadata ] [log_master] [dj_openapi_middle-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [dj_openapi_middle] [2016-03-30 08:00:10,188][INFO ][cluster.metadata ] [log_master] [union_openapi-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [union_openapi] [2016-03-30 08:00:12,024][INFO ][cluster.metadata ] [log_master] [mcp_log-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [mcp_log] [2016-03-30 08:00:13,870][INFO ][cluster.metadata ] [log_master] [own_openapi-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [own_openapi] [2016-03-30 08:00:17,354][INFO ][cluster.metadata ] [log_master] [dj_openapi_front-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [dj_openapi_front] [2016-03-30 08:00:19,204][INFO ][cluster.metadata ] [log_master] [own_payment-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [own_payment] [2016-03-30 08:00:19,341][INFO ][cluster.metadata ] [log_master] [callback_user-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [callback_user] [2016-03-30 08:00:21,357][INFO ][cluster.metadata ] [log_master] [callback_payment-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [callback_payment] [2016-03-30 08:00:35,142][INFO ][cluster.metadata ] [log_master] [callback_openapi-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [callback_openapi] [2016-03-30 08:00:40,467][INFO ][cluster.metadata ] [log_master] [risk_ctrl-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [risk_ctrl] [2016-03-30 08:00:42,393][INFO ][cluster.metadata ] [log_master] [own_user-2016-03-30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [own_user] [2016-03-30 08:01:00,892][DEBUG][action.admin.indices.mapping.put] [log_master] failed to put mappings on indices [[union_user-2016-03-30]], type [union_user] ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [union_user]) within 30s] at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:343) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2016-03-30 08:01:00,892][DEBUG][action.admin.indices.mapping.put] [log_master] failed to put mappings on indices [[union_user-2016-03-30]], type [union_user] ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [union_user]) within 30s] at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:343) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2016-03-30 08:01:00,899][DEBUG][action.bulk ] [log_master] [union_user-2016-03-30][2] failed to execute bulk item (index) index {[union_user-2016-03-30][union_user][AVPE0kc4VKH7icABsVlx], source[{"message":"FAT AL: 03-29 23:48:20: duokoo_user * 25953 [ logid: ][ reqip: ][DuokooUser.cpp:4646]query fetchSdkLoadingConfSort have no data, [DuokooUser.cpp:4646] [sql:select status, starttime, endtime, appids, channels, picpath, UNIX_ TIMESTAMP(createtime), UNIX_TIMESTAMP(starttime), UNIX_TIMESTAMP(endtime),UNIX_TIMESTAMP() from mcp_user.mcp_sdk_loading_config where type=1 and status=1 and UNIX_TIMESTAMP() <= UNIX_TIMESTAMP(endtime) ]","@version":"1","@ti mestamp":"2016-03-30T00:00:28.290Z","path":"/home/work/duokoo/log/duokoo_user.log.wf","host":"0.0.0.0","type":"union_user"}]} ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [union_user]) within 30s] 求各位大神帮忙解答一下,是哪里的原因

用RestHighLevelClient将数据从MongoDb同步到ElasticSearch报错

如图,当批量增加时报错 ![图片说明](https://img-ask.csdn.net/upload/201904/22/1555923782_384390.png) 我的代码如下: Test类: ``` @Test public void aa (UserVo userVo) throws IOException { try { MongoClient mongo = new MongoClient("47.106.125.227", 27017);//连接mongo DB db = mongo.getDB("qhc");//获取数据库 DBCollection table = db.getCollection("sjs_list");//获取表名 RestHighLevelClient client = new RestHighLevelClient( RestClient.builder( new HttpHost("localhost", 9200, "http")));//连接es //找到数据 List<UserVo> userVoList=new ArrayList<>(); userVoList.add(userVo); //放入数据 batchInsertToEsSync(client,userVoList,"user","_search");//导出到es 表 字段 } catch (Exception e) { e.printStackTrace(); } } ``` batchInsertToEsSync: ``` public void batchInsertToEsSync(RestHighLevelClient client,List<UserVo> objs,String tableName,String type) throws IOException {//导出 BulkRequest bulkRequest=new BulkRequest(); for(UserVo obj:objs) { IndexRequest req = new IndexRequest(tableName, type); Map<String,Object> map=new HashMap<>(); //获取数据 String id = obj.getId(); String title = obj.getTitle(); String content = obj.getContent(); String source = obj.getSource(); String date = obj.getDate(); String pageUrl = obj.getPageUrl(); String areaTag = obj.getAreaTag(); String affairsTag = obj.getAffairsTag(); String contentTag = obj.getContentTag(); List<FuJian> enclosure= obj.getEnclosure(); String img = obj.getImg(); String summary = obj.getSummary(); String tag = obj.getTag(); String labelName = obj.getLabelName(); //添加数据 map.put("_id",id); map.put("_title",title); map.put("_content",content); map.put("_source",source); map.put("_date",date); map.put("_pageUrl",pageUrl); map.put("_areaTag",areaTag); map.put("_affairsTag",affairsTag); map.put("_contentTag",contentTag); map.put("_enclosure",enclosure); map.put("_img",img); map.put("_summary",summary); map.put("_tag",tag); map.put("_labelName",labelName); req.id(map.get("_id").toString()); req.source(map, XContentType.JSON); bulkRequest.add(req); } BulkResponse bulkResponse=client.bulk(bulkRequest); for (BulkItemResponse bulkItemResponse : bulkResponse) { if (bulkItemResponse.isFailed()) { System.out.println(bulkItemResponse.getId()+","+bulkItemResponse.getFailureMessage()); } } } ``` pom.xml一部分: ``` <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>6.2.3</version> <exclusions> <exclusion> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> </exclusion> </exclusions> </dependency> ``` 是包冲突还是?我的elasticsearch版本是6.2.3,jdk1.8,有遇到过的朋友加下QQ1479756648 ,万分感谢

es 批量插入MySQL数据 数据类型为timestamp 怎么设置es日期时间类型

\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [-62170012800] with format [date_time]\ 上面的是es(7.x)设置为date_time 类型时报的错误 MySQL中的数据是2019-08-30 12:20:24 格式如下: `receipt_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00'

在elasticsearch中提升特定文档

<div class="post-text" itemprop="text"> <p>I'm creating an autocomplete query for hotels and destinations. I want than when i type Can or Canc i see Cancun in the first results of my query. My php-code is the follow:</p> <pre><code>$params = [ 'index' =&gt; ['hotels', 'destination_ngrams'], 'type' =&gt; ['hotel','city'], "size" =&gt; 100 , 'body' =&gt; [ 'query' =&gt; [ 'multi_match' =&gt; [ "type" =&gt; "best_fields", "query" =&gt; $text, "fields" =&gt; ["destination_name_*^3","hotel_name"], "fuzziness" =&gt; "AUTO" ], ] ] ]; </code></pre> <p>I already have a list of popular "citys and hotels" that i want to see first in my autocomplete.</p> <p>Or, how can I give priority to destinations that start exactly with the same letters?</p> <p>I think i can put a "relevance" value when I make the bulk code, but i don't be shure what way to follow</p> <p>Anyone can help me?, i apresiate it alot!</p> <p>Thanks!!</p> </div>

Olivere软件包中的BulkIndexer用于Golang替换Elastigo

<div class="post-text" itemprop="text"> <p>I notice that I can use BulkIndexer if I want to send data into elasticsearch in bulk. As stated in the Elastigo documentation </p> <blockquote> <p>A bulk indexer creates goroutines, and channels for connecting and sending data to elasticsearch in bulk, using buffers.</p> </blockquote> <p>Code in elastigo to insert in bulk </p> <pre><code>var c_es = elastigo.NewConn() var indexer = c_es.NewBulkIndexer(50) func insertInBulkElastic(){ //Create a custom error function when inserting data into elasticsearch //in bulk indexer.Sender = func(buf *bytes.Buffer) error { // @buf is the buffer of docs about to be written respJson, err := c_es.DoCommand("POST", "/_bulk", nil, buf) if err != nil { // handle it better than this fmt.Println("Error", string(respJson)) // fmt.Println("Error", err) } if err == nil { fmt.Println("The data was inserted successfullly to elastic search") } return err } } </code></pre> <p>Does anyone know how to send bulk request using olivere for golang?</p> <p>Thanks </p> </div>

如何使用Olivere Package将数据添加到Elasticsearch中的现有索引中

<div class="post-text" itemprop="text"> <p>I am not sure whether it is correct or not. But I think that if I want to add data into existing index in elasticsearch using olivere in golang, the only option available is to delete the index and re index the data. </p> <p>This is my code : </p> <pre><code>bulkRequest := client.Bulk() for j := 0; j &lt; 20; j++ { n++ tweet := Tweet{User: "Rudi", Message: "This is the new from string representations of basic data types. " + strconv.Itoa(n)} req := elastic.NewBulkIndexRequest().Index("employee").Type("tweet").Id(strconv.Itoa(n)).Doc(tweet) bulkRequest.Add(req) } bulkResponse, err := bulkRequest.Do(context.Background()) </code></pre> <p>Does anyone know a better solution? </p> <p>Thanks </p> </div>

Elasticsearch 6.0 curl 导入 Kibana 示例数据出错,求救

按照官网的示例 导入数据 三个json格式数据文件 shakespeare.json, accounts.json, logs.jsonl ``` curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/doc/_bulk?pretty' --data-binary @shakespeare_6.0.json curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl ``` ![图片说明](https://img-ask.csdn.net/upload/201711/25/1511599223_96934.png) 网上搜索没有找到解决方式 "error" : "Content-Type header [application/x-www-form-urlencoded] is not supported", "status" : 406

6.3版本elk问题求助!!!!!

[2018-08-31T15:03:37,206][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>13} [2018-08-31T15:03:38,322][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"}) [2018-08-31T15:03:38,322][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"}) [2018-08-31T15:03:38,322][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"}) [2018-08-31T15:03:38,322][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"}) [2018-08-31T15:03:38,322][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"There are no external requests known to support wildcards that don't support replacing their indices"}) 6.3版本elk运行一段时间logstash日志出现这些信息,logstash的events received rate骤降为0.怎么解决

es buikSave批量插入报错 transport client is closed

![图片说明](https://img-ask.csdn.net/upload/201812/31/1546230912_607053.png)# 、 ### 报错信息如图所示,现在需求是插入8000w数据,我是按照1000 每次,多线程插入的,但是插入到一定数量以后,就会报这个错,希望大佬可以帮我解决下,

学Python后到底能干什么?网友:我太难了

感觉全世界营销文都在推Python,但是找不到工作的话,又有哪个机构会站出来给我推荐工作? 笔者冷静分析多方数据,想跟大家说:关于超越老牌霸主Java,过去几年间Python一直都被寄予厚望。但是事实是虽然上升趋势,但是国内环境下,一时间是无法马上就超越Java的,也可以换句话说:超越Java只是时间问题罢。 太嚣张了会Python的人!找工作拿高薪这么简单? https://edu....

在中国程序员是青春饭吗?

今年,我也32了 ,为了不给大家误导,咨询了猎头、圈内好友,以及年过35岁的几位老程序员……舍了老脸去揭人家伤疤……希望能给大家以帮助,记得帮我点赞哦。 目录: 你以为的人生 一次又一次的伤害 猎头界的真相 如何应对互联网行业的「中年危机」 一、你以为的人生 刚入行时,拿着傲人的工资,想着好好干,以为我们的人生是这样的: 等真到了那一天,你会发现,你的人生很可能是这样的: ...

为什么程序猿都不愿意去外包?

分享外包的组织架构,盈利模式,亲身经历,以及根据一些外包朋友的反馈,写了这篇文章 ,希望对正在找工作的老铁有所帮助

Java校招入职华为,半年后我跑路了

何来 我,一个双非本科弟弟,有幸在 19 届的秋招中得到前东家华为(以下简称 hw)的赏识,当时秋招签订就业协议,说是入了某 java bg,之后一系列组织架构调整原因等等让人无法理解的神操作,最终毕业前夕,被通知调往其他 bg 做嵌入式开发(纯 C 语言)。 由于已至于校招末尾,之前拿到的其他 offer 又无法再收回,一时感到无力回天,只得默默接受。 毕业后,直接入职开始了嵌入式苦旅,由于从未...

Java基础知识面试题(2020最新版)

文章目录Java概述何为编程什么是Javajdk1.5之后的三大版本JVM、JRE和JDK的关系什么是跨平台性?原理是什么Java语言有哪些特点什么是字节码?采用字节码的最大好处是什么什么是Java程序的主类?应用程序和小程序的主类有何不同?Java应用程序与小程序之间有那些差别?Java和C++的区别Oracle JDK 和 OpenJDK 的对比基础语法数据类型Java有哪些数据类型switc...

@程序员:GitHub这个项目快薅羊毛

今天下午在朋友圈看到很多人都在发github的羊毛,一时没明白是怎么回事。 后来上百度搜索了一下,原来真有这回事,毕竟资源主义的羊毛不少啊,1000刀刷爆了朋友圈!不知道你们的朋友圈有没有看到类似的消息。 这到底是啥情况? 微软开发者平台GitHub 的一个区块链项目 Handshake ,搞了一个招募新会员的活动,面向GitHub 上前 25万名开发者派送 4,246.99 HNS币,大约价...

用python打开电脑摄像头,并把图像传回qq邮箱【Pyinstaller打包】

前言: 如何悄悄的打开朋友的摄像头,看看她最近过的怎么样,嘿嘿!这次让我带你们来实现这个功能。 注: 这个程序仅限在朋友之间开玩笑,别去搞什么违法的事情哦。 代码 发送邮件 使用python内置的email模块即可完成。导入相应的代码封装为一个send函数,顺便导入需要导入的包 注: 下面的代码有三处要修改的地方,两处写的qq邮箱地址,还有一处写的qq邮箱授权码,不知道qq邮箱授权码的可以去百度一...

做了5年运维,靠着这份监控知识体系,我从3K变成了40K

从来没讲过运维,因为我觉得运维这种东西不需要太多的知识面,然后我一个做了运维朋友告诉我大错特错,他就是从3K的运维一步步到40K的,甚至笑着说:我现在感觉自己什么都能做。 既然讲,就讲最重要的吧。 监控是整个运维乃至整个产品生命周期中最重要的一环,事前及时预警发现故障,事后提供详实的数据用于追查定位问题。目前业界有很多不错的开源产品可供选择。选择一款开源的监控系统,是一个省时省力、效率最高的方...

C++(继承):19---虚基类与虚继承(virtual)

一、菱形继承 在介绍虚继承之前介绍一下菱形继承 概念:A作为基类,B和C都继承与A。最后一个类D又继承于B和C,这样形式的继承称为菱形继承 菱形继承的缺点: 数据冗余:在D中会保存两份A的内容 访问不明确(二义性):因为D不知道是以B为中介去访问A还是以C为中介去访问A,因此在访问某些成员的时候会发生二义性 缺点的解决: 数据冗余:通过下面“虚继承”技术来解决(见下) 访问...

再不跳槽,应届毕业生拿的都比我多了!

跳槽几乎是每个人职业生涯的一部分,很多HR说“三年两跳”已经是一个跳槽频繁与否的阈值了,可为什么市面上有很多程序员不到一年就跳槽呢?他们不担心影响履历吗? PayScale之前发布的**《员工最短任期公司排行榜》中,两家码农大厂Amazon和Google**,以1年和1.1年的员工任期中位数分列第二、第四名。 PayScale:员工最短任期公司排行榜 意外的是,任期中位数极小的这两家公司,薪资...

我以为我学懂了数据结构,直到看了这个导图才发现,我错了

数据结构与算法思维导图

技术大佬:我去,你写的 switch 语句也太老土了吧

昨天早上通过远程的方式 review 了两名新来同事的代码,大部分代码都写得很漂亮,严谨的同时注释也很到位,这令我非常满意。但当我看到他们当中有一个人写的 switch 语句时,还是忍不住破口大骂:“我擦,小王,你丫写的 switch 语句也太老土了吧!” 来看看小王写的代码吧,看完不要骂我装逼啊。 private static String createPlayer(PlayerTypes p...

华为初面+综合面试(Java技术面)附上面试题

华为面试整体流程大致分为笔试,性格测试,面试,综合面试,回学校等结果。笔试来说,华为的难度较中等,选择题难度和网易腾讯差不多。最后的代码题,相比下来就简单很多,一共3道题目,前2题很容易就AC,题目已经记不太清楚,不过难度确实不大。最后一题最后提交的代码过了75%的样例,一直没有发现剩下的25%可能存在什么坑。 笔试部分太久远,我就不怎么回忆了。直接将面试。 面试 如果说腾讯的面试是挥金如土...

和黑客斗争的 6 天!

互联网公司工作,很难避免不和黑客们打交道,我呆过的两家互联网公司,几乎每月每天每分钟都有黑客在公司网站上扫描。有的是寻找 Sql 注入的缺口,有的是寻找线上服务器可能存在的漏洞,大部分都...

讲一个程序员如何副业月赚三万的真实故事

loonggg读完需要3分钟速读仅需 1 分钟大家好,我是你们的校长。我之前讲过,这年头,只要肯动脑,肯行动,程序员凭借自己的技术,赚钱的方式还是有很多种的。仅仅靠在公司出卖自己的劳动时...

win10暴力查看wifi密码

刚才邻居打了个电话说:喂小灰,你家wifi的密码是多少,我怎么连不上了。 我。。。 我也忘了哎,就找到了一个好办法,分享给大家: 第一种情况:已经连接上的wifi,怎么知道密码? 打开:控制面板\网络和 Internet\网络连接 然后右击wifi连接的无线网卡,选择状态 然后像下图一样: 第二种情况:前提是我不知道啊,但是我以前知道密码。 此时可以利用dos命令了 1、利用netsh wlan...

上班一个月,后悔当初着急入职的选择了

最近有个老铁,告诉我说,上班一个月,后悔当初着急入职现在公司了。他之前在美图做手机研发,今年美图那边今年也有一波组织优化调整,他是其中一个,在协商离职后,当时捉急找工作上班,因为有房贷供着,不能没有收入来源。所以匆忙选了一家公司,实际上是一个大型外包公司,主要派遣给其他手机厂商做外包项目。**当时承诺待遇还不错,所以就立马入职去上班了。但是后面入职后,发现薪酬待遇这块并不是HR所说那样,那个HR自...

女程序员,为什么比男程序员少???

昨天看到一档综艺节目,讨论了两个话题:(1)中国学生的数学成绩,平均下来看,会比国外好?为什么?(2)男生的数学成绩,平均下来看,会比女生好?为什么?同时,我又联想到了一个技术圈经常讨...

总结了 150 余个神奇网站,你不来瞅瞅吗?

原博客再更新,可能就没了,之后将持续更新本篇博客。

副业收入是我做程序媛的3倍,工作外的B面人生是怎样的?

提到“程序员”,多数人脑海里首先想到的大约是:为人木讷、薪水超高、工作枯燥…… 然而,当离开工作岗位,撕去层层标签,脱下“程序员”这身外套,有的人生动又有趣,马上展现出了完全不同的A/B面人生! 不论是简单的爱好,还是正经的副业,他们都干得同样出色。偶尔,还能和程序员的特质结合,产生奇妙的“化学反应”。 @Charlotte:平日素颜示人,周末美妆博主 大家都以为程序媛也个个不修边幅,但我们也许...

MySQL数据库面试题(2020最新版)

文章目录数据库基础知识为什么要使用数据库什么是SQL?什么是MySQL?数据库三大范式是什么mysql有关权限的表都有哪几个MySQL的binlog有有几种录入格式?分别有什么区别?数据类型mysql有哪些数据类型引擎MySQL存储引擎MyISAM与InnoDB区别MyISAM索引与InnoDB索引的区别?InnoDB引擎的4大特性存储引擎选择索引什么是索引?索引有哪些优缺点?索引使用场景(重点)...

女朋友过生日,我花了20分钟给她写了一个代理服务器

女朋友说:“看你最近挺辛苦的,我送你一个礼物吧。你看看想要什么,我来准备。” 我想了半天,从书到鞋子到电子产品最后到生活用品,感觉自己什么都不缺,然后和她说:“你省省钱吧,我什么都不需要。” 她坚持要送:“不行,你一定要说一个礼物,我想送你东西了。” 于是,我认真了起来,拿起手机,上淘宝逛了几分钟,但还是没能想出来缺点什么,最后实在没办法了:“这样吧,如果你实在想送东西,那你就写一个代理服务器吧”...

记一次腾讯面试,我挂在了最熟悉不过的队列上……

腾讯后台面试,面试官问:如何自己实现队列?

如果你是老板,你会不会踢了这样的员工?

有个好朋友ZS,是技术总监,昨天问我:“有一个老下属,跟了我很多年,做事勤勤恳恳,主动性也很好。但随着公司的发展,他的进步速度,跟不上团队的步伐了,有点...

我入职阿里后,才知道原来简历这么写

私下里,有不少读者问我:“二哥,如何才能写出一份专业的技术简历呢?我总感觉自己写的简历太烂了,所以投了无数份,都石沉大海了。”说实话,我自己好多年没有写过简历了,但我认识的一个同行,他在阿里,给我说了一些他当年写简历的方法论,我感觉太牛逼了,实在是忍不住,就分享了出来,希望能够帮助到你。 01、简历的本质 作为简历的撰写者,你必须要搞清楚一点,简历的本质是什么,它就是为了来销售你的价值主张的。往深...

程序员写出这样的代码,能不挨骂吗?

当你换槽填坑时,面对一个新的环境。能够快速熟练,上手实现业务需求是关键。但是,哪些因素会影响你快速上手呢?是原有代码写的不够好?还是注释写的不够好?昨夜...

带了6个月的徒弟当了面试官,而身为高级工程师的我天天修Bug......

即将毕业的应届毕业生一枚,现在只拿到了两家offer,但最近听到一些消息,其中一个offer,我这个组据说客户很少,很有可能整组被裁掉。 想问大家: 如果我刚入职这个组就被裁了怎么办呢? 大家都是什么时候知道自己要被裁了的? 面试软技能指导: BQ/Project/Resume 试听内容: 除了刷题,还有哪些技能是拿到offer不可或缺的要素 如何提升面试软实力:简历, 行为面试,沟通能...

!大部分程序员只会写3年代码

如果世界上都是这种不思进取的软件公司,那别说大部分程序员只会写 3 年代码,恐怕就没有程序员这种职业。

离职半年了,老东家又发 offer,回不回?

有小伙伴问松哥这个问题,他在上海某公司,在离职了几个月后,前公司的领导联系到他,希望他能够返聘回去,他很纠结要不要回去? 俗话说好马不吃回头草,但是这个小伙伴既然感到纠结了,我觉得至少说明了两个问题:1.曾经的公司还不错;2.现在的日子也不是很如意。否则应该就不会纠结了。 老实说,松哥之前也有过类似的经历,今天就来和小伙伴们聊聊回头草到底吃不吃。 首先一个基本观点,就是离职了也没必要和老东家弄的苦...

2020阿里全球数学大赛:3万名高手、4道题、2天2夜未交卷

阿里巴巴全球数学竞赛( Alibaba Global Mathematics Competition)由马云发起,由中国科学技术协会、阿里巴巴基金会、阿里巴巴达摩院共同举办。大赛不设报名门槛,全世界爱好数学的人都可参与,不论是否出身数学专业、是否投身数学研究。 2020年阿里巴巴达摩院邀请北京大学、剑桥大学、浙江大学等高校的顶尖数学教师组建了出题组。中科院院士、美国艺术与科学院院士、北京国际数学...

立即提问
相关内容推荐