doujiao3346 2019-08-06 08:54
浏览 107

如何使用这种架构在Elastic Search中复制索引?

I have a scenario where I have to import data (millions of records) from multiple sources and save it in a database. A user should get results in under 2-3 seconds when they try to search for any information related to that data.

For this, I designed an architecture where I used golang to import data from multiple sources and pushed data in AWS SQS. I've created a lambda function which triggers when AWS SQS has some data. This lambda function then pushes data in AWS Elastic Search. I've created a Rest API using which I give results to the user.

I use CRON to do this importing work every morning. Now my problem is if a new batch of data comes I want to delete the existing data and replace all of them with the new data. I'm stuck at how I can achieve this deleting and adding new data part.

I thought of creating a temporary index and then replacing it with the original index. But the problem is I do not know when importing has ended and can make this index switch.

  • 写回答

1条回答 默认 最新

  • duanhe7471 2019-08-16 14:06

    The concept you're after is an index alias. The basic workflow would be:

    1. Import today's data into an index with my-index-2019-09-16 (for example).
    2. Make sure the import is complete and worked correctly.
    3. Point the alias to the new index (it's an atomic switch between the indices):

      POST /_aliases
          "actions" : [
              { "remove" : { "index" : "my-index-2019-09-15", "alias" : "my-index" } },
              { "add" : { "index" : "my-index-2019-09-16", "alias" : "my-index" } }
    4. Delete the old index.

    You will double the disk space during the import process, but otherwise this should work without any issues and you only delete data once it has a proper replacement.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?



    • ¥15 易语言编写cs.16 绘制 矩阵出现了点问题 不知道这个矩阵可不可以使用
    • ¥15 k210显示failed init to model
    • ¥15 Evil-droid生成的APK手机已经下载但无法建立任务
    • ¥25 c语言韩信点兵的变式
    • ¥15 怎么根据书上的例子完成这个问题呢?
    • ¥15 ECharts 增加Zoom,整行包括右边的Text一起滑动
    • ¥15 关于网上一个easyx制作的见缝插针小游戏(c++)
    • ¥15 开地址法双散列函数处理碰撞
    • ¥15 想问一下这个是什么情况 虚拟机Linux打不开了
    • ¥15 联通光猫掉注册了怎么重新注册上去