如何使用这种架构在Elastic Search中复制索引？

Question

I have a scenario where I have to import data (millions of records) from multiple sources and save it in a database. A user should get results in under 2-3 seconds when they try to search for any information related to that data.

For this, I designed an architecture where I used golang to import data from multiple sources and pushed data in AWS SQS. I've created a lambda function which triggers when AWS SQS has some data. This lambda function then pushes data in AWS Elastic Search. I've created a Rest API using which I give results to the user.

I use CRON to do this importing work every morning. Now my problem is if a new batch of data comes I want to delete the existing data and replace all of them with the new data. I'm stuck at how I can achieve this deleting and adding new data part.

I thought of creating a temporary index and then replacing it with the original index. But the problem is I do not know when importing has ended and can make this index switch.

Answer 1

The concept you're after is an index alias. The basic workflow would be:

1. Import today's data into an index with my-index-2019-09-16 (for example).
2. Make sure the import is complete and worked correctly.

3. Point the alias to the new index (it's an atomic switch between the indices):

   POST /_aliases
   {
       "actions" : [
           { "remove" : { "index" : "my-index-2019-09-15", "alias" : "my-index" } },
           { "add" : { "index" : "my-index-2019-09-16", "alias" : "my-index" } }
       ]
   }
   

4. Delete the old index.

You will double the disk space during the import process, but otherwise this should work without any issues and you only delete data once it has a proper replacement.