2019-08-06 08:54
浏览 107

如何使用这种架构在Elastic Search中复制索引?

I have a scenario where I have to import data (millions of records) from multiple sources and save it in a database. A user should get results in under 2-3 seconds when they try to search for any information related to that data.

For this, I designed an architecture where I used golang to import data from multiple sources and pushed data in AWS SQS. I've created a lambda function which triggers when AWS SQS has some data. This lambda function then pushes data in AWS Elastic Search. I've created a Rest API using which I give results to the user.

I use CRON to do this importing work every morning. Now my problem is if a new batch of data comes I want to delete the existing data and replace all of them with the new data. I'm stuck at how I can achieve this deleting and adding new data part.

I thought of creating a temporary index and then replacing it with the original index. But the problem is I do not know when importing has ended and can make this index switch.

图片转代码服务由CSDN问答提供 功能建议

我有一种情况,我必须从多个来源导入数据(数百万条记录)并将其保存在数据库中 。 用户尝试搜索与该数据有关的任何信息时,应该在2-3秒内获得结果。</ p>

为此,我设计了一种体系结构,其中使用golang从中导入数据 AWS SQS中包含多个源和推送的数据。 我创建了一个lambda函数,该函数在AWS SQS包含某些数据时触发。 然后,此lambda函数将数据推送到AWS Elastic Search中。 我创建了一个Rest API,使用它可以向用户提供结果。</ p>

我每天早上都使用CRON进行此导入工作。 现在我的问题是,如果要一批新数据来,我想删除现有数据并用新数据替换所有数据。 我一直在坚持如何实现删除和添加新数据部分的工作。 </ p>

我考虑过创建一个临时索引,然后将其替换为原始索引。 但是问题是我不知道导入何时结束,是否可以使此索引切换。</ p> </ div>

1条回答 默认 最新

相关推荐 更多相似问题