如何使用mysql优化solr索引

Question

I am using Mysql for indexing data to SOLR. Earlier the data was less, so the indexing happened quickly without any lag. Now my data contains almost 3 million rows and the mysql query always times out and because of that SOLR can't index the data accordingly.Is there any way to index SOLR fastly using mysql or can do any tweaking in SOLR? Please help.

Answer 1

If you're not using incremental / delta indexing, you should start doing that instead. That way only the rows that have changed since the last index ran will be indexed again, allowing you to lessen the impact and the amount of rows from MySQL.

In addition the JDBCDataSource (which I guess you're using) supports the batchSize parameter, which tells the JDBC driver to limit the amount of documents in each query - and issue more than one query instead.

You should also take care to have usable indexes on your data if you're performing any sort of filtering on the SQL content when retrieving it (such as for a delta import).