在PHP中搜索大数据，所有数字，10亿字节

Question

I was wondering how I could quickly search a data string of up to 1 billion bytes of data. The data is all numeric. Currently, we have the data split into 250k files and the searches using strpos (fastest built-in function) on each file until it finds something. Is there a way I can index to make it go faster? Any suggestions?

Eventually I would like to find multiple occurrences, which, as of now, would be done with the offset parameter on strpos.

Any help would surely lead to recognition where needed.

Thanks! - James Hartig

Answer 1

Well, your tags indicate what you should do (the tag I am referring to is "indexing").

Basically, you should have separate files which would have the indexes for the data. It would have the data strings that you are looking for, as well as the file and byte positions that it is in.

You would then access the index, look up your value and then find the location(s) in the original file(s) for the data string, and process from there.