自上次修改以来如何获取文件的添加内容

Question

I'm working on a project in golang that needs to index recently added file content (using framework called bleve), and I'm looking for a solution to get content of a file since last modification. My current work-around is to record the last indexed position of each file, and during indexing process later on I only retrieve file content starting from the previous recorded position.

So I wonder if there's any library or built-in functionality for this? (doesn't need to be restricted to go, any language could work)

I'll really appreciate it if anyone has a better idea than my work-around as well!

Thanks

Answer 1

It depends on how the files change.

If the files are append-only, then you only need to record the last offset where you stopped indexing, and start from there.

If the changes can happen anywhere, and the changes are mostly replacing old bytes with new bytes (like changing pixels of an image), then perhaps you can consider computing checksum for small chucks, and only index those chunks that has different checksums.

You can check out crypto package in Go standard library for computing hashes.

If the changes are line insertion/deletion to text files (like changes to source code), then maybe a diff algorithm can help you find the differences. Something like https://github.com/octavore/delta.