在磁盘上存储时间序列数据的滑动窗口

Question

I've written a server which consumes a live stream of chronologically ordered events (time-series) and maintains a sliding window (24h) of the data. It only supports one type of query:

get N seconds of events starting at timestamp T

I'm currently using leveldb with unix timestamps as the keys. This works alright but, since leveldb is a LSM, the deletes are expensive (tombstones).

Can anyone suggest a better solution/datasore for this use case?

• The data is too big to store in memory
• The server is written in Go

Answer 1

The simplest solution is just always have 2 leveldb's - for the current day and the previous one. And you write to the current only.

And once a day you just

delete(previous)
previous := current
current := new()

You may then save almost 2 days instead of one, but rotation is really easy.