Golang：从切片中删除早于1h的结构

Question

So I'm building a small utility that listens on a socket and stores incoming messages as a structs in a slice:

var points []Point

type Point struct {
    time   time.Time
    x    float64
    y    float64
}

func main() {
    received = make([]Point, 0)
    l, err := net.Listen("tcp4", ":8900")
    // (...)
}


func processIncomingData(data string) {

    // Parse icoming data that comes as: xValue,yValue
    inData = strings.Split(data, ",")
    x, err := strconv.ParseFloat(inData[0], 64);
    if err != nil {
        fmt.Println(err)
    }
    y, err := strconv.ParseFloat(inData[1], 64);
    if err != nil {
        fmt.Println(err)
    }

    // Store the new Point
    points = append(points, Point{
        time:   time.Now(),
        x:    x,
        y:    y,
    })

    // Remove points older than 1h ?

}

Now, as you might imagine this will quickly fill my RAM. What's the best way (faster execution) to remove points older than 1h after appening each new one? I'll be getting new points 10-15 times peer second.

Thank you.

Answer 1

An approach I've used several times is to start a goroutine early in the project that looks something like this:

go cleanup()

...

func cleanup() {
  for {
    time.Sleep(...)
    // do cleanup
  }
}

Then what you could do is iterate over points using time.Since(point.time) to figure out how old each piece of data is. If it's too old, there's a slice trick to remove an item from a slice given it's position:

points = append(points[:i], points[i+1:]...)

(where i is the index to remove)

Because the points are in the slice in order of the time they were added, you could speed things up by simply finding the first index that isn't an hour old and doing points = points[i:] to chop off the old points from the beginning of the slice.

You may run into problems if you get a request that accesses the array while you're cleaning it up. Adding a sync.Mutex can help with that. Just lock the mutex before the cleanup and also attempt to lock the mutex anywhere else you write to the array. This may be premature optimization though. I'd experiment without the mutex before adding it in as this would effectively make interacting with points a serial operation and slow the service down.

The time.Sleep(...) in the loop is to prevent from cleaning too often. You might be tempted to set it to an hour since you want to delete points older than that but you might end up with a situation where a point is added immediately after a cleanup. On the next cleanup it'll be 59 mins old and you don't delete it, on the NEXT cleanup it's nearly 2 hours old. My rule of thumb is that I attempt to clean up every 1/10 the amount of time I want to allow an object to stay in memory but that's rather arbitrary. This approach means an object could be at most 1h 5m 59s old when it's deleted.