I am currently hitting an api to gather data for my own processing and what not. Currently I am doing 100 http.Get per second and am wondering what the best methodology is to do around 1000 concurrent http.Gets per second.
Here is what I have right now:
waitTime := time.Second
var lastID uint64 = 1234567890
for {
for i := 0; i < 100; i++ {
var tmpID uint64 = lastID
lastID++
go func(ID uint64) {
err = scrape(ID) // this does the http.Get and saves the
// resulting json into postgresql
if err != nil {
errStr := strings.TrimSpace(err.Error())
if strings.HasSuffix(errStr, "Too Many request to server") {
log.Println("hit a real 429")
panic(err)
}
}
}(tmpID)
}
time.Sleep(waitTime - time.Now().Sub(now)) // this is here to
// ensure I dont go over the limit
}
The api I am hitting is rate limited to 1000 req/s.
The reason for my go func(ID)
is so I can just incrementally increase my ID without having to worry about using a lock for access "what the next ID is".
I just feel like I am doing this wrong. I am pretty new to go in general as well.
I also assume I have to raise my ulimit
on my ubuntu server to something over 1000 as well to handle all these open connections.
any tips or suggestions are greatly appreciated!