nsq go客户端无法跟上

Question

I'm a Go novice coming from the Node world and I'm building a consumer using the official Bitly Go client. I am using AddConcurrentHandlers to spawn 50 goroutines to handle the fire hose of messages. The issue is that my consumer falls behind leaving an exponential of unprocessed/received messages on nsq. Has anyone else encountered this?

I built the same thing in Node to see if there was a server or NSQ configuration issue and its able to process all messages as quickly as they come in.

GO CODE:

q, _ := nsq.NewConsumer("chat", "golangbetches", config)

q.AddConcurrentHandlers(nsq.HandlerFunc(func(message *nsq.Message) error {
  l.Debug("Got a message: %v", message)
  message.Finish()
  return nil
}), 50)

err := q.ConnectToNSQLookupd("<address here>")

Answer 1

cfg.MaxInFlight handles the, "maximum number of messages this comsumer instance will allow in-flight..." More details are available in the consumer source

Set cfg.MaxInFlight to something reasonable, as it defaults to 1

An example configuration is available in the documentation where it is set to 1000. This may or may not be suitable for your application; and, you'd do well to monitor it, as a misconfiguration may result in truncated messages.