I have a Go app that is using the Radix Redis client for Go. Its job is to run as a background process and wait for a Redis list to have items added to it for processing.
Background (feel free to skip): I have a Node.js web app that has a need to request data from an external API that's been known to take at least an entire second or more to respond. The data isn't required immediately so I'm having the Node app add an item to a Redis list and move on to the important stuff. Meanwhile, I have a Go app that is supposed to run behind the scenes and do some work each time it finds an item has been added to the list.
Right now it's just a main function that connects to Redis and checks to see if a list has items in it using the Redis BRPOP command (because I want Redis to block while its waiting for jobs). Running the program results in a connection being made and then...
- If there's an item in a list, the Go app processes it however it needs to
- If there's no items in the list then the Go app just quits
I need the Go app to just sit there and wait for new items and I really don't want to resort to polling especially since the BRPOP and BLPUSH commands are made specifically so apps don't need to poll.
My current solution is to just create an infinite loop that runs the BRPOP command over and over again forever. This feels like the wrong way to do it.
Here's the entire program so far (I'm completely new to Go but not to programming so don't laugh too hard):
package main
import (
"fmt"
"github.com/fzzy/radix/redis"
)
func main() {
client, err := redis.Dial("tcp", "localhost:6379")
if err != nil {
fmt.Println("An error message just for debug purposes")
} else {
// Here I check to make sure the server is responding
ping := client.Cmd("PING")
fmt.Println(ping) // Just to be sure I'm doing things right
for {
fmt.Println("Looking for a new Q item...")
workItem, err := client.Cmd("BRPOP", "q:test").Str()
if err != nil {
fmt.Println("No error, " + workItem)
}
}
}
}
So what is the proper way to make a Go app wait for data to become available in a Redis list?
Update: Based on some of the comments I'll explain a little further what I'm expecting and what's happening.
My understanding of a blocking Redis command is that it will just pause the program calling it until it has something to return. So in my case, when running the program with no items in the queue the program prints:
Looking for a new Q item...
No error,
It prints the above infinitely which makes sense since I have a never-ending for loop but because I'm using the blocking command, I was expecting it to behave more like a push. I expected the loop to just pause and wait until there's an item in the queue before moving on to the next iteration. My concern with it continuing to loop even with no queue items is that eventually the process will start to consume resources unnecessarily and slow down Redis because of the constant barrage of BRPOP commands (despite it being a blocking command).
So I suppose now the new questions would be - Is there a better way to keep the Go program running while waiting for new items in the Redis list (or my makeshift queue)? - Will running this program in its current form consume more resources than it should and/or degrade Redis performance?
Solved
Thanks to @JimB's question about the "error you're ignoring..." I took a second look at the code to make sure I wasn't missing something obvious. I was: Turns out that in my loop I had switched up the error and success handling code. So each time it errored out I would print a success message to the console just so I could see what was happening inside the program. After switching the code and correctly handling the error I found that I was calling BRPOP with too few arguments. I fixed the code and now the loop will just hang there until there's something in the list to handle.
Corrected lines below:
// We now import "log" now as well
// Inside the for loop...
if err != nil {
log.Fatal(err) // This alerted me to the wrong # of arguments issue
} else {
fmt.Println("No error to be had " + workItem) // Will not print until there is an item in the list
}
I guess I was doing it right all along. If there are any comments or suggestions about performance or better ways to do this, however, I'm still all hears and would love to hear them.
