I have a list of urls to process, but I want to run a maximum number of goroutines at a time. For example, if I have 30 urls, I only want 10 goroutines working in parallel.
My attempt at this is the following:
parallel := flag.Int("parallel", 10, "max parallel requests allowed")
flag.Parse()
urls := flag.Args()
var wg sync.WaitGroup
client := rest.Client{}
results := make(chan string, *parallel)
for _, url := range urls {
wg.Add(1)
go worker(url, client, results, &wg)
}
for res := range results {
fmt.Println(res)
}
wg.Wait()
close(results)
My understanding is that if I create a buffered channel of size parallel, then the code will block until I read off the results channel, which will unblock my code and allow another goroutine to be spawned. However, this code doesn't seems to block after processing all the urls. Can someone explain to me how I can use channels to limit the number of goroutines running?