I'm getting in 'stdin' lines of URL's like: $ echo -e 'https://golang.org https://godoc.org https://golang.org' | go run 1.go . The task is to get from each WEB-page number of word "Go". But I'm not allowed to start more than 5 goroutines and can use only standard library Here is my code:
package main
import (
"fmt"
"net/http"
"bufio"
"os"
"regexp"
"io/ioutil"
"time"
)
func worker(id int, jobs<-chan string, results chan<-int) {
t0 := time.Now()
for url := range jobs {
resp, err := http.Get(url)
if err != nil {
fmt.Println("problem while opening url", url)
results<-0
//continue
}
defer resp.Body.Close()
html, err := ioutil.ReadAll(resp.Body)
if err != nil {
continue
}
regExp:= regexp.MustCompile("Go")
matches := regExp.FindAllStringIndex(string(html), -1)
t1 := time.Now()
fmt.Println("Count for", url, ":", len(matches), "Elapsed time:",
t1.Sub(t0), "works id", id)
results<-len(matches)
}
}
func main(){
scanner := bufio.NewScanner(os.Stdin)
jobs := make(chan string, 100)
results := make(chan int, 100)
t0 := time.Now()
for w:= 0; w<5; w++{
go worker(w, jobs, results)
}
var tasks int = 0
res := 0
for scanner.Scan() {
jobs <- scanner.Text()
tasks ++
}
close(jobs)
for a := 1; a <= tasks; a++ {
res+=<-results
}
close(results)
t2 := time.Now()
fmt.Println("Total:",res, "Elapsed total time:", t2.Sub(t0) );
}
I thought it works until I passed more than 5 URL (one of them was incorrect) to stdin. The output was:
goroutine 9 [running]:
panic ...
Obviously, extra goroutnes have been started. How to fix it? May be there are more convenient way to limit number of goroutines?
