Go中的预定轮询任务

Question

I have written some code that will concurrently poll URLs every 30 minutes:

func (obj * MyObj) Poll() {
    for ;; {
        for _, url := range obj.UrlList {
            //Download the current contents of the URL and do something with it
        }
        time.Sleep(30 * time.Minute)
}

//Start the routine in another function
go obj.Poll()

How would I then add to obj.UrlList elsewhere in the code and ensure that the next time the URLs are polled that the UrlList in the Poll goroutine as also been updated and as such will also poll the new URL?

I understand that memory is shared through communicating rather than vice versa in Go and I've investigated channels however I'm not sure how to implement them in this example.

Answer 1

Here's an untested, but safe model for periodically fetching some URLs with the ability to dynamically add new URLs to the list of URLs safely. It should be obvious to the reader what would be required if you wanted to remove a URL as well.

type harvester struct {
    ticker *time.Ticker // periodic ticker
    add    chan string  // new URL channel
    urls   []string     // current URLs
}

func newHarvester() *harvester {
    rv := &harvester{
        ticker: time.NewTicker(time.Minute * 30),
        add:    make(chan string),
    }
    go rv.run()
    return rv
}

func (h *harvester) run() {
    for {
        select {
        case <-h.ticker.C:
            // When the ticker fires, it's time to harvest
            for _, u := range h.urls {
                harvest(u)
            }
        case u := <-h.add:
            // At any time (other than when we're harvesting),
            // we can process a request to add a new URL
            h.urls = append(h.urls, u)
        }
    }
}

func (h *harvester) AddURL(u string) {
    // Adding a new URL is as simple as tossing it onto a channel.
    h.add <- u
}