对一个通道使用多个接收器

Question

I am trying write a REST service in golang using go-json-rest

The purpose of the service is just to convert the received data to CSV and log it. Since the load may be heavy, I would like to do the logging using goroutines. Currently I have created four LogWorkers(goroutine) Each goroutine will log the CSV into separate files.

When I execute the code, the log is always triggered from the last goroutine. I see a single file created in my log folder which is from fourth routine.

Here is my server code

package main

import (
    "github.com/ant0ine/go-json-rest/rest"
    "log"
    "net/http"
    "strconv"
    "time"
)

const workerCount = 4
var evChannel = make(chan Event)
var workers = make([]*LogWorker, workerCount)
const maxLogFileSize = 100 // In MB
const maxLogFileBackups = 30
const maxLogFileAge = 5
const logFileName = "/home/sam/tmp/go_logs/event_"

func main() {
    // Initialize workers
    // Four workers is being created
    for i := 0; i < workerCount; i++ {
        var fileName = logFileName + strconv.Itoa(i)
        workers[i] = NewLogWorker(fileName, maxLogFileSize, maxLogFileBackups, maxLogFileAge)
        go workers[i].Work(evChannel)
    }    

    // Initialize REST API
    api := rest.NewApi()
    //api.Use(rest.DefaultDevStack...)
    api.Use(rest.DefaultCommonStack...)
    router, err := rest.MakeRouter(
        rest.Post("/events", StoreEvents),
    )
    if err != nil {
        log.Fatal(err)
    }
    api.SetApp(router)
    log.Fatal(http.ListenAndServe(":4545", api.MakeHandler()))
}

func StoreEvents(w rest.ResponseWriter, r *rest.Request) {
    event := Event{}
    err := r.DecodeJsonPayload(&event)
    if err != nil {
        rest.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    // TODO : Add validation if needed
    // Add code to parse the request and add further information to event 
    // log.Println()
    select {
        case evChannel <- event:
        case <- time.After(5 * time.Second):
      // throw away the message, so sad
    }    
    // evChannel <- event
    //log.Println(Csv(event))
    w.WriteHeader(http.StatusOK)
}

here is my worker code

package main

import (
    "gopkg.in/natefinch/lumberjack.v2"
    "log"
    "fmt"
)

type LogWorker struct {
    FileName string
    MaxSize int // In megabytes
    MaxBackups int // No of backups per worker
    MaxAge int // maximum number of days to retain old log files
}

func NewLogWorker(fileName string, maxSize int, maxBackups int, maxAge int) (lw *LogWorker) {
    return &LogWorker {fileName, maxSize, maxBackups, maxAge}
}

func (lw *LogWorker) Work(evChannel chan Event) {
    fmt.Println(lw.FileName)
    log.SetOutput(&lumberjack.Logger {
        Filename:   lw.FileName,
        MaxSize:    lw.MaxSize,
        MaxBackups: lw.MaxBackups,
        MaxAge:     lw.MaxAge,
    })
    log.SetFlags(0)

    for {
        event := <- evChannel
        log.Println(Csv(event))
    }
}

Please note that event is a struct which contains some string fields. Already there is a similar question in SO. When I tried to execute the goroutine in playground, it still prints the value from last go routine. The answer provided has some wait.Done. As my worker needs to run continuously I don't I think I can use it.

Please help me to find why my all goroutines (LogWorkers) are not used?

Answer 1

You are setting the log package's default global logger's output in each goroutine.

You probably want to do something more like:

func (lw *LogWorker) Work(evChannel chan Event) {
    fmt.Println(lw.FileName)
    lg := log.New(&lumberjack.Logger {
        Filename:   lw.FileName,
        MaxSize:    lw.MaxSize,
        MaxBackups: lw.MaxBackups,
        MaxAge:     lw.MaxAge,
    }, "", 0)

    for {
        event := <- evChannel
        lg.Println(Csv(event))
    }
}

this will give you a logger per goroutine.

In your version, you likely just had the last one to execute (probably last goroutine spawned, but not guaranteed)

And to improve a bit more, you probably also want your for loop written as:

for event := range evChannel {
    lg.Println(Csv(event))
}

This way, it will terminate the goroutine when the channel is closed rather than spin on empty values coming out of a closed channel. See here for reference