drgawfsf1069
drgawfsf1069
2019-07-26 16:04

解析文件中的数据时大小不一的地图的地图

已采纳

I am building an API for Nagios that is inspired by this project. I've started recreating the code that reads the status.dat file and stores the data in a number of objects which are then used to create hosts, services, info dictionaries which is found in the core.py file.

Below is my Go version of the python code which seems to work as expected. It is still in its early stages so I apologise for any coding bad practice.

var mu = &sync.RWMutex{}

func openStatusFile() *os.File {
    file, err := os.Open("/usr/local/nagios/var/status.dat")
    if err != nil {
    }
    return file
}

func nextStanza() <-chan map[string]string {

    myChannel := make(chan map[string]string)

    scanner := bufio.NewScanner(openStatusFile())

    current := make(map[string]string)

    go func() {
        for scanner.Scan() {
            mainline := scanner.Text()
            line := strings.TrimSpace(mainline)
            if strings.HasSuffix(line, "{") {
                if len(current) != 0 {
                    myChannel <- current
                }
                result := strings.SplitN(line, " ", 2)
                mu.Lock()
                current["type"] = result[0]
                mu.Unlock()
            } else if strings.Contains(line, "=") {
                result := strings.SplitN(line, "=", 2)
                key := result[0]
                val := result[1]
                mu.Lock()
                current[key] = val
                mu.Unlock()
            }
        }
        close(myChannel)
    }()
    return myChannel
}

In the main function, I create my nested map to hold just the host data for now and this completes without any complaints. The problem I am getting, is that when I check the length of this map, I am expecting to see 104 hosts, but I get different results every time I run this test file.

func main() {

    hoststatus := nextStanza()

    hosts := make(map[string]map[string]string)
    // services := make(map[string]map[string]map[string]string)
    var host string
    // var service string

    for obj := range hoststatus {
        var hostPlaceHolder string
        var typePlaceHolder string

        mu.Lock()
        hostPlaceHolder = obj["host_name"]
        mu.Unlock()

        if hostPlaceHolder != "" {
            host = hostPlaceHolder
        }

        mu.Lock()
        typePlaceHolder = obj["type"]
        mu.Unlock()

        if typePlaceHolder == "hoststatus" {
            mu.Lock()
            hosts[host] = obj
            mu.Unlock()
        }
    }
    fmt.Println(len(hosts))
}

First run:

$ go run -race mytest.go
93

Second run:

$ go run -race mytest.go
95

Third run:

$ go run -race mytest.go
63

You get the idea.

I feel the issue is to do with the map, because if I just print the hosts without putting them into a map, I see all the hosts I am expecting. What would be the reason for the map be a different size on each run?

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

2条回答

  • dqg63264 dqg63264 2年前

    I was able to resolve my issue by emptying the current map after it is sent to the channel.

    myChannel <- current
    current = make(map[string]string)
    

    Then in the main() function after the for obj := range hoststatus loop, I placed this data into a separate map to then work from.

    hostStatusMap := make(map[string]string)
        for k, v := range obj {
            hostStatusMap[k] = v
        }
    

    I was also able to remove the locks scattered throughout the code and it now returns the correct length of hosts on each run.

    点赞 评论 复制链接分享
  • dongse5408 dongse5408 2年前

    Your code has the following race condition

    func nextStanza() <-chan map[string]string {
    
        myChannel := make(chan map[string]string)
    
        scanner := bufio.NewScanner(openStatusFile())
    
        current := make(map[string]string)
    
        go func() {
            for scanner.Scan() {
                mainline := scanner.Text()
                line := strings.TrimSpace(mainline)
                if strings.HasSuffix(line, "{") {
                    if len(current) != 0 {
                        myChannel <- current
                    }
                    result := strings.SplitN(line, " ", 2)
                    mu.Lock()
                    current["type"] = result[0]
                    mu.Unlock()
                } else if strings.Contains(line, "=") {
                    result := strings.SplitN(line, "=", 2)
                    key := result[0]
                    val := result[1]
                    mu.Lock()
                    current[key] = val
                    mu.Unlock()
                }
            }
            close(myChannel)
        }()
        return myChannel
    }
    

    When you start a goroutine on the anonymous function you are not creating a WaitGroup for it. What this means is that the function nextStanza() is going to initiate the goroutine then return without waiting for the anonymous goroutine to terminate - thus ending the goroutine upon the parent function's closure.

    I would suggest using a waitgroup, this way you can guarantee that the anonymous function terminates.

    A simple example illustrating what is occurring:

    With a race condition

    import (
        "fmt"
        "time"
        // "sync"
        )
    
    func main() {
        go func() {
            time.Sleep(3 * time.Second)
            fmt.Println("hai")
        }()
        return
    }
    

    Without a race condition

    import (
        "fmt"
        "time"
        "sync"
        )
    
    func main() {
        var wg sync.WaitGroup
        wg.Add(1)
        go func() {
            time.Sleep(3 * time.Second)
            fmt.Println("hai")
            wg.Done()
        }()
        wg.Wait()
        return
    }
    
    点赞 评论 复制链接分享