dongsou8980
2017-12-28 05:15
浏览 151

在fsnotify上以递归方式重新生成文件Remove / Rename(Golang)

The Goal:

I am attempting to monitor a file that is subject to being moved or deleted at any time. If and when it is, I'd like to re-generate this file so that an app can continue to write to it.

Attempted:

I have attempted to do this by implementing two functions, monitorFile() to listen for fsnotify events and send the removed filename over a channel to listen() which upon receiving the filepath string over un-buffered channel mvrm (move or rename), will recursively re-generate the file.

Observed behavior:

I can echo 'foo' >> ./inlogs/test.log and see a write notification, and can even rm ./inlogs/test.log (or mv) and see that the file is re-generated... but only once. If I rm or mv the file a second time, the file is not re-generated.

  • Strangely, the undesired behavior does not occur on local Mac OSx (System Version: macOS 10.13.2 (17C88), Kernel Version: Darwin 17.3.0), but does on two different Linux machines with builds:

Linux 3.13.0-32-generic #57-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

Linux 4.9.51-10.52.amzn1.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux

Diagnostics Attempted:

The differing behavior makes me think I have a race condition. However go build -race provides no output.

I wonder if done chan is receiving due to such a race condition?

Apologies that this is not 'Playground-able', but any advice or observation of where this might by racy or buggy would be welcome.

watcher.go:

package main

import (
    "os"
    "log"
    "fmt"

    "github.com/go-fsnotify/fsnotify"
)

//Globals
var mvrm chan string

func main() {
    mvrm = make(chan string)
    listen(mvrm)
    monitorFile("./inlogs/test.log", mvrm)
}

func listen(mvrm chan string) {
    go func() {
        for {
            select {
            case fileName := <-mvrm :
                fmt.Println(fileName)
                newFile, err := os.OpenFile(fileName, os.O_RDWR | os.O_CREATE | os.O_APPEND , 0666)
                if err == nil {
                    defer newFile.Close()

                    // Recursively re-spawn monitoring
                    go listen(mvrm)
                    go monitorFile(fileName, mvrm)
                } else {
                    log.Fatal("Err re-spawning file")
                }
            default:
                continue
            }
        }
    }()
}

func monitorFile(filepath string, mvrm chan string) {
    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        log.Fatal(err)
    }
    defer watcher.Close()

    done := make(chan bool)
    go func() {
        for {
            select {
            case event := <-watcher.Events:
                switch event.Op {
                case fsnotify.Write :
                    log.Println("Write!")
                    continue
                case fsnotify.Chmod :
                    log.Println("Chmod!")
                    continue
                case fsnotify.Remove, fsnotify.Rename :
                    log.Println("Moved or Deleted!")
                    mvrm <- event.Name
                    continue
                default:
                    log.Printf("Unknown: %v
", event.Op)
                    continue
                }
            case err := <-watcher.Errors:
                log.Println("Error:", err)
            }
        }
    }()

    err = watcher.Add(filepath)
    if err != nil {
        log.Fatal(err)
    }
    <-done
}

EDIT:

With some great feedback, I've paired this down. In Linux it is now re-generating the file as intended, but after monitoring with top, I see it is spawning a new PID every time the file is moved or deleted, so I do still have a leak. Suggestions as to how I might eliminate this behavior welcome.

https://play.golang.org/p/FrlkktoK2-s

  • 写回答
  • 好问题 提建议
  • 关注问题
  • 收藏
  • 邀请回答

2条回答 默认 最新

  • ds000001 2017-12-28 22:06
    已采纳

    Please see the code comments, most of the discussion in code comments.

    https://play.golang.com/p/qxq58h1nQjp

    Outside the golang universe, but facebook has a tool that does pretty much what you are looking for, just not as much go code fun :): https://github.com/facebook/watchman

    package main
    
    import (
        "log"
        "os"
    
        // couldn't find the go-fsnotify, this is what pops up on github
        "github.com/fsnotify/fsnotify"
    )
    
    func main() {
        monitorFile("./inlogs/test.log")
    }
    
    func monitorFile(filepath string) {
    
        // starting watcher
        watcher, err := fsnotify.NewWatcher()
        if err != nil {
            log.Fatal(err)
        }
        defer watcher.Close()
    
        // monitor events
        go func() {
            for {
                select {
                case event := <-watcher.Events:
                    switch event.Op {
                    case fsnotify.Create:
                        log.Println("Created")
    
                    case fsnotify.Write:
                        log.Println("Write")
    
                    case fsnotify.Chmod:
                        log.Println("Chmod")
    
                    case fsnotify.Remove, fsnotify.Rename:
                        log.Println("Moved or Deleted")
    
                        respawnFile(event.Name)
    
                        // add the file back to watcher, since it is removed from it
                        // when file is moved or deleted
                        log.Printf("add to watcher file:  %s
    ", filepath)
                        // add appears to be concurrently safe so calling from multiple go routines is ok
                        err = watcher.Add(filepath)
                        if err != nil {
                            log.Fatal(err)
                        }
    
                        // there is  not need to break the loop
                        // we just continue waiting for events from the same watcher
    
                    }
                case err := <-watcher.Errors:
                    log.Println("Error:", err)
                }
            }
        }()
    
        // add file to the watcher first time
        log.Printf("add to watcher 1st: %s
    ", filepath)
        err = watcher.Add(filepath)
        if err != nil {
            log.Fatal(err)
        }
    
        // to keep waiting forever, to prevent main exit
        // this is to replace the done channel
        select {}
    }
    
    func respawnFile(filepath string) {
        log.Printf("re creating file %s
    ", filepath)
    
        // you just need the os.Create()
        respawned, err := os.Create(filepath)
        if err != nil {
            log.Fatalf("Err re-spawning file: %v", filepath)
        }
        defer respawned.Close()
    
        // there is no need to call monitorFile again, it never returns
        // the call to "go monitorFile(filepath)" was causing another go routine leak
    }
    

    Have fun!

    已采纳该答案
    评论
    解决 无用
    打赏 举报
  • douyan1896 2017-12-29 04:37

    I do not have enough reputation to comment so I will pretend this to be an answer.

    On Linux, fsnotify use inotify to monitor fs changes, which means calling every add will run a syscall to create a new process and that's why you see it spawning PIDs.

    If that is a problem to you, it is also a common practice to monitor the dir of the file and filter events related to it. It means less syscall but more go code. Pick it yourself.

    评论
    解决 无用
    打赏 举报

相关推荐 更多相似问题