dongsou8980
dongsou8980
2017-12-28 05:15

在fsnotify上以递归方式重新生成文件Remove / Rename(Golang)

  • filesystems
  • linux
  • concurrency
已采纳

The Goal:

I am attempting to monitor a file that is subject to being moved or deleted at any time. If and when it is, I'd like to re-generate this file so that an app can continue to write to it.

Attempted:

I have attempted to do this by implementing two functions, monitorFile() to listen for fsnotify events and send the removed filename over a channel to listen() which upon receiving the filepath string over un-buffered channel mvrm (move or rename), will recursively re-generate the file.

Observed behavior:

I can echo 'foo' >> ./inlogs/test.log and see a write notification, and can even rm ./inlogs/test.log (or mv) and see that the file is re-generated... but only once. If I rm or mv the file a second time, the file is not re-generated.

  • Strangely, the undesired behavior does not occur on local Mac OSx (System Version: macOS 10.13.2 (17C88), Kernel Version: Darwin 17.3.0), but does on two different Linux machines with builds:

Linux 3.13.0-32-generic #57-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

Linux 4.9.51-10.52.amzn1.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux

Diagnostics Attempted:

The differing behavior makes me think I have a race condition. However go build -race provides no output.

I wonder if done chan is receiving due to such a race condition?

Apologies that this is not 'Playground-able', but any advice or observation of where this might by racy or buggy would be welcome.

watcher.go:

package main

import (
    "os"
    "log"
    "fmt"

    "github.com/go-fsnotify/fsnotify"
)

//Globals
var mvrm chan string

func main() {
    mvrm = make(chan string)
    listen(mvrm)
    monitorFile("./inlogs/test.log", mvrm)
}

func listen(mvrm chan string) {
    go func() {
        for {
            select {
            case fileName := <-mvrm :
                fmt.Println(fileName)
                newFile, err := os.OpenFile(fileName, os.O_RDWR | os.O_CREATE | os.O_APPEND , 0666)
                if err == nil {
                    defer newFile.Close()

                    // Recursively re-spawn monitoring
                    go listen(mvrm)
                    go monitorFile(fileName, mvrm)
                } else {
                    log.Fatal("Err re-spawning file")
                }
            default:
                continue
            }
        }
    }()
}

func monitorFile(filepath string, mvrm chan string) {
    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        log.Fatal(err)
    }
    defer watcher.Close()

    done := make(chan bool)
    go func() {
        for {
            select {
            case event := <-watcher.Events:
                switch event.Op {
                case fsnotify.Write :
                    log.Println("Write!")
                    continue
                case fsnotify.Chmod :
                    log.Println("Chmod!")
                    continue
                case fsnotify.Remove, fsnotify.Rename :
                    log.Println("Moved or Deleted!")
                    mvrm <- event.Name
                    continue
                default:
                    log.Printf("Unknown: %v
", event.Op)
                    continue
                }
            case err := <-watcher.Errors:
                log.Println("Error:", err)
            }
        }
    }()

    err = watcher.Add(filepath)
    if err != nil {
        log.Fatal(err)
    }
    <-done
}

EDIT:

With some great feedback, I've paired this down. In Linux it is now re-generating the file as intended, but after monitoring with top, I see it is spawning a new PID every time the file is moved or deleted, so I do still have a leak. Suggestions as to how I might eliminate this behavior welcome.

https://play.golang.org/p/FrlkktoK2-s

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

2条回答

  • ds000001 ds000001 4年前

    Please see the code comments, most of the discussion in code comments.

    https://play.golang.com/p/qxq58h1nQjp

    Outside the golang universe, but facebook has a tool that does pretty much what you are looking for, just not as much go code fun :): https://github.com/facebook/watchman

    package main
    
    import (
        "log"
        "os"
    
        // couldn't find the go-fsnotify, this is what pops up on github
        "github.com/fsnotify/fsnotify"
    )
    
    func main() {
        monitorFile("./inlogs/test.log")
    }
    
    func monitorFile(filepath string) {
    
        // starting watcher
        watcher, err := fsnotify.NewWatcher()
        if err != nil {
            log.Fatal(err)
        }
        defer watcher.Close()
    
        // monitor events
        go func() {
            for {
                select {
                case event := <-watcher.Events:
                    switch event.Op {
                    case fsnotify.Create:
                        log.Println("Created")
    
                    case fsnotify.Write:
                        log.Println("Write")
    
                    case fsnotify.Chmod:
                        log.Println("Chmod")
    
                    case fsnotify.Remove, fsnotify.Rename:
                        log.Println("Moved or Deleted")
    
                        respawnFile(event.Name)
    
                        // add the file back to watcher, since it is removed from it
                        // when file is moved or deleted
                        log.Printf("add to watcher file:  %s
    ", filepath)
                        // add appears to be concurrently safe so calling from multiple go routines is ok
                        err = watcher.Add(filepath)
                        if err != nil {
                            log.Fatal(err)
                        }
    
                        // there is  not need to break the loop
                        // we just continue waiting for events from the same watcher
    
                    }
                case err := <-watcher.Errors:
                    log.Println("Error:", err)
                }
            }
        }()
    
        // add file to the watcher first time
        log.Printf("add to watcher 1st: %s
    ", filepath)
        err = watcher.Add(filepath)
        if err != nil {
            log.Fatal(err)
        }
    
        // to keep waiting forever, to prevent main exit
        // this is to replace the done channel
        select {}
    }
    
    func respawnFile(filepath string) {
        log.Printf("re creating file %s
    ", filepath)
    
        // you just need the os.Create()
        respawned, err := os.Create(filepath)
        if err != nil {
            log.Fatalf("Err re-spawning file: %v", filepath)
        }
        defer respawned.Close()
    
        // there is no need to call monitorFile again, it never returns
        // the call to "go monitorFile(filepath)" was causing another go routine leak
    }
    

    Have fun!

    点赞 评论 复制链接分享
  • douyan1896 douyan1896 4年前

    I do not have enough reputation to comment so I will pretend this to be an answer.

    On Linux, fsnotify use inotify to monitor fs changes, which means calling every add will run a syscall to create a new process and that's why you see it spawning PIDs.

    If that is a problem to you, it is also a common practice to monitor the dir of the file and filter events related to it. It means less syscall but more go code. Pick it yourself.

    点赞 评论 复制链接分享