使用旧注册表使Filebeat从旧文件偏移开始

Question

I'm running Filebeat (used to be know as "logstash-forwarder") on a docker container using prima/filebeat image.

The log files are located in a volume loaded to the container and I want to be able to remove the container and rerun it without it re-sending the logs to the logstash.

I tried to load the /.filebeat registry file as a volume so it will be reloaded on startup but all I get are these errors:

2016/02/03 13:47:29.107457 file_other.go:39: ERR Rotate error: rename /.filebeat.new /.filebeat: device or resource busy
2016/02/03 13:47:29.107788 registrar.go:105: ERR Writing of registry returned error: rename /.filebeat.new /.filebeat: device or resource busy. Continuing..

Does anybody happen to know how to do such a thing?

Answer 1

With Filebeat version 1.2.3 (other versions may be the same, version 1.3 just came out a few days ago and I've not tried it yet) you will need to specify the path to the registry file. The registry file is the file where Filebeat keeps it read offset (in other words, the current read point in the log(s) file(s) it is processing).

You need to specify the registry path in your filebeat.yml file, after the prospectors section.

Something like:

filebeat:
  prospectors:
    -
      document_type: wildfly-server
      input_type: log
      paths:
        - /path/to/my/log.log
      multiline:
         pattern: 'your pattern'
         negate: true
         match: after

  registry_file: /data/.filebeat

logging:
  level: debug
  to_syslog: true

output:
  logstash:
    hosts:
      - "127.0.0.1:5000"

Then in your docker compose file you need to mount a volume pointing to your registry_file path. Something like:

filebeat:
  image: prima/filebeat:latest
  volumes:
     - /data/filebeat:/data
  hostname: qa

Then you should see in the host the .filebeat file located in the /data/filebeat folder.