问题遇到的现象和发生背景
Server A [192.168.17.78]:实时存储每天的操作日志(示例2022-04-13.log)
Server B [192.168.50.166]:搭建了实时收集的环境(JDK、Scala、Maven、Hadoop、Zookeeper、Flume)
Server A是实际的生产环境,Server B是数据处理的测试环境。
所以Server B只能去监控Server A上的log文本数据。
Server A是不能去搭建环境的。
问题相关代码
Flume的access-collect.conf编写如下:
a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = TAILDIR
a1.sources.r1.channels = c1
# 官网上的TAILDIR ,没有标注支持bind和port
a1.sources.r1.bind = 192.168.17.78
a1.sources.r1.port = 4141
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /www/wwwroot/hs.chnau99999.com/jar/shopLogs/2022-04-13.log
a1.sources.r1.headers.f1.headerKey1 = shop
a1.sources.r1.fileHeader = true
a1.sinks.k1.type = logger
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
运行结果及报错内容
2022-04-13 15:25:49,262 (lifecycleSupervisor-1-7) [INFO - org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:92)] r1 TaildirSource source starting with directory: {f1=/www/wwwroot/hs.chnau99999.com/jar/shopLogs/2022-04-13.log}
2022-04-13 15:25:49,262 (lifecycleSupervisor-1-7) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)] Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: /root/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.IllegalStateException: Directory does not exist: /www/wwwroot/hs.chnau99999.com/jar/shopLogs
我的解答思路和尝试过的方法
想的方法都尝试过了。有没有测试成功的方法?
我想要达到的结果
Server B上搭建实时监控Server A中某文件的环境,并获实时取该文件中新增数据,存储在Server C的数据库中。