在做毕业设计过程中,在大同kafka-flume-hdfs通道时,hdfs无法接收到flume采集的数据,查看flume启动日志文件发现报错:
No configuration directory set! Use --conf <dir> to override.
2021-12-17 13:49:27,952 WARN conf.FlumeConfiguration: Agent configuration for 'a1' has no configfilters.
这是我的flume配置文件内容
## 组件
a1.sources=r1
a1.channels=c1
a1.sinks=k1
## source1
a1.sources.r1.type=org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.batchSize=5000
a1.sources.r1.batchDurationMillis=2000
a1.sources.r1.kafka.bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.sources.r1.kafka.topics=topic_log
## 时间戳拦截器
# a1.sources.r1.interceptors=i1
# a1.sources.r1.interceptors.i1.type=cn.fzy.flume.interceptor.TimeStampInterceptor$Builder
## channel1
a1.channels.c1.type=file
a1.channels.c1.checkpointDir=/opt/module/flume/checkpoint/behavior1
a1.channels.c1.dataDirs=/opt/module/flume/data/behavior1/
## sink1
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=/origin_data/gmall/log/topic_log/%Y-%m-%d
a1.sinks.k1.hdfs.filePrefix=log-
a1.sinks.k1.hdfs.round = false
## 控制生成的小文件
a1.sinks.k1.hdfs.rollInterval = 10
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.rollCount = 0
## 控制输出文件是原生文件。
a1.sinks.k1.hdfs.fileType = CompressedStream
a1.sinks.k1.hdfs.codeC = lzop
## 拼装
a1.sources.r1.channels = c1
a1.sinks.k1.channel= c1
这是我的命令语句
nohup /opt/module/flume/bin/flume-ng agent \
--conf-file /opt/module/flume/conf/kafka-flume-hdfs.conf \
--name a1 -Dflume.root.logE >/opt/module/flume/log2.txt 2>&1 &
面向百度编程,有问题直接百度,许多的分享说是配置的名字是 a1 在命令语句就要是 a1,但是我的命令语句中的名字跟配置文件中的组件名字是一样的,是在是不知道什么原因,请各位帮忙看看,谢谢了