这个问题困扰两天了——Flume能检测到Spooldir目录中的文件增加,但是就是传不到hdfs中,总是报错:java.io.IOException:File type DataStream not supported
我的hadoop集群第一台是master,另外两台是slave。
flume的配置如下:
Name the components on this agent
agent1.sources = spooldirSource
agent1.channels = fileChannel
agent1.sinks = hdfsSink
Describe/configure the source
agent1.sources.spooldirSource.type=spooldir
agent1.sources.spooldirSource.spoolDir=/usr/local/flume/data/spooldir
Describe the sink
agent1.sinks.hdfsSink.type=hdfs
agent1.sinks.hdfsSink.hdfs.path=hdfs://192.168.174.128:9000/flume/%y-%m-%d/%H%M/%S
agent1.sinks.hdfsSink.hdfs.round = true
agent1.sinks.hdfsSink.hdfs.roundValue = 10
agent1.sinks.hdfsSink.hdfs.roundUnit = minute
agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
agent1.sinks.hdfsSink.hdfs.fileType=DataStream
Describe the channel
agent1.channels.fileChannel.type = file
agent1.channels.fileChannel.dataDirs=/hadoop/flume/datadir
Bind the source and sink to the channel
agent1.sources.spooldirSource.channels=fileChannel
agent1.sinks.hdfsSink.channel=fileChannel
另外,/hadoop/flume/datadir这个目录是在master上会自动创建的目录吗,不太懂。