flume 的hdfs sink效率低的问题

哈喽,大家好,我现在遇到了一个问题。 我的flume向hdfs中写文件时,效率比较低 大约1G/3分钟 我单独测试时用put方式 1分钟能达到8G 如果用file sink也能达到1分钟1G 日志没有任何异常 只是DEBUG的时候发现每次提交一个块用时将近20秒 有高手能帮忙分析下是什么原因么

client.sources = r1
client.channels = c1
client.sinks = k1

client.sources.r1.type = spooldir
client.sources.r1.spoolDir = /var/data/tmpdata
client.sources.r1.fileSuffix = .COMPLETED
client.sources.r1.deletePolicy = never
client.sources.r1.batchSize = 500
client.sources.r1.channels = c1

client.channels.c1.type = memory
client.channels.c1.capacity = 1000000
client.channels.c1.transactionCapacity = 50000
client.channels.c1.keep-alive = 3

client.sinks.k1.type = hdfs
client.sinks.k1.hdfs.path = /flume/events/%Y%m%d/%H
client.sinks.k1.hdfs.useLocalTimeStamp = true
client.sinks.k1.hdfs.rollInterval = 3600
client.sinks.k1.hdfs.rollSize = 1000000000
client.sinks.k1.hdfs.rollCount = 0
client.sinks.k1.hdfs.batchSize = 500
client.sinks.k1.hdfs.callTimeout = 30000
client.sinks.k1.hdfs.fileType = DataStream
client.sinks.k1.channel = c1

12 Aug 2015 16:14:24,739 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:14:54,740 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:15:24,740 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:15:54,741 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:16:24,742 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:16:54,742 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:17:24,743 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:17:54,744 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:18:24,745 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:18:54,746 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:19:24,746 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
日志没有问题 就是慢

2个回答

楼主,你好,想问下,这个问题你解决了吗,我也遇到这个问题,谢谢

多起几个客户端压(多管道)。batch ×event长度=1M试试

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问