00点 2015-08-12 08:13 采纳率: 0%
浏览 4525

flume 的hdfs sink效率低的问题

哈喽,大家好,我现在遇到了一个问题。 我的flume向hdfs中写文件时,效率比较低 大约1G/3分钟 我单独测试时用put方式 1分钟能达到8G 如果用file sink也能达到1分钟1G 日志没有任何异常 只是DEBUG的时候发现每次提交一个块用时将近20秒 有高手能帮忙分析下是什么原因么

client.sources = r1
client.channels = c1
client.sinks = k1

client.sources.r1.type = spooldir
client.sources.r1.spoolDir = /var/data/tmpdata
client.sources.r1.fileSuffix = .COMPLETED
client.sources.r1.deletePolicy = never
client.sources.r1.batchSize = 500
client.sources.r1.channels = c1

client.channels.c1.type = memory
client.channels.c1.capacity = 1000000
client.channels.c1.transactionCapacity = 50000
client.channels.c1.keep-alive = 3

client.sinks.k1.type = hdfs
client.sinks.k1.hdfs.path = /flume/events/%Y%m%d/%H
client.sinks.k1.hdfs.useLocalTimeStamp = true
client.sinks.k1.hdfs.rollInterval = 3600
client.sinks.k1.hdfs.rollSize = 1000000000
client.sinks.k1.hdfs.rollCount = 0
client.sinks.k1.hdfs.batchSize = 500
client.sinks.k1.hdfs.callTimeout = 30000
client.sinks.k1.hdfs.fileType = DataStream
client.sinks.k1.channel = c1

12 Aug 2015 16:14:24,739 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:14:54,740 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:15:24,740 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:15:54,741 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:16:24,742 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:16:54,742 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:17:24,743 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:17:54,744 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:18:24,745 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:18:54,746 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
12 Aug 2015 16:19:24,746 DEBUG conf-file-poller-0 - Checking file:../conf/flume-client.conf for changes
日志没有问题 就是慢

  • 写回答

2条回答 默认 最新

  • babyxingqing 2017-03-22 14:50
    关注

    楼主,你好,想问下,这个问题你解决了吗,我也遇到这个问题,谢谢

    评论

报告相同问题?

悬赏问题

  • ¥15 maple软件,用solve求反函数出现rootof,怎么办?
  • ¥65 汇编语言除法溢出问题
  • ¥15 Visual Studio问题
  • ¥15 state显示变量是字符串形式,但是仍然红色,无法引用,并显示类型不匹配
  • ¥20 求一个html代码,有偿
  • ¥100 关于使用MATLAB中copularnd函数的问题
  • ¥20 在虚拟机的pycharm上
  • ¥15 jupyterthemes 设置完毕后没有效果
  • ¥15 matlab图像高斯低通滤波
  • ¥15 针对曲面部件的制孔路径规划,大家有什么思路吗