dtzjvj3915 2019-09-03 11:45
浏览 139
已采纳

我们如何在kafka中快速编写单个消息(非批处理)?

I am new to Golang and Kafka and I am using segmentio kafka-go to connect to Kafka server using Golang. As of now I want to push every event of user in Kafka, so I want to push single message(and not in batch), but since the write operation provided by this library takes same time for either batch or single message, it is taking a lot of time. Is there any way of writing single message fast so that i can push million events in kafka in less time?

I have tested it for single message, and batch messages, it is taking same time (min was 10ms).

  • 写回答

2条回答 默认 最新

  • doudizhi947129 2019-09-09 01:07
    关注

    I think your problem is just the WriterConfig.

    For example, if your config looks like the example on segmentio/kafka-go docs:

    w := kafka.NewWriter(kafka.WriterConfig{
        Brokers:      []string{"localhost:9092"},
        Topic:        "topic-A",
        Balancer:     &kafka.LeastBytes{},
    })
    

    You could try setting batch size and batch timeout:

    w := kafka.NewWriter(kafka.WriterConfig{
        Brokers:      []string{"localhost:9092"},
        Topic:        "topic-A",
        Balancer:     &kafka.LeastBytes{},
        BatchSize:    1,
        BatchTimeout: 10 * time.Millisecond,
    })
    

    It happens because kafka-go waits by default 1 second until the batch reach the maximum size, which is by default 100 messages, as we can see in the code.

    Hope it helps you.


    Update: Be aware that sending the messages one by one slows the process. For example: sending 100 messages in batch took on my computer 0.0107s. Sending the same 100 messages one by one took 0.0244s.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(1条)

报告相同问题?

悬赏问题

  • ¥15 寻一个支付宝扫码远程授权登录的软件助手app
  • ¥15 解riccati方程组
  • ¥15 display:none;样式在嵌套结构中的已设置了display样式的元素上不起作用?
  • ¥15 使用rabbitMQ 消息队列作为url源进行多线程爬取时,总有几个url没有处理的问题。
  • ¥15 Ubuntu在安装序列比对软件STAR时出现报错如何解决
  • ¥50 树莓派安卓APK系统签名
  • ¥65 汇编语言除法溢出问题
  • ¥15 Visual Studio问题
  • ¥20 求一个html代码,有偿
  • ¥100 关于使用MATLAB中copularnd函数的问题