dongliming2416 2017-02-01 15:46
浏览 736
已采纳

如何在Golang Kafka 10中将使用者设置为从特定偏移量开始

My need is to make the producer to start from the last message it processed before it crashed. Fortunately I am in the case of having only one topic, with one partition and one consumer.

To do so I tried https://github.com/Shopify/sarama but it doesn't seems to be available yet. I am now using https://godoc.org/github.com/bsm/sarama-cluster, which allow me to commit every message offset.

I cannot retrieve the last committed offset I cannot figure out how to make a sarama consumer to start from said offset. The only parameter I've found so far is Config.Producer.Offsets.Initial.

  1. How to retrieve the last committed offset?
  2. How to make the consumer start from the last message whose offset has been committed? OffsetNewest will make it start from the last message produced, not the last processed b the consumer.
  3. Is it possible to do so using only Shopify/sarama and not bsm/sarama-cluster ?

Thank in advance

P.S. I am using Kafka 10.0, so the offsets are stores in a kafka and not in zookeeper.

EDIT1: Partial solution: fetch all the messages since sarama.OffsetOldest and skip all of them until we found a non processed one.

  • 写回答

1条回答 默认 最新

  • dowb58485 2018-01-18 13:27
    关注

    If offset was already saved for a partition, sarama-cluster will resume consumption from that offset. The Config.Producer.Offsets.Initial option is used only if no saved offset is present (first run for a consumer group).

    You can verify this by adding the following line at the beginning of your main() function:

    sarama.Logger = log.New(os.Stdout, "sarama: ", log.LstdFlags)
    

    Then you'll see something like the following in the output:

    cluster/consumer CID-17db1be4-a162-411c-a106-4d198191176a consume sample/0 from 12

    The 12 in that is the offset Sarama is going to start from for that partition (sample/0).

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥15 jupyterthemes 设置完毕后没有效果
  • ¥15 matlab图像高斯低通滤波
  • ¥15 针对曲面部件的制孔路径规划,大家有什么思路吗
  • ¥15 钢筋实图交点识别,机器视觉代码
  • ¥15 如何在Linux系统中,但是在window系统上idea里面可以正常运行?(相关搜索:jar包)
  • ¥50 400g qsfp 光模块iphy方案
  • ¥15 两块ADC0804用proteus仿真时,出现异常
  • ¥15 关于风控系统,如何去选择
  • ¥15 这款软件是什么?需要能满足我的需求
  • ¥15 SpringSecurityOauth2登陆前后request不一致