I use Go and did a server upgrade to kafka 0.11 and sarama client to v1.13.0. I am also using sarama-cluster instead of wvanbergen/kafka for consumer group management now. Note that kafka upgrade works with wvanbergen/kafka and old version of sarama client(v1.8)
I see that my consumers are not consuming any messages and there is line in log that says that the consumer group is dead.
Here are the kafka logs:
2017-10-12 19:39:05,030 - INFO [kafka-request-handler-7:Logging$class@70] - [GroupCoordinator 1]: Preparing to rebalance group job_event_consumer with old generation 0 (__consumer_offsets-5)
2017-10-12 19:39:05,032 - INFO [kafka-request-handler-3:Logging$class@70] - [GroupCoordinator 1]: Preparing to rebalance group user_forwarder with old generation 0 (__consumer_offsets-20)
2017-10-12 19:39:08,031 - INFO [executor-Rebalance:Logging$class@70] - [GroupCoordinator 1]: Stabilized group job_event_consumer generation 1 (__consumer_offsets-5)
2017-10-12 19:39:08,032 - INFO [executor-Rebalance:Logging$class@70] - [GroupCoordinator 1]: Stabilized group user_forwarder generation 1 (__consumer_offsets-20)
2017-10-12 19:39:08,032 - INFO [kafka-request-handler-5:Logging$class@70] - [GroupCoordinator 1]: Assignment received from leader for group job_event_consumer for generation 1
2017-10-12 19:39:08,033 - INFO [kafka-request-handler-5:Logging$class@70] - [GroupCoordinator 1]: Assignment received from leader for group user_forwarder for generation 1
2017-10-12 19:39:15,743 - INFO [kafka-request-handler-1:Logging$class@70] - [GroupCoordinator 1]: Preparing to rebalance group user_forwarder with old generation 1 (__consumer_offsets-20)
2017-10-12 19:39:15,744 - INFO [kafka-request-handler-2:Logging$class@70] - [GroupCoordinator 1]: Preparing to rebalance group job_event_consumer with old generation 1 (__consumer_offsets-5)
2017-10-12 19:39:45,743 - INFO [executor-Rebalance:Logging$class@70] - [GroupCoordinator 1]: Stabilized group user_forwarder generation 2 (__consumer_offsets-20)
2017-10-12 19:39:45,744 - INFO [executor-Rebalance:Logging$class@70] - [GroupCoordinator 1]: Stabilized group job_event_consumer generation 2 (__consumer_offsets-5)
2017-10-12 19:39:47,043 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Member mothership_staging-2c8c20d0-4a6b-4b41-b208-e822553bc9c7 in group user_forwarder has failed, removing it from the group
2017-10-12 19:39:47,044 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Preparing to rebalance group user_forwarder with old generation 2 (__consumer_offsets-20)
2017-10-12 19:39:47,272 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Member mothership_staging-31a167c1-f70e-4af6-a21d-da500149c084 in group job_event_consumer has failed, removing it from the group
2017-10-12 19:39:47,272 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Preparing to rebalance group job_event_consumer with old generation 2 (__consumer_offsets-5)
2017-10-12 19:40:15,743 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Member mothership_staging-7c1655dc-a7bc-4d39-b632-9af33c7ba934 in group user_forwarder has failed, removing it from the group
2017-10-12 19:40:15,744 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Group user_forwarder with generation 3 is now empty (__consumer_offsets-20)
2017-10-12 19:40:15,745 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Member mothership_staging-6ef84d2e-fca7-4c7c-920f-2b0d4ef8ec86 in group job_event_consumer has failed, removing it from the group
2017-10-12 19:40:15,746 - INFO [executor-Heartbeat:Logging$class@70] - [GroupCoordinator 1]: Group job_event_consumer with generation 3 is now empty (__consumer_offsets-5)
2017-10-12 19:44:11,330 - INFO [group-metadata-manager-0:Logging$class@70] - [Group Metadata Manager on Broker 1]: Group job_event_consumer transitioned to Dead in generation 3
2017-10-12 19:44:11,331 - INFO [group-metadata-manager-0:Logging$class@70] - [Group Metadata Manager on Broker 1]: Group user_forwarder transitioned to Dead in generation 3
There are no errors in consumer or any other logs. What could be causing this issue?
