Kafka生产者未在分区上分发消息

I am creating kafka topic as following:

kafka-topics --create --zookeeper xx.xxx.xx:2181 --replication-factor 2 --partitions 200 --topic test6 --config retention.ms=900000

and then I produce messages with golang using the following library:

    "gopkg.in/confluentinc/confluent-kafka-go.v1/kafka"

the producer configuration looks like this:

    for _, message := range bigslice {

    topic := "test6"
        p.Produce(&kafka.Message{
            TopicPartition: kafka.TopicPartition{Topic: &topic},
            Value:          []byte(message),
        }, nil)

        }

the problem that I've sent more than 200K messages but they all lands in partition 0.

what could be wrong in this situation?

1个回答

Messages with the same key are being added to the same partition. If this is not the case, then try to include Partition: kafka.PartitionAny:

for _, message := range bigslice {

    topic := "test6"
        p.Produce(&kafka.Message{
            TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
            Value: []byte(message),
        }, nil)

    }
dpl9717
dpl9717 设置为Partition后:kafka.PartitionAny我看到消息在分区上划分了。 但是,将相同的密钥添加到同一分区是我试图实现的行为。 如果要发送字符串,如何定义密钥? 还是应该在这种情况下使用avro并分配密钥?
大约一年之前 回复
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问
相关内容推荐