douluan5738 2017-11-02 09:07
浏览 178
已采纳

使用Golang Sarama软件包无法使用来自本地运行的Kafka服务器的消息

I am making a simple Telegram bot that would read messages from a local Kafka server and print it out to a chat. Both zookeeper and kafka server config files are at their defaults. Console consumer works. The problem rises when I try to consume messages from code using Golang Sarama package. Before I added these lines:

case err := <-pc.Errors(): log.Panic(err)

the program only printed the messages once, after which it would stall. Now it panics prinitng this to the log: kafka: error while consuming test1/0: kafka: broker not connected

Here's the code:

    type kafkaResponse struct {
        telega  *tgbotapi.Message
        message []byte
    }

    type kafkaRequest struct {
        telega *tgbotapi.Message
        topic  string
    }    
    var kafkaBrokers = []string{"localhost:9092"}
    func main() {
                //channels for request response
                var reqChan = make(chan kafkaRequest)
                var respChan = make(chan kafkaResponse)

                //starting kafka client routine to listen to topic channnel
                go consumer(reqChan, respChan, kafkaBrokers)

                //bot thingy here
                bot, err := tgbotapi.NewBotAPI(token)
                if err != nil {
                    log.Panic(err)
                }
                bot.Debug = true
                log.Printf("Authorized on account %s", bot.Self.UserName)
                u := tgbotapi.NewUpdate(0)
                u.Timeout = 60
                updates, err := bot.GetUpdatesChan(u)
                for {
                    select {
                    case update := <-updates:
                        if update.Message == nil {
                            continue
                        }
                        switch update.Message.Text {

                        case "Topic: test1":
                            topic := "test1"
                            reqChan <- kafkaRequest{update.Message, topic}
                        }
                    case response := <-respChan:
                        bot.Send(tgbotapi.NewMessage(response.telega.Chat.ID, string(response.message)))
                    }

                }

here's the consumer.go:

 func consumer(reqChan chan kafkaRequest, respChan chan kafkaResponse, brokers []string) {
            config := sarama.NewConfig()
            config.Consumer.Return.Errors = true

            // Create new consumer
            consumer, err := sarama.NewConsumer(brokers, config)
            if err != nil {
                panic(err)
            }
            defer func() {
                if err := consumer.Close(); err != nil {
                    panic(err)
                }
            }()

            select {
            case request := <-reqChan:
                //get all partitions on the given topic
                partitionList, err := consumer.Partitions(request.topic)
                if err != nil {
                    fmt.Println("Error retrieving partitionList ", err)
                }

                initialOffset := sarama.OffsetOldest
                for _, partition := range partitionList {
                    pc, _ := consumer.ConsumePartition(request.topic, partition, initialOffset)

                    go func(pc sarama.PartitionConsumer) {
                        for {
                            select {
                            case message := <-pc.Messages():
                                respChan <- kafkaResponse{request.telega, message.Value}
                            case err := <-pc.Errors():
                                log.Panic(err)
                            }
                        }
                    }(pc)
                }
            }
        }
  • 写回答

1条回答 默认 最新

  • douzhong5902 2017-11-02 11:27
    关注

    You are closing your consumer after setting up all the PartitionConsumers in the code

    defer func() {
                if err := consumer.Close(); err != nil {
                    panic(err)
                }
            }()
    

    However, the documentation specifies that you should only close the consumer after all the PartitionConsumers have been closed.

    // Close shuts down the consumer. It must be called after all child
    // PartitionConsumers have already been closed.
    Close() error
    

    I would recommend you add a sync.WaitGroup to the function go func(pc sarama.PartitionConsumer) {

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥50 关于#html5#的问题:H5页面用户手机返回的时候跳转到指定页面例如(语言-javascript)
  • ¥15 无法使用此凭据登录,因为你的域不可用,如何解决?(标签-Windows)
  • ¥15 yolov9的训练时间
  • ¥15 二叉树遍历没有报错但无法正常运行
  • ¥15 在linux系统下vscode运行robocup3d上场球员报错
  • ¥15 Python语言实验
  • ¥15 SAP HANA SQL 增加合计行
  • ¥20 用C#语言解决一个英文打字练习器,有偿
  • ¥15 srs-sip外部服务 webrtc支持H265格式
  • ¥15 在使用abaqus软件中,继承到assembly里的surfaces怎么使用python批量调动