I tried running the official metricbeat docker image as described here (https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html) on a GCP kubernetes cluster as a deamonset and changed the settings so it should route traffic to the existing elastic search pod, but I keep getting the error:
2018-02-22T14:04:54.515Z WARN transport/tcp.go:36 DNS lookup failure "elasticsearch-logging.kube-system.svc.cluster.local": lookup elasticsearch-logging.kube-system.svc.cluster.local: no such host
2018-02-22T14:04:55.516Z ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch-logging.kube-system.svc.cluster.local:9200: lookup elasticsearch-logging.kube-system.svc.cluster.local: no such host
2018-02-22T14:04:55.517Z WARN transport/tcp.go:36 DNS lookup failure "elasticsearch-logging.kube-system.svc.cluster.local": lookup elasticsearch-logging.kube-system.svc.cluster.local: no such host
2018-02-22T14:04:57.517Z ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch-logging.kube-system.svc.cluster.local:9200: lookup elasticsearch-logging.kube-system.svc.cluster.local: no such host
2018-02-22T14:04:57.519Z WARN transport/tcp.go:36 DNS lookup failure "elasticsearch-logging.kube-system.svc.cluster.local": lookup elasticsearch-logging.kube-system.svc.cluster.local: no such host
2018-02-22T14:05:01.519Z ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch-logging.kube-system.svc.cluster.local:9200: lookup elasticsearch-logging.kube-system.svc.cluster.local: no such host
2018-02-22T14:05:01.532Z WARN transport/tcp.go:36 DNS lookup failure "elasticsearch-logging.kube-system.svc.cluster.local": lookup elasticsearch-logging.kube-system.svc.cluster.local: no such host
The hostname is fine, because other pods are successfully pushing data to elastic. Now, after some research this turns out to be an issue of the Golang DNS resolver (not metricbeat itself). Anyone else running into this issue? Anyone a solution?
