如何显示使用Golang客户端库从Kubernetes中运行的所有Pod中捕获的Prometheus中的自定义应用程序指标

Question

I am trying to get some custom application metrics captured in golang using the prometheus client library to show up in Prometheus.

I have the following working:

• I have a go application which is exposing metrics on localhost:8080/metrics as described in this article:

  https://godoc.org/github.com/prometheus/client_golang/prometheus

• I have a kubernates minikube running which has Prometheus, Grafana and AlertManager running using the operator from this article:

  https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus

• I created a docker image for my go app, when I run it and go to localhost:8080/metrics I can see the prometheus metrics showing up in a browser.

• I use the following pod.yaml to deploy my docker image to a pod in k8s

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  labels:
    zone: prod
    version: v1
  annotations:
   prometheus.io/scrape: 'true'
   prometheus.io/port: '8080'

spec:
   containers:
    - name: my-container
      image: name/my-app:latest
      imagePullPolicy: IfNotPresent
      ports:
      - containerPort: 8080

• If I connect to my pod using:

kubectl exec -it my-app-pod -- /bin/bash

then do wget on "localhost:8080/metrics", I can see my metrics

So far so good, here is where I am hitting a wall. I could have multiple pods running this same image. I want to expose all the images to prometheus as targets. How do I configure my pods so that they show up in prometheus so I can report on my custom metrics?

Thanks for any help offered!

Answer 1

You need 2 things:

• a ServiceMonitor for the Prometheus Operator, which specifies which services will be scraped for metrics
• a Service which matches the ServiceMonitor and points to your pods

There is an example in the docs over here: https://coreos.com/operators/prometheus/docs/latest/user-guides/running-exporters.html