I wonder if/how it's possible to add dynamic labels. I don't know the key or the quantity of the labels which I would like to add to my gauge values.
What I tried
labelsContainers = []string{"node", "container", "pod", "qos", "namespace", "nodepool"}
// Requested resources
requestedContainerCPUCoresGauge = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: "pod_container_resource_requests",
Name: "cpu_cores",
Help: "Requested CPU cores in Kubernetes configuration",
},
labelsContainers)
for _, containerMetric := range containerMetrics {
containerLabels := prometheus.Labels{
"node": containerMetric.Node,
"container": containerMetric.Container,
"qos": containerMetric.Qos,
"pod": containerMetric.Pod,
"namespace": containerMetric.Namespace,
}
for key, value := range containerMetric.NodeLabels {
containerLabels[key] = value
}
requestedContainerCPUCoresGauge.With(containerLabels).Set(containerMetric.RequestedCPUCores)
requestedContainerRAMBytesGauge.With(containerLabels).Set(containerMetric.RequestedMemoryBytes)
}
The problem
This throws a panic. I assume this is because the prom client expects exactly those labels defined in labelsContainers
and doesn't allow further labels. How can I create a gauge which allows additional (unknown) labels?