The actual intent was to capture the IP of the instance and list all of them as a dynamic variable via a query and then enable all the node metrics via the IPs. This was a bad approach since lot of prometheus node-metrics are exported with label node=ip-XX-XX-XXX-XXX-<region>-.compute.internal
(where XX are digits). Also grafana variables reports match of only the 1st sub-group - ideally it should be all the subgroups.
However, even if i were to capture the subgroup say XX.XX.XXX.XXX this would mean for every metric that I need to capture and graph I would need to address for matches for all the IPs in the EKS nodes - this is not possible especially if we want to enable "All" variable in which grafana replaces the variable call with regex of all the variables matches. Ex. if my k8s node had following IPs:
* 10.10.0.1
* 10.10.0.2
* 10.10.0.3
and I call this variable as Node
Grafana will list following variables:
* All
* 10.10.0.1
* 10.10.0.2
* 10.10.0.3
On selecting All
grafana will do replace of $Node to ./*(10.10.0.1|10.10.0.2|10.10.0.3)*/
However if i had a Query for getting the node info : kube_pod_info{node=~"$Node"}
-> this would fail because node name is in a different format we may try and solve it by using label_replace
but again we cannot address using $Node
in case All
variable is used.
Conclusion : use the variable $Node
(i.e.
* ip-10-10-0-1.us-sdlls-as.compute.internal
* ip-10-10-0-2.us-sdlls-as.compute.internal
* ip-10-10-0-3.us-sdlls-as.compute.internal
) - as-is this would make all other panels much more convenient and easy to configure.