在golang的单独端口中公开Prometheus指标

Question

In a microservice running gin-gonic, I'm trying to collect metrics in a router and expose them on a second one, but it seems this is not possible. Does anyone have experience in it?

All the documentation I found set the prometheus handler into the same application router.

Expected result

See promhttp_metric_handler_requests_total{code="200"} increasing for each visit of the :8000/test route

Actual result

promhttp_metric_handler_requests_total{code="200"} increase only for each visit of the :7100/metrics route

Example code

(following is a full application that can explain the issue)

package main

import (
    "github.com/gin-gonic/gin"
    "github.com/zsais/go-gin-prometheus"
    "net/http"
)

func main() {
    router1 := gin.New()
    router1.GET("/test", func(c *gin.Context) {
        c.String(http.StatusOK, "it works")
    })

    p := ginprometheus.NewPrometheus("test")
    router1.Use(p.HandlerFunc())

    router2 := gin.New()
    p.SetMetricsPath(router2)

    go func() { router2.Run(":7100") }()
    router1.Run(":8000")
}

Result

The result of the previous code, after I loaded several time the endpoint :8000/test is the following and you can see that statistics of the namespace test are "empty"

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0.0"} 0.0
go_gc_duration_seconds{quantile="0.25"} 0.0
go_gc_duration_seconds{quantile="0.5"} 0.0
go_gc_duration_seconds{quantile="0.75"} 0.0
go_gc_duration_seconds{quantile="1.0"} 0.0
go_gc_duration_seconds_sum 0.0
go_gc_duration_seconds_count 0.0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10.0
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.11.2"} 1.0
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.043376e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 2.043376e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.443509e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 374.0
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.234368e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.043376e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.3578112e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.104768e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 6219.0
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 8192.0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.668288e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0.0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0.0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 6593.0
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 6912.0
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384.0
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 33592.0
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152.0
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.473924e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.038915e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 425984.0
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 425984.0
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.1891192e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7.0
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1.0
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1.0
promhttp_metric_handler_requests_total{code="500"} 0.0
promhttp_metric_handler_requests_total{code="503"} 0.0
# HELP test_request_duration_seconds The HTTP request latencies in seconds.
# TYPE test_request_duration_seconds summary
test_request_duration_seconds{quantile="0.5"} NaN
test_request_duration_seconds{quantile="0.9"} NaN
test_request_duration_seconds{quantile="0.99"} NaN
test_request_duration_seconds_sum 0.0
test_request_duration_seconds_count 0.0
# HELP test_request_size_bytes The HTTP request sizes in bytes.
# TYPE test_request_size_bytes summary
test_request_size_bytes{quantile="0.5"} NaN
test_request_size_bytes{quantile="0.9"} NaN
test_request_size_bytes{quantile="0.99"} NaN
test_request_size_bytes_sum 0.0
test_request_size_bytes_count 0.0
# HELP test_response_size_bytes The HTTP response sizes in bytes.
# TYPE test_response_size_bytes summary
test_response_size_bytes{quantile="0.5"} NaN
test_response_size_bytes{quantile="0.9"} NaN
test_response_size_bytes{quantile="0.99"} NaN
test_response_size_bytes_sum 0.0
test_response_size_bytes_count 0.0

Answer 1

Thanks to all for trying to find a solution, I simply found that I was registering middleware after routes: in this way, routes declared before middleware are not "covered" by the same middleware.

This code is correctly working:

package main

import (
    "github.com/gin-gonic/gin"
    "github.com/zsais/go-gin-prometheus"
    "net/http"
)

func main() {
    router1 := gin.New()
    router2 := gin.New()

    p := ginprometheus.NewPrometheus("test")
    router1.Use(p.HandlerFunc())
    p.SetMetricsPath(router2)

    router1.GET("/test", func(c *gin.Context) {
        c.String(http.StatusOK, "it works")
    })

    go func() { router2.Run(":7100") }()
    router1.Run(":8000")
}

Answer 2

Prometheus uses promhttp_metric_XXXXX for data it gathers data about itself, here:

As you can gather from http://localhost:9090/metrics, one metric that Prometheus exports about itself is called promhttp_metric_handler_requests_total (the total number of /metrics requests the Prometheus server has served).

What you are looking for is called gin_requests_total, example of 10 requests:

gin_requests_total{code="200",handler="main.main.func1",host="localhost:8000",method="GET",url="/test"} 10

Check p.SetListenAddressWithRouter() for using a separate router to expose metrics.

EDIT I see that you name your subsystem "test". So, you should be looking for test_XXXXX, precisely, test_requests_total