Prometheus alertmanager的处理顺序

Question

I'm currently redesigning an existing alert configuration of alert manager. There's only a small issue, I don't understand alert manager fully.

Assume I have the following configuration;

routes:

match:
  severity:"warning"
receiver: "hipchat-teamX"

match_re:
  application:"(foo|bar)"
receiver: "hipchat-teamX"

match_re:
  application:"(barfoo.*)"
receiver: "hipchat-teamY"

Then an alert is firing from Prometheus with the following values, application:"barfooOne" severity:"warning"

To which receiver will the alert send? And why?

Does alert manager handle an up down structure?

And when do you decide to declare a new child of the route (routes)?

Answer 1

I found an solution with some help of the developers at GitHub.

You may see the processing of the config in the following way, similar to an if statement with AND and OR declarations.

Alertmanager applies the config to a up-down, left-right principle. Where the up-down is similar to the OR statement you will use in a regular if statement.

When you're going from left to right, you could compare it with the AND statement in if commands.

Further you should keep in mind that when the first statement matches, and there are no 'AND' parts declared, alertmanager pushes the alert to the receiver without looking further for any other match. So in the example above, if prometheus fires an alert with the following labels {severity: "warning", application: "barfoo"}, the alert is sent to hipchat-teamX.

Note the following, the label value of application matches the route for hipchat-teamY. So why it isn't sent to that route? The answer is not that diffucult, it's because the first one matches, and there are no subroutes declared, so it will stop looking for any other match.

However, if the alertmanager has config like the code below, the alert will be sent to the receiver 'hipchat-teamY'.

routes:

match:
  severity:"warning"
receiver: "hipchat-teamX"

  routes:
  match_re:
    application:"(barfoo.*)"
  receiver: "hipchat-teamY"

match_re:
  application:"(foo|bar)"
receiver: "hipchat-teamX"

match_re:
  application:"(barfoo.*)"
receiver: "hipchat-teamY"

To make it a bit more advanced, asume prometheus sends a new alert with the following labels {severity: "critical", application: "barfoo"}. With the following config, the alert is routed to hipchat-teamX.

And with the labels {severity: "critical", application: "foo"}, the alert is sent to hipchat-teamY.

routes:

match:
  severity:"warning"
receiver: "hipchat-teamX"

  routes:
  match_re:
    application:"(barfoo.*)"
  receiver: "hipchat-teamY"

  match_re:
    application:"(foo|bar)"
  receiver: "hipchat-teamX"

match:
  severity:"critical"
receiver: "hipchat-teamY"

  routes: 
  match_re:
    application:"(barfoo.*)"
  receiver: "hipchat-teamX"

  match_re:
    application:"(foo|bar)"
  receiver: "hipchat-teamY"

I hope this explanation will help others who are experiencing the same issue or question.