adding node exporter info to prometheus query

Question

I am running multiple docker stacks all providing the infrastructure for a product. Internally those docker stacks are monitored using Prometheus and alert to a teams channel. I want all docker stacks alerting to the same teams channel but need a way to tell from which server the notification came.

For example with my rule

  - alert: InstanceDown
    expr: up == 0
    for: 0m
    labels:
      severity: critical

I am generating alerts for instances that go down. But since I have multiple stacks where all the services have the same internal host-names and IPs the alert generated and send to teams is missing the information from which server the alert came.

I was thinking to maybe merge some node-exporter information like node_dmi_info into the alert expr (up ==0), but I was not able to figure that out yet.

And I am also assuming that the alert only has access to the results of its expression so that I can't simply add something like

labels:
      host: "{{ $labels.product_name }}" 

trying to access node-exporter information.

Answer 1

I ended up injecting the hostname coming from the hostname command into prometheus and using that for the alerts by using the flag: --enable-feature=expand-external-labels and the following config:

global:
  ...
  external_labels:
        cluster: ${CLUSTER_NAME}

rule_files:
  - rules.yml

alerting:
  alert_relabel_configs:
    - source_labels: [cluster]
      action: replace
      regex: (.*)
      replacement: "$1"
      target_label: cluster
  alertmanagers:
    - static_configs:
      - targets:
        - 'prometheus-alert:9093'

as explained here: https://www.lisenet.com/2021/use-external-labels-with-prometheus-alerts/