Questions tagged [prometheus-alertmanager]
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
prometheus-alertmanager
368
questions with no upvoted or accepted answers
15
votes
4
answers
13k
views
How to send alert for every error in my logs using Promtail / Loki - AlertManager?
I'm using Promtail + Loki to collect my logs and I can't figure how can I alert for every error in my log files. I'm also using Prometheus, Alertmanager and Grafana. I've seen some people have managed ...
- 173
6
votes
0
answers
995
views
Is it possible to have dynamic groupBy in Prometheus and Alertmanager?
In my Prometheus alert configuration, I have some great summary annotations, containing different labels.
- alert: PodsPending
expr: kube_pod_status_phase{phase="Pending"} > 0
labels:
...
- 4,618
5
votes
0
answers
2k
views
Line break in Prometheus' HTML Alert / Rule view
I have several Prometheus alerting rules which are quite long. In my editor I manage to add line breaks to the rules' expressions (see line expr) with a | at the beginning:
- alert: IF_Critical
...
- 71
5
votes
1
answer
3k
views
Is it possible to conditionally alert with prometheus based on if another alert is firing or not?
The idea here is that I'm considering alerting on a metric that corroborates a metric that another tool is monitoring. For example: I might have a service doing some task by feeding off a queue and ...
- 6,400
5
votes
2
answers
14k
views
PromQL. How to add label value from different metric
If I have two metrics:
kube_pod_container_status_restarts_total {..., pod="my_pod_name_42", ...}
and
container_memory_usage_bytes {..., pod_name="my_pod_name_42", cluster_name="megatron", ...}
And ...
4
votes
0
answers
775
views
Configure Loki log-based alerts with kubernetes CRD
I like the idea of Prometheus Operator where alerting configurations are deployable kubernetes objects defined with CRDs.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
...
- 1,330
4
votes
3
answers
5k
views
Use conditional operator in Prometheus alert rules to set severity
I would like to use the conditional operator into the Prometheus alert.rules definition to set a specific severity.
For instance, if the environment is production, I want to set the severity to ...
- 2,029
4
votes
0
answers
764
views
How to get Custom Metrics from Azure app-insight in Prometheus server?
Because set alert rule in Azure is too expensive, I want set alert rule in Prometheus instead.
However, is there any way to expose the metrics on Azure(app insight) to Prometheus server?
(I already ...
- 143
4
votes
0
answers
5k
views
Context deadline exceeded for Prometheus service endpoint
Prometheus Version - 6.7.0
In order to scrape only specific port for a service, I have used below configuration. In Prometheus the target service endpoint always goes in status
context deadline ...
- 353
4
votes
0
answers
2k
views
Prometheus check if a metric disappear
I want have make alerts based on a metric which in my case, sometimes, may disappear
Name of metrics count_items returns number every 10minutes if number < 750000 an alarm is triggered if a metric ...
- 53
3
votes
1
answer
546
views
Relabel configs in Prometheus alert manager
I have alert rules in Prometheus and I want to achieve that I don't see the cluster label in the alert but instead I see the env label. I tried to use this code but absolutely no result.
- alert: SQL
...
- 47
3
votes
1
answer
3k
views
Set different maximum values for each field in the Bar Gauge | Grafana
I was trying to set up a dashboard so that I could monitor the number of messages within certain queues. The problem is that in order to create a suitable alert I need to set a maximum value for each ...
3
votes
0
answers
993
views
How independently import files in values.yaml
I'm using the prometheus stack project to implement via helm with kubernetes, but I would like to customize some rules. I'm trying to use the additionalPrometheusRules parameter for this, but I'm ...
- 751
3
votes
0
answers
420
views
How to get the full log error message from Grafana into Alertmanager for alerting?
I can get the labels from the log messages on Grafana for which I am alerting through alertmanager. However, I want to get the full log message into the alert message that gets sent to slack or email. ...
- 31
3
votes
0
answers
753
views
PromQL for K8s pods that are over the CPU / RAM limits
I've deployed kube-prometheus-stack and default rules / alerts helped me a lot, but I need to create custom ones.
Specifically, I would like to get information when my pods are 50% (warning) or 70% (...
- 83