Prometheus & Grafana

Prometheus is an open-source metrics monitoring and alerting toolkit.

Grafana is a visualization and dashboard platform for metrics/logs/traces and more.

Prometheus stores/queries metrics; Grafana visualizes and explores them.

A numeric time-series measurement representing system/application behavior.

A sequence of timestamped values for a metric + label set.

Key-value metadata dimension attached to metric series.

Enable filtering, aggregation, and slicing by dimensions (instance, job, status, etc.).

Periodic HTTP pull of metrics endpoint from target.

Frequency at which Prometheus collects metrics from target.

Endpoint/source being scraped for metrics.

Component exposing metrics for systems that don’t natively expose Prometheus format.

Node Exporter for host/system metrics.

HTTP endpoint exposing metrics in Prometheus text/OpenMetrics format.

Core component scraping, storing, and querying metrics.

Prometheus query language for selecting and transforming time series.

Query evaluated at a single timestamp.

Query evaluated across time window for graphing/trends.

Prometheus expression with condition threshold triggering alert.

Component handling alert deduplication, grouping, routing, silencing, notifications.

Decoupled alert processing and multi-source alert routing.

Backend system Grafana queries (Prometheus, Loki, Elasticsearch, etc.).

Collection of panels visualizing related signals.

Single visualization unit (graph, stat, table, heatmap, etc.).

Templated parameter for dynamic filtering across panels.

Reusable dashboards across services/environments.

Monotonically increasing metric (resets on restart).

Value that can go up/down arbitrarily (temperature, queue depth).

Distribution metric with buckets + count + sum.

Client-side calculated quantiles + count + sum (limitations in aggregation).

Prefer histograms for aggregatable latency percentiles across instances.

Number of unique time series generated by metric label combinations.

Memory/storage/query cost explosion.

Labeling metrics with user_id/request_id raw values.

How long Prometheus stores data locally.

Time series database used internally by Prometheus.

Automatic target discovery from platforms (Kubernetes, Consul, cloud APIs, etc.).

Manually defined scrape targets in config.

Transforming/filtering target or metric labels during ingestion pipeline.

Precomputed PromQL result stored as new time series.

Faster dashboards/alerts and standardized query logic.

Max duration allowed for a single scrape request.

Prevents slow targets from blocking scrape cycle reliability.

Built-in metric indicating scrape target availability (1 up, 0 down).

Simpler target discovery and centralized scraping control.

Usually via Pushgateway for short-lived batch jobs (special case).

Not general event store; misuse can create stale/unowned metrics.

Only infrastructure metrics, no application/business metrics.

Alerting on every metric without SLO context.

Monitor availability, latency, errors, saturation (golden signals).

Service overview: request rate, error rate, latency, resource usage.

Actionable alerts for user-impacting conditions, not noise.

If no action exists, alert causes fatigue not reliability.

Temporary suppression of matching alerts.

Suppress lower-priority alerts when higher-priority root alert is firing.

Reduce alert storms and focus responders.

Ad-hoc query interface for interactive troubleshooting.

Visual marker on graphs (deploys/incidents/events).

Correlate metric changes with release events quickly.

Protect Prometheus/Grafana access and secret data sources.

Back up dashboards/alert rules as code.

Avoid heavy unbounded queries and high-cardinality labels.

Dashboard ownership and alert ownership per service.

Start from user impact, then drill into components.

Standardize metric naming and label conventions early.

Prioritize signal quality over dashboard quantity.

Per-second average increase of counter over range.

Instantaneous counter rate based on last two points (spikier).

Use rate for alerts/trends; irate for fast-changing visual diagnostics.

Total counter increase over time range.

Difference between first/last values in range (for gauges).

Estimate quantiles from histogram bucket series.

Need proper sum by (le,...) semantics to preserve bucket structure.

Detect missing series for expected metrics.

Joining series across metrics using label matching rules.

Control which labels participate in vector matching.

Resolve one-to-many/many-to-one vector join cardinality.

Often colon-separated semantic names (team conventions vary).

Reusable queries and consistent alert semantics.

How often Prometheus evaluates recording/alert rules.

Condition must remain true for duration before firing.

Filters transient spikes/flapping conditions.

Condition met but FOR duration not yet satisfied.

Alert active and routed to Alertmanager.

Hierarchical routing logic by labels to receivers.

Notification destination/integration (Slack, PagerDuty, email, etc.).

Labels used to batch related alerts into one notification.

Timing controls for first, subsequent, and repeat notifications.

Probes endpoints externally (HTTP, TCP, DNS, ICMP) for availability/latency.

Measure user-like reachability, not only internal app metrics.

Exports Kubernetes object state metrics (deployments, pods, etc.).

Node exporter: host OS metrics; kube-state-metrics: Kubernetes object metadata/state.

CRD defining scrape config for Kubernetes services.

CRD defining scrape config for pods directly.

Simplifies Kubernetes-native management of Prometheus stack.

Forward metrics to remote long-term storage backend.

Long retention, global query, durable storage beyond local Prometheus.

Query external storage through Prometheus interface (backend dependent).

Prometheus scraping aggregated metrics from other Prometheus servers.

Federation pulls selected series; remote_write streams samples out.

Store lower-resolution historical data for long-term cost efficiency.

Attach trace/sample IDs to metrics points for metric-to-trace correlation.

Faster root-cause pivot from latency spikes to traces.

Monitor Rate, Errors, Duration for request-driven services.

Monitor Utilization, Saturation, Errors for resources.

Service Level Indicator: measured reliability metric.

Target objective for SLI over time.

Allowed unreliability derived from SLO target.

Align alerts with user impact and reliability goals.

Combines short/long windows for fast and stable SLO breach detection.

Alerting directly on CPU/memory without service-impact context.

Symptoms (user impact) first, causes second.

Dense dashboard with no narrative hierarchy.

Overview -> drilldown -> component diagnostics.

Access control grouping dashboards by team/domain sensitivity.

Managing dashboards/datasources/alerts via files/APIs as code.

Version control, review, reproducibility, rollback.

AuthN/AuthZ, datasource credential scoping, audit logging.

HA alert routing and tested notification paths.

Recording rules and constrained query ranges.

Team can trace alert -> dashboard -> runbook quickly.

Operational guide URL embedded in alert annotations.

Immediate context and remediation steps for responders.

Discard unnecessary/high-cardinality series at scrape time.

Split target load across Prometheus instances.

Treat observability configs as production code with CI checks.

Define naming/labeling/alert standards org-wide.

Continuously prune low-value metrics and dashboards.

Separate collection, storage, alerting, and visualization responsibilities.

SRE + dev teams co-own service-level observability.

Optimize for actionable clarity, not metric volume.

Two independent Prometheus servers scrape same targets for redundancy.

Duplicate alerts/samples handling requires Alertmanager/remote backend dedup logic.

Balancing local autonomy with centralized visibility and cost.

Horizontally scalable long-term Prometheus-compatible storage/query layers.

Years of retention, global querying, and durable storage.

Caching, splitting, and optimizing query execution.

Merge blocks/chunks for storage efficiency and query performance.

Rapid uncontrolled series growth causing memory/query failures.

Identify top offenders, relabel/drop, redesign labels, enforce budgets.

Per-team/service cap on allowed active series dimensions.

Prevents shared observability platform exhaustion.

Series marked stale when targets disappear/sample stream stops.

Avoid misleading joins/aggregations during target churn.

Emerging histogram representation improving accuracy/performance tradeoffs (version/ecosystem dependent).

Queue buildup/data loss risk during remote endpoint slowdown.

Queue tuning, WAL durability awareness, backend HA, alerting.

Write-ahead log for durability and crash recovery.

Stagger scrape timings to reduce synchronized load spikes.

Separate data access/quotas/routing for teams/customers.

One tenant’s high-cardinality/high-query load degrades others.

Per-tenant quotas, query limits, isolated ingestion paths.

Limit expensive PromQL patterns and enforce timeout/max samples.

Precomputing too many low-value series increasing storage load.

Precompute only expensive/high-traffic/high-value queries.

Many related alerts fire simultaneously during major outage.

Inhibition, grouping, symptom-based paging, cause alerts as tickets.

Page on user-impacting symptoms, not every internal anomaly.

Too sensitive causes noise; too lax delays detection.

Historical incident replay + game days + SLO error budget policy.

Metrics may leak sensitive labels/data if instrumentation is careless.

Never include PII/secrets in labels/metric values.

Secure transport/authentication between Prometheus and targets.

Dashboard sprawl, inconsistent ownership, variable query quality.

Folder taxonomy, review process, dashboard lifecycle policies.

Create, review, deprecate, archive dashboards by usage/ownership.

Track dashboard/query usage to prune low-value assets.

Overlay deploys, config changes, and alerts on metrics timeline.

Speeds root cause identification and rollback decisions.

Synthetic probes baseline availability; RUM captures real client experience.

Broader coverage of controlled and real-world behavior.

Validate alerts/dashboards detect expected failure modes.

Practice incidents to test signal quality and runbooks.

Audit logs, alert history, SLO reports, access records.

Backup configs, replicate storage, restore dashboards/alerts quickly.

Monitoring must remain available during incidents, not fail first.

Fewer actionable alerts beat many noisy alerts.

Every metric/label/query should justify its operational value.

Protect telemetry pipelines and prevent sensitive data exposure.

Standardize instrumentation, labels, and alert semantics organization-wide.

Design observability as layered system: collect, store, alert, visualize.

Continuously test alerts, runbooks, and escalation paths.

Developers and SREs share ownership of service observability.

Plan cardinality and query growth before incidents force redesign.

Dashboards are operational products with users and maintenance lifecycle.

Prometheus/Grafana excellence is actionable, scalable, and SLO-driven observability.