Self-Hosted Observability Stack

Overview

A complete, production-grade observability platform built to monitor infrastructure, services, and applications across multiple environments. The stack covers all three pillars of observability: metrics (Prometheus), visualization (Grafana), and alerting (Alertmanager) with multiple exporters for comprehensive coverage.

Designed for deployment on a single VM via Docker Compose, with Ansible playbooks for production multi-host deployments. Includes SLO-based alerting with multi-window burn-rate calculations, recording rules for performance optimization, and operational runbooks for incident response.

Key Features

Metrics Collection

• Prometheus with 10+ scrape targets and file-based service discovery
• Node Exporter for host-level metrics (CPU, memory, disk, network)
• Blackbox Exporter for HTTP, TCP, ICMP, and DNS probing
• cAdvisor for container-level metrics
• Pushgateway for short-lived job metrics
• Custom Python Exporter for business-specific metrics (SSL expiry, backup age, disk health)

Recording Rules

• Pre-computed CPU, memory, disk I/O, and network rates
• HTTP error rates and latency percentiles
• SLO burn-rate calculations across 5m/30m/1h/6h windows (Google SRE methodology)

Alerting

• Symptom-based alerting — alert on user-facing symptoms, not causes
• 50+ alert rules covering node health, HTTP services, containers, probes, and self-monitoring
• Multi-receiver routing — severity-based routing to Slack, email, and PagerDuty-like webhooks
• Inhibit rules — suppress cascading alerts when root cause is known
• Rich notification templates — Slack messages with context, runbook links, and silence buttons

Dashboards

• Cluster Overview — high-level system health at a glance
• Node Exporter Full — comprehensive host metrics per server
• Container Overview — Docker/container resource usage
• Blackbox Prober — probe success rates, latency, SSL certificate expiry
• HTTP Service SLO — burn-rate, error budget, latency percentiles
• Prometheus Self-Monitor — scrape performance, TSDB stats, cardinality

Architecture

CODE
16 linesCopy
1Grafana (Visualization)
2    |
3    +-- Prometheus (Metrics Collection + Recording Rules + Alert Rules)
4    |       |
5    |       +-- Node Exporter (Host Metrics)
6    |       +-- Blackbox Exporter (HTTP/TCP/ICMP Probes)
7    |       +-- cAdvisor (Container Metrics)
8    |       +-- Pushgateway (Short-lived Jobs)
9    |       +-- Custom Exporter (Business Metrics)
10    |       +-- File-based Service Discovery
11    |
12    +-- Alertmanager (Alert Routing + Dedup)
13            |
14            +-- Slack (Warning Alerts)
15            +-- Email (Info Alerts)
16            +-- PagerDuty (Critical Alerts)

Technical Implementation

Prometheus Configuration

Prometheus is configured with a 15-second global scrape interval, 30-day data retention, and file-based service discovery for dynamic target management. Recording rules pre-compute expensive queries to reduce query-time load.

SLO-Based Alerting

Alert rules follow Google SRE best practices with multi-window burn-rate calculations:

• Fast burn (5m window) — catches sudden outages within minutes
• Medium burn (30m/1h windows) — detects sustained degradation
• Slow burn (6h window) — identifies chronic issues eating error budget

Custom Python Exporter

Built using the

• SSL certificate expiry days for configured domains
• System uptime in seconds
• Disk SMART health status
• Backup freshness (hours since last successful backup)

CI/CD Validation

GitHub Actions pipeline validates all configurations on every PR:

• CODE

  1 lineCopy

  promtool check config
  on Prometheus config
• CODE

  1 lineCopy

  promtool check rules
  on all recording and alerting rules
• CODE

  1 lineCopy

  amtool check-config
  on Alertmanager config
• CODE

  1 lineCopy

  yamllint
  on all YAML files
• CODE

  1 lineCopy

  promtool test rules
  runs unit tests with sample data

Deployment

Local (Docker Compose)

Bash
3 linesCopy
1cp .env.example .env
2# Edit .env with your settings
3docker compose -f docker/docker-compose.yml up -d

Production (Ansible)

Bash
2 linesCopy
1cd ansible/
2ansible-playbook -i inventory/hosts.yml playbooks/deploy-stack.yml

Impact

• 50+ alert rules covering infrastructure, services, and self-monitoring
• 6 pre-built Grafana dashboards for immediate visibility
• 4 operational runbooks for incident response procedures
• promtool unit tests ensuring alert rules fire correctly
• Custom metrics for business-specific monitoring (SSL, backups, disk health)
• Sub-minute detection of critical failures with fast-burn SLO alerts

Future Plans

• Add Grafana Tempo for distributed tracing
• Integrate Thanos for long-term metric storage and global query
• Add kube-state-metrics integration for Kubernetes clusters
• Build SLO dashboard with error budget tracking and burn-rate annotations
• Add SNMP Exporter for network device monitoring
• Implement Grafana OnCall for on-call rotation management