Metrics Pipeline Debugging Runbook

Overview

This runbook provides step-by-step guidance for diagnosing and resolving issues in a metrics pipeline consisting of:

A Python program emitting StatsD metrics
Vector (StatsD source → Prometheus exporter sink)
Prometheus (scraping metrics from Vector)
Grafana (visualizing Prometheus data)

Debugging Workflow

1. Verify Python Program is Emitting StatsD Metrics

Check if metrics are being sent

Run the following command to manually send a StatsD metric:

echo "orchestrator_jobs_active:1|g" | nc -u -w1 127.0.0.1 9000

Check Vector logs for received metrics

journalctl -u vector -f | grep statsd

✅ If logs indicate received metrics → Vector is receiving data.

❌ If no logs appear → The Python program might not be sending metrics correctly or is targeting the wrong address/port.

2. Verify Vector is Exposing Metrics

Check if Vector’s Prometheus exporter is publishing metrics

curl -s http://127.0.0.1:9598/metrics | grep orchestrator

✅ If metrics appear → Vector is working correctly.

❌ If no metrics appear →

Ensure that Vector’s flush_period_secs is correctly configured.
Check Vector logs for errors: journalctl -u vector -f.

3. Verify Prometheus is Scraping Metrics

Check Prometheus targets

curl -s "http://localhost:9090/api/v1/targets" | jq '.data.activeTargets[] | {scrapeUrl, lastScrape, lastError, health}'

✅ If health is "up" and lastError is empty → Prometheus is successfully scraping.

❌ If health is "down" → There is a scraping issue (check lastError).

Check if Prometheus has seen the metric

curl -s "http://localhost:9090/api/v1/series?match[]=orchestrator_jobs_active" | jq .

✅ If the query returns data → Prometheus has recorded the metric.

❌ If empty → Check Vector and StatsD configuration.

Query latest metric values

curl -s "http://localhost:9090/api/v1/query?query=orchestrator_jobs_active" | jq .

✅ If a value is returned → The metric is stored.

❌ If no value is returned → Metrics may be expiring before they are scraped.

4. Fix Potential Issues

A. Vector’s `flush_period_secs` is Too Short

If metrics disappear before Prometheus scrapes them, increase flush_period_secs in vector.toml:

[sinks.prometheus_exporter]
type = "prometheus_exporter"
inputs = ["statsd"]
address = "0.0.0.0:9598"
flush_period_secs = 30  # Set higher than Prometheus scrape interval

Restart Vector:

systemctl restart vector

B. Prometheus Scrape Interval is Too Long

Ensure Prometheus scrapes more frequently than Vector's flush interval (prometheus.yml):

scrape_configs:
- job_name: "vector"
  scrape_interval: 5s  # Must be less than Vector’s flush_period_secs
  static_configs:
    - targets: ["0.0.0.0:9598"]

Restart Prometheus:

systemctl restart prometheus

C. Metrics Are Not Updating

Manually send a StatsD metric:

echo "orchestrator_jobs_active:5|g" | nc -u -w1 127.0.0.1 9000

Immediately query Prometheus:

curl -s "http://localhost:9090/api/v1/query?query=orchestrator_jobs_active" | jq .

If the metric appears but disappears later, Vector may be expiring stale metrics too soon.

D. Verify Prometheus Retention Settings

Check retention settings:

prometheus --storage.tsdb.retention.time

If retention is low (e.g., 1h), increase it:

prometheus --storage.tsdb.retention.time=7d

Final Debugging Checklist

✅ Python program emits StatsD metrics (nc -u -w1 127.0.0.1 9000)
✅ Vector receives metrics (journalctl -u vector -f | grep statsd)
✅ Vector exposes metrics correctly (curl -s http://127.0.0.1:9598/metrics)
✅ Prometheus scrapes successfully (api/v1/targets with health: "up")
✅ Metrics are stored in Prometheus (api/v1/query?query=orchestrator_jobs_active)

Following this runbook will help identify and resolve issues efficiently when debugging the metrics pipeline.

Andrew's Notes