~/gmora

PromQL Cheat Sheet

6 min read #prometheus#promql

Prometheus Query Language reference for monitoring and alerting.

Metric Types #

TypeDescriptionUse Case
CounterMonotonically increasing valueRequest counts, errors
GaugeValue that can go up or downCPU usage, memory, temperature
HistogramDistribution of observationsRequest latency, response sizes
SummarySimilar to histogram with quantilesPrecomputed percentiles

Data Types #

Instant Vector - Set of time series at a single timestamp

http_requests_total

Range Vector - Set of time series over a time range

http_requests_total[5m]

Scalar - Simple numeric value

5

String - String literal (rarely used)

"error"

Selectors #

Label Matching #

# Exact match
http_requests_total{method="GET"}

# Multiple labels
http_requests_total{method="GET", status="200"}

# Not equal
http_requests_total{status!="200"}

# Regex match
http_requests_total{status=~"5.."}

# Negative regex
http_requests_total{status!~"2.."}

Time Ranges #

# 5 minutes
http_requests_total[5m]

# 1 hour
http_requests_total[1h]

# 1 day
http_requests_total[1d]

# Units: s, m, h, d, w, y

Rate Functions #

rate() #

Calculate per-second rate over time range:

# Requests per second
rate(http_requests_total[5m])

# With labels
rate(http_requests_total{job="api"}[5m])

irate() #

Instant rate using last two data points (more sensitive to spikes):

irate(http_requests_total[5m])

increase() #

Total increase over time range:

# Total requests in last 5 minutes
increase(http_requests_total[5m])

# Requests in last hour
increase(http_requests_total[1h])

Aggregation Operators #

Basic Aggregation #

# Sum across all instances
sum(rate(http_requests_total[5m]))

# Average
avg(rate(http_requests_total[5m]))

# Minimum
min(http_requests_total)

# Maximum
max(http_requests_total)

# Count
count(up == 1)

Aggregation by Labels #

# Sum per service
sum by (service) (rate(http_requests_total[5m]))

# Sum per service and method
sum by (service, method) (rate(http_requests_total[5m]))

# Sum, removing instance label
sum without (instance) (rate(http_requests_total[5m]))

All Aggregation Functions #

sum()       # Sum of all values
avg()       # Average
min()       # Minimum
max()       # Maximum
count()     # Count of elements
stddev()    # Standard deviation
stdvar()    # Standard variance
topk(N)     # Top N elements
bottomk(N)  # Bottom N elements
quantile()  # Calculate quantile

Binary Operators #

Arithmetic #

# Addition
node_memory_MemTotal_bytes - node_memory_MemFree_bytes

# Percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Division
rate(http_requests_total[5m]) / rate(http_requests_total[5m] offset 1h)

Comparison #

# Greater than
up > 0

# Less than
node_load1 < 0.8

# Equal
http_response_status == 200

# Not equal
http_response_status != 200

Logical #

# AND
up == 1 and on(instance) node_load1 > 0.8

# OR
up == 0 or node_load1 > 1.0

# UNLESS
up == 1 unless node_load1 > 0.8

Vector Matching #

One-to-One #

# Match by instance and job
method:http_requests:rate5m{method="GET"} 
  / ignoring(method) 
  method:http_requests:rate5m

Many-to-One #

# Match multiple pods to one node
rate(container_cpu_usage[5m]) 
  * on(instance) group_left(node_name) 
  node_metadata

Common Queries #

CPU Usage #

# CPU usage by instance
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# CPU usage by mode
sum by (mode) (rate(node_cpu_seconds_total[5m]))

Memory Usage #

# Memory usage percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Available memory in GB
node_memory_MemAvailable_bytes / 1024^3

Disk Usage #

# Disk usage percentage
(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100

# Disk I/O rate
rate(node_disk_read_bytes_total[5m])
rate(node_disk_written_bytes_total[5m])

Network #

# Network receive rate in MB/s
rate(node_network_receive_bytes_total[5m]) / 1024^2

# Network transmit rate
rate(node_network_transmit_bytes_total[5m]) / 1024^2

Request Metrics #

# Request rate
sum(rate(http_requests_total[5m]))

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))

# Error percentage
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

# Requests per second by service
sum by (service) (rate(http_requests_total[5m]))

Histogram Functions #

histogram_quantile() #

Calculate quantiles from histogram buckets:

# P95 latency
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# P50 latency
histogram_quantile(0.50,
  rate(http_request_duration_seconds_bucket[5m])
)

# P99 latency by service
histogram_quantile(0.99,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

Histogram Bucket Functions #

# Average request duration
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

Time Functions #

offset #

Query data from the past:

# Current vs 1 hour ago
http_requests_total offset 1h

# Compare current rate to rate 1 day ago
rate(http_requests_total[5m])
/
rate(http_requests_total[5m] offset 1d)

Time Manipulation #

time()              # Current timestamp
minute()            # Current minute (0-59)
hour()              # Current hour (0-23)
day_of_week()       # Day of week (0-6, Sunday=0)
day_of_month()      # Day of month (1-31)
days_in_month()     # Days in current month
month()             # Current month (1-12)
year()              # Current year

Useful Functions #

clamp_max / clamp_min #

Limit values to range:

# Cap at 100
clamp_max(cpu_usage, 100)

# Floor at 0
clamp_min(cpu_usage, 0)

round #

Round values:

# Round to nearest integer
round(cpu_usage)

# Round to 1 decimal
round(cpu_usage, 0.1)

absent #

Check if metric is missing:

# Alert if metric disappears
absent(up{job="api"})

label_replace #

Modify labels:

label_replace(
  up,
  "new_label",
  "$1",
  "instance",
  "(.*):.*"
)

predict_linear #

Predict future values:

# Predict disk usage in 4 hours
predict_linear(node_filesystem_free_bytes[1h], 4*3600)

Subqueries #

Execute query over a time range:

# Max rate over last hour, calculated per minute
max_over_time(
  rate(http_requests_total[5m])[1h:1m]
)

# Average of P95 over last day
avg_over_time(
  histogram_quantile(0.95, rate(http_request_duration_bucket[5m]))[1d:5m]
)

Alert Query Patterns #

Service Down #

# Alert if service is down
up{job="api"} == 0

# Alert if service down for 5 minutes
up{job="api"} == 0 for 5m

High Error Rate #

# Error rate above 5%
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) > 0.05

High Latency #

# P95 latency above 500ms
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
) > 0.5

Resource Exhaustion #

# Disk full in less than 4 hours
predict_linear(node_filesystem_free_bytes[1h], 4*3600) < 0

# Memory usage above 90%
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.9

Performance Tips #

Good vs Bad #

# High cardinality, no filtering
rate(http_requests_total[1h])

# Unnecessary regex
http_requests{status=~"200"}

# Filtered and aggregated
sum by (service) (rate(http_requests_total{job="api"}[5m]))

# Direct match
http_requests{status="200"}

Common Pitfalls #

Quick Reference #

PatternQuery
Request ratesum(rate(http_requests_total[5m]))
Error rate %sum(rate(http_requests{status=~"5.."}[5m])) / sum(rate(http_requests[5m])) * 100
CPU usage %100 - (avg(rate(node_cpu{mode="idle"}[5m])) * 100)
Memory usage %(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
P95 latencyhistogram_quantile(0.95, rate(http_duration_bucket[5m]))
Disk usage %(1 - (node_filesystem_avail / node_filesystem_size)) * 100
Top 5 services by requeststopk(5, sum by (service) (rate(http_requests[5m])))