~/gmora

SLIs, SLOs, and SLAs

5 min read #slo#sre#observability

1. Planning #

Service Level Objectives (SLOs) distinguish themselves from standard alerting and monitoring by focusing strictly on operations that matter to the user.

The Planning Workflow #

StepDescriptionExample
1. OperationA customer-facing operation important enough to warrant an SLO.A new order is generated, submitted, and acknowledged.
2. SLI (Indicator)The measurable data needed from the system to compute the SLOs.Latency in seconds from “order submitted” to “order acknowledged” (derived from logs/telemetry).
3. Aggregation WindowA timeframe meaningful to the business to spot trends/hotspots.8 hours.
4. Target ReliabilityThe percentage of success required. (Prod usually > 99%).90% (for this example).
5. SLO (Objective)The combination of the Indicator and the Target.New orders are acknowledged within 1 second and achieve this level of service 90% of the time.

Categories of SLIs #

Based on “Database Reliability Engineering” by Laine Campbell & Charity Majors.

Measurement of delay in a communication flow.

Usually measured in units of time. Best practice is to measure boundaries:

  • Upper bound: “Created within 2 seconds.”
  • Range: “Created between 1 and 2 seconds.”

Is the service functional?

A simple measurement of whether or not a service is generally available given a valid request.

  • Example: HTTP 200 OK vs HTTP 500 Error.

Quality of Data.

“Do users receive the data they expected?”

Hybrid Example (Consistency + Latency):

“Users see an online check deposit reflected in their transaction history within 2 seconds.”

Quantity of data sent/received within a timeframe.

  • Example: “Transactions processed per second: 3”.

Context matters: An overnight batch process measures throughput very differently from a real-time streaming service.

2. Implementing #

Implementation is the delivery phase where SLOs impact development efforts.

  1. Basics: Ensure SLIs identified in planning are present in logs or time-series DBs.
  2. Architecture Impact:
    • Targeting 99%? Likely no major architectural changes.
    • Targeting 99.999%? Profound impact on infra and design.

Code Instrumentation Example #

Scenario: User clicks “checkout”.

SLI: Seconds to process a valid order (derived from logs).

We need 3 distinct log events:

  1. Request initiation.
  2. Success with latency calculation.
  3. Failure events.
@throws(classOf[ValidationException])
def checkout(cartItems: ShoppingCart,
             creditCardFrom: CreditCardFrom,
             shippingAdressFrom: AddressFrom): Order = {

  logger.info("checkout requested") // (1)

  try {
    startTime = timer.start() // (2)
    validCard = validateCreditCard(creditCardForm)
    validAddress = validateShippingAdress(shippingAddressFrom)
    order = processOrder(validCard, validAddress, cartItems))
    endTime = timer.end()
    elapsedTime = endTime - startTime

    // Log success with specific latency
    logger.info(s"order ${order.id} successfully took ${elapsedTime}ms") // (3)
    return order
  }
  catch (e: ValidationException) {
    // Log known validation errors (client side errors)
    logger.info("checkout request failed due to a validation error") // (4)
    throw e
  }
  catch (e: ServiceException) {
    // Log server side errors
    logger.error("checkout request failed") // (5)
    throw e
  }
}
  1. Total Requests Marker: Essential for the denominator in our formula.
  2. Timer: Capture high-precision timestamps.
  3. Success Marker: Capture the successful ID and the elapsedTime.
  4. Client Error: These often shouldn’t count against your reliability score (depending on policy).
  5. Server Error: These definitely count against reliability.

3. Operations #

Once SLIs are implemented, we need visualization (Dashboards) and Reporting.

The Tooling Stack #

  • Sloth: Generates Prometheus rules from simple SLO spec files. Saves time vs. hand-coding rules.
  • Prometheus: Standard for metrics-based SLIs.
  • Grafana: Visualizing the data (Supports Prometheus & Loki).
  • Loki: Like Prometheus, but for logs. Great for legacy apps where you can’t change code easily but have logs.
  • Thanos: Solves Prometheus retention limits (Unlimited history).
  • NOBL9: Turn-key platform for SLOs (if you want to buy vs build).

Workflow: Logging to Dashboard (Loki path) #

  1. Promtail: Configure to scrape specific log files containing SLI data.
  2. Extraction: Define parser configs (Regex/JSON) in Promtail to isolate fields (latency, status).
  3. Loki: Query language extracts the metrics.
  4. Grafana: Visualize the query over the Aggregation Window.

4. The Maths #

How to calculate the final percentage.

Scenario A: Standard Calculation #

Objective: New orders acknowledged within 1s.

Total Requests: 110 | Validation Errors: 10 | Service Errors: 10 | Slow Requests: 2

SLO=SuccessfulRequestsTotalRequests SLO = \frac{Successful Requests}{Total Requests}

Breakdown:

  • Valid Requests = (Total - Validation Errors) = 110−10=100110 - 10 = 100
  • Failures = (Service Errors + Slow Requests) = 10+2=1210 + 2 = 12
  • Successes = (Valid Requests - Failures) = 100−12=88100 - 12 = 88
SLO=88100=88% SLO = \frac{88}{100} = 88\%

Scenario B: Filtering “Service Errors” #

Sometimes, we only want to measure Latency performance, excluding crashes (which might be covered by an Availability SLO).

Formula adjustment: Remove Service Errors from the denominator.

SLO=88(100−10)=8890=97% SLO = \frac{88}{(100 - 10)} = \frac{88}{90} = 97\%

5. Understanding & Error Budgets #

This phase involves periodically reviewing reliability (monthly/quarterly). Data-driven decisions beat gut feelings.

The Burn Rate #

If you target 90% but achieve 99.9%, you have a surplus of reliability. This is your Error Budget.

  • Under Budget (Failing): Freeze features, focus on reliability.
  • Over Budget (Success): Spend the budget on risky changes, refactoring, or speed.
Advanced: Calculating Error Budget

Example for a 99.9% Availability Target:

Error Budget = 1 - Availability SLO
Error Budget = 1 - 99.9% = 0.1%

Time Calculation (Monthly):
0.1% of 30 days = 43.2 minutes

Note: Don’t worry about this when starting out. Get the measurements working first.

Resources #

Books

  • Implementing Service Level Objectives - Alex Hidalgo
  • Database Reliability Engineering - Laine Campbell & Charity Majors
  • Google’s SRE Book List

Tools

Articles & Lectures