Hosted Metrics Grafana
Category: Engineering & DevOps
This page is generated from the Air Pipe marketplace. Browse it live to install into your organization.
Air Pipe managed pack. On Air Pipe managed, every request to your configs is already counted and stored for you: requests, failures and latency, per config and interface. This pack is the Grafana dashboard for those metrics, plus a three-route demo API so the dashboard has something to show the moment you import it.
No exporter to run, no /metrics to scrape from your configs, and nothing to add to them.
What's included
| File | Purpose |
|---|---|
grafana-dashboard.json | The dashboard: import it into Grafana and pick your Air Pipe data source |
config.yml | Demo API with a fast, a slow and a failing route, to fill the panels |
What the dashboard shows
| Row | Panels |
|---|---|
| Overview | Requests per second, share of failed requests, p95 latency, series used against your limit |
| Traffic | Requests per second by interface and by status, failures by interface, the busiest interfaces in the time range |
| Latency | p95 by interface; p50, p95, p99 and mean across everything selected |
| Plan | Series in use against the limit, and requests not recorded because the limit was reached |
The Config and Interface drop-downs at the top narrow every panel.
Set it up
Hosted metrics comes with every paid plan. You need your organization's UUID and an API key that may read metrics.
1. A key that can read metrics
- In Roles, create a role (for example
Metrics reader) with themetrics:readscope. - In API keys, create a key with a role-based permission and choose that role.
The key can only read metrics: its role decides what it can do.
2. The data source
In Grafana, add a data source of type Prometheus:
| Setting | Value |
|---|---|
| Prometheus server URL | https://api.airpipe.io/<your-org-uuid>/production/_ap/metrics |
| Authentication | Basic authentication: any user name, your API key as the password |
| Scrape interval | 30s (new values arrive every 30 seconds) |
| HTTP method | POST |
Press Save & test. For staging, add a second data source with staging in the URL.
3. The dashboard
Dashboards → New → Import, upload grafana-dashboard.json, and choose the data source
from step 2.
4. Something to look at
Deploy config.yml to staging (or production), then call it a few times:
BASE="https://api.airpipe.io/<your-org-uuid>/staging"
for i in $(seq 1 20); do
curl -s -o /dev/null "$BASE/demo/fast"
curl -s -o /dev/null "$BASE/demo/error"
done
curl -s "$BASE/demo/slow"
Within a minute the dashboard shows the three interfaces: demo/fast near zero latency,
demo/slow at 1.5 seconds, and demo/error in every failure panel.
Already run Prometheus?
The same address is a scrape target, so your own Prometheus, Grafana Alloy, Datadog agent or OpenTelemetry collector can pull the metrics instead of Grafana querying for them:
scrape_configs:
- job_name: airpipe
scheme: https
metrics_path: /<your-org-uuid>/production/_ap/metrics
authorization:
credentials: <your API key>
honor_labels: true
scrape_interval: 30s
Point the dashboard at your own Prometheus and it works the same. honor_labels: true keeps
the instance label that tells the Air Pipe workers apart, which the panels sum over.
The demo routes
| Route | Answers |
|---|---|
GET /demo/fast | 200 at once |
GET /demo/slow | 200 after 1.5 seconds |
GET /demo/error | 500, always, on purpose |
Good to know
- Latency percentiles are estimates from histogram buckets (0.1, 0.5, 1, 2.5, 5, 10, 20, 30 and 45 seconds), so a p95 is as precise as the bucket it falls in.
- Series used is per worker. Each interface uses 12 series plus one per status class it answers with. Past your plan's limit, new interfaces are not recorded, and the Requests not recorded panel counts them.
- Retention follows your plan: 3 days on Starter up to 14 on Platform. Ranges further back show nothing.
- Cost: each query, and each scrape that reaches the store, is 1/100 of a request unit. This dashboard (16 queries) refreshing every minute while open is about 10 units an hour; a scrape every 30 seconds is about 29 units a day per environment.
Full reference: Hosted metrics.
Configuration
config.yml
name: Hosted Metrics Demo
description: >
Three small routes, fast, slow and failing, so a hosted metrics dashboard has requests,
errors and latency to show. No database and nothing to configure: deploy it on Air Pipe
managed, call the routes, and watch them in Grafana.
docs: true
interfaces:
demo/fast:
output: http
method: GET
summary: Fast route
description: Answers at once. Shows up in the request rate and the lowest latency bucket.
tags: [demo]
response_example: |-
{ "ok": true, "route": "fast" }
actions:
- name: Respond
json_output: |-
{ "ok": true, "route": "fast" }
demo/slow:
output: http
method: GET
summary: Slow route
description: Waits 1.5 seconds before answering, so the latency panels have a slow interface to show.
tags: [demo]
response_example: |-
{ "ok": true, "route": "slow" }
actions:
- name: Wait
delay: "1500ms"
- name: Respond
run_when_succeeded: [Wait]
json_output: |-
{ "ok": true, "route": "slow" }
demo/error:
output: http
method: GET
summary: Failing route
description: >
Always answers 500, so the error panels and any alert on them can be checked without
breaking anything real.
tags: [demo]
notes: |
This route fails on purpose. It is safe to call repeatedly.
actions:
- name: AlwaysFail
input: a|params|
assert:
http_code_on_error: 500
error_message: this route always fails, on purpose
tests:
- value: never_sent
is_not_null: true
grafana-dashboard.json
{
"__inputs": [
{
"name": "DS_AIRPIPE",
"label": "Air Pipe hosted metrics",
"description": "Prometheus data source pointing at https://api.airpipe.io/<org>/<env>/_ap/metrics",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "10.0.0"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
}
],
"title": "Air Pipe hosted metrics",
"uid": "airpipe-hosted-metrics",
"description": "Requests, failures and latency for every Air Pipe config and interface, from Air Pipe managed hosted metrics.",
"tags": [
"airpipe",
"hosted-metrics"
],
"schemaVersion": 39,
"version": 1,
"editable": true,
"graphTooltip": 1,
"refresh": "1m",
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"30s",
"1m",
"5m",
"15m"
]
},
"templating": {
"list": [
{
"name": "config",
"label": "Config",
"type": "query",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"definition": "label_values(airpipe_interface_requests_total, config)",
"query": {
"query": "label_values(airpipe_interface_requests_total, config)",
"refId": "config-var",
"qryType": 1
},
"refresh": 2,
"includeAll": true,
"multi": true,
"allValue": ".*",
"sort": 1,
"current": {
"selected": true,
"text": [
"All"
],
"value": [
"$__all"
]
},
"options": []
},
{
"name": "interface",
"label": "Interface",
"type": "query",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"definition": "label_values(airpipe_interface_requests_total{config=~\"$config\"}, interface)",
"query": {
"query": "label_values(airpipe_interface_requests_total{config=~\"$config\"}, interface)",
"refId": "interface-var",
"qryType": 1
},
"refresh": 2,
"includeAll": true,
"multi": true,
"allValue": ".*",
"sort": 1,
"current": {
"selected": true,
"text": [
"All"
],
"value": [
"$__all"
]
},
"options": []
}
]
},
"annotations": {
"list": []
},
"panels": [
{
"id": 1,
"type": "row",
"title": "Overview",
"collapsed": false,
"gridPos": {
"x": 0,
"y": 0,
"w": 24,
"h": 1
},
"panels": []
},
{
"id": 2,
"type": "stat",
"title": "Requests / s",
"description": "All matching interfaces, summed across workers.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 0,
"y": 1,
"w": 6,
"h": 4
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sum(rate(airpipe_interface_requests_total{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval]))",
"legendFormat": "__auto",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
}
},
{
"id": 3,
"type": "stat",
"title": "Failed requests",
"description": "Share of requests that answered 5xx (HTTP, MCP) or failed (websocket, schedule).",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 6,
"y": 1,
"w": 6,
"h": 4
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sum(rate(airpipe_interface_requests_total{config=~\"$config\", interface=~\"$interface\", status=~\"5xx|error\"}[$__rate_interval])) / sum(rate(airpipe_interface_requests_total{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval]))",
"legendFormat": "__auto",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 0.01
},
{
"color": "red",
"value": 0.05
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
}
},
{
"id": 4,
"type": "stat",
"title": "p95 latency",
"description": "95th percentile, estimated from the histogram buckets.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 12,
"y": 1,
"w": 6,
"h": 4
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "histogram_quantile(0.95, sum by (le) (rate(airpipe_interface_duration_seconds_bucket{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval])))",
"legendFormat": "__auto",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
}
},
{
"id": 5,
"type": "stat",
"title": "Series used",
"description": "Series in use against your plan's limit, on the busiest worker. Past 100%, new series are dropped.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 18,
"y": 1,
"w": 6,
"h": 4
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "max(airpipe_hosted_metrics_series) / max(airpipe_hosted_metrics_series_limit)",
"legendFormat": "__auto",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 0.8
},
{
"color": "red",
"value": 0.95
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
}
},
{
"id": 6,
"type": "row",
"title": "Traffic",
"collapsed": false,
"gridPos": {
"x": 0,
"y": 5,
"w": 24,
"h": 1
},
"panels": []
},
{
"id": 7,
"type": "timeseries",
"title": "Requests / s by interface",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 0,
"y": 6,
"w": 12,
"h": 8
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sum by (config, interface) (rate(airpipe_interface_requests_total{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval]))",
"legendFormat": "{{config}} {{interface}}",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": true
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 8,
"type": "timeseries",
"title": "Requests / s by status",
"description": "HTTP status class for HTTP and MCP; ok or error for websocket and schedule.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 12,
"y": 6,
"w": 12,
"h": 8
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sum by (status) (rate(airpipe_interface_requests_total{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval]))",
"legendFormat": "{{status}}",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": true
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 9,
"type": "timeseries",
"title": "Failed requests / s by interface",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 0,
"y": 14,
"w": 12,
"h": 8
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sum by (config, interface) (rate(airpipe_interface_requests_total{config=~\"$config\", interface=~\"$interface\", status=~\"5xx|error\"}[$__rate_interval]))",
"legendFormat": "{{config}} {{interface}}",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": true
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 10,
"type": "table",
"title": "Busiest interfaces",
"description": "Requests in the selected time range.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 12,
"y": 14,
"w": 12,
"h": 8
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sort_desc(sum by (config, interface, transport) (increase(airpipe_interface_requests_total{config=~\"$config\", interface=~\"$interface\"}[$__range])))",
"legendFormat": "__auto",
"refId": "A",
"range": false,
"instant": true,
"format": "table"
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {},
"overrides": []
},
"options": {
"showHeader": true,
"sortBy": [
{
"displayName": "Value",
"desc": true
}
]
}
},
{
"id": 11,
"type": "row",
"title": "Latency",
"collapsed": false,
"gridPos": {
"x": 0,
"y": 22,
"w": 24,
"h": 1
},
"panels": []
},
{
"id": 12,
"type": "timeseries",
"title": "p95 latency by interface",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 0,
"y": 23,
"w": 12,
"h": 8
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "histogram_quantile(0.95, sum by (le, config, interface) (rate(airpipe_interface_duration_seconds_bucket{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval])))",
"legendFormat": "{{config}} {{interface}}",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": true
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 13,
"type": "timeseries",
"title": "Latency percentiles",
"description": "All matching interfaces.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 12,
"y": 23,
"w": 12,
"h": 8
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "histogram_quantile(0.50, sum by (le) (rate(airpipe_interface_duration_seconds_bucket{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval])))",
"legendFormat": "p50",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "histogram_quantile(0.95, sum by (le) (rate(airpipe_interface_duration_seconds_bucket{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval])))",
"legendFormat": "p95",
"refId": "B",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "histogram_quantile(0.99, sum by (le) (rate(airpipe_interface_duration_seconds_bucket{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval])))",
"legendFormat": "p99",
"refId": "C",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sum(rate(airpipe_interface_duration_seconds_sum{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval])) / sum(rate(airpipe_interface_duration_seconds_count{config=~\"$config\", interface=~\"$interface\"}[$__rate_interval]))",
"legendFormat": "mean",
"refId": "D",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": true
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 14,
"type": "row",
"title": "Plan",
"collapsed": false,
"gridPos": {
"x": 0,
"y": 31,
"w": 24,
"h": 1
},
"panels": []
},
{
"id": 15,
"type": "timeseries",
"title": "Series in use",
"description": "Per worker, against the limit. Interfaces idle for 15 minutes free their series.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 0,
"y": 32,
"w": 12,
"h": 7
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "max(airpipe_hosted_metrics_series)",
"legendFormat": "in use (busiest worker)",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "max(airpipe_hosted_metrics_series_limit)",
"legendFormat": "limit",
"refId": "B",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": true
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 16,
"type": "timeseries",
"title": "Requests not recorded",
"description": "Requests whose interface needed a new series past the limit. Raise the limit with your plan, or let idle interfaces age out.",
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"gridPos": {
"x": 12,
"y": 32,
"w": 12,
"h": 7
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_AIRPIPE}"
},
"expr": "sum by (reason) (increase(airpipe_hosted_metrics_dropped_runs_total[$__rate_interval]))",
"legendFormat": "{{reason}}",
"refId": "A",
"range": true,
"instant": false
}
],
"interval": "30s",
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": true
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
}
]
}