Kubernetes Service Discovery

Category: Engineering & DevOps · 🔒 Self-hosted only

This page is generated from the Air Pipe marketplace. Browse it live to install into your organization.

Ask your cluster about itself. Discover the pods behind a Deployment — by label, with readiness, node and image — then do something to every one of them: health-check each replica, or broadcast a call to the whole fleet.

Air Pipe discovers the members and fans your actions out across them, so you stop maintaining a list of addresses that is wrong the moment anything scales.

Self-hosted only. Every route here asks the cluster about itself: the kubernetes backend authenticates with the pod's own ServiceAccount, and the dns backend resolves in-cluster Service names. Managed (hosted) Air Pipe has no route to your API server or your cluster DNS. Run this with a self-hosted agent inside the cluster.

Not on Kubernetes? Use the Container Service Discovery pack — same shape, Docker/Compose/DNS backends.

What's included

File	Purpose
`discovery.yml`	Four routes: list pods, find peers, health-check every replica, broadcast to all

Endpoints

Route	Method	What it does	Needs RBAC
`/k8s/pods`	GET	Ready pods matching a label, with node, image, labels	✅
`/k8s/peers`	GET	The same pods via DNS — no API access at all	➖
`/k8s/health`	GET	Calls `/livez` on every replica, reports which answered	➖
`/k8s/broadcast`	POST	Calls one path on every pod (cache flush, reload, …)	➖

Three of the four need no privilege whatsoever — they use DNS. Only /k8s/pods talks to the API server, and only because label selectors and pod metadata live there.

Requirements

A self-hosted Air Pipe agent running inside the cluster
Engine 1.40.1+ (the discover action)
A headless Service for the DNS routes (clusterIP: None)
For /k8s/pods only: a ServiceAccount with list on pods

Setup

1. A headless Service

clusterIP: None is the important line — it makes DNS return one address per ready pod instead of a single virtual IP.

apiVersion: v1
kind: Service
metadata:
  name: airpipe-mesh
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: airpipe
  ports:
    - { name: http, port: 4111, targetPort: 4111 }

2. RBAC — only for `/k8s/pods`

apiVersion: v1
kind: ServiceAccount
metadata:
  name: airpipe
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: airpipe-discovery
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: airpipe-discovery
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: airpipe-discovery
subjects:
  - kind: ServiceAccount
    name: airpipe

list on pods and nothing else — no watch, no get, nothing on secrets or configmaps. It is namespace-scoped, so discovering another namespace means a RoleBinding in that namespace, which keeps every grant visible where it applies.

Skip this and /k8s/pods fails with a 500 whose error names the fix and quotes the API's own 403 — the 403 is the cluster refusing the agent, not the caller. The other three routes still work.

3. Pod identity

exclude_self stops a pod calling itself, and needs the pod's own address:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Remember serviceAccountName: airpipe on the pod spec if you want /k8s/pods.

4. Variables

Variable	Default	Purpose
`AIRPIPE_MESH_SERVICE`	`airpipe-mesh`	Headless Service name for the DNS routes

Quick start

# Every ready pod carrying app.kubernetes.io/name=airpipe
curl -s localhost:4111/k8s/pods | jq '.data.Pods.data'

# Target a different app
curl -s "localhost:4111/k8s/pods?app=api" | jq '.data.Pods.data'

# Health-check every replica
curl -s localhost:4111/k8s/health | jq '.data.CheckAll.data'

# Broadcast a call to every pod
curl -s -X POST localhost:4111/k8s/broadcast \
  -H 'content-type: application/json' \
  -d '{"path":"/livez"}' | jq '.data.FanOut.data'

GET /k8s/pods
[
  {
    "name": "airpipe-7b5b7df54-kc2kv",
    "address": "10.42.1.20",
    "port": 4111,
    "url": "http://10.42.1.20:4111",
    "source": "kubernetes",
    "ready": true,
    "namespace": "airpipe",
    "node": "k3s-ap-agent-1",
    "image": "airpipeio/agent:1.42.1",
    "labels": { "app.kubernetes.io/name": "airpipe" }
  }
]

GET /k8s/health

{ "succeeded": [ { "data": { "Livez": { "data": { "status": "ok" } } } } ], "failed": [] }

How it works

Discovery returns an array, and lookup: runs the nested actions once per member with that member as their input — so a|body::url| is the member's URL:

- name: Peers
  discover:
    dns: { name: airpipe-mesh, exclude_self: a|env::POD_IP->default()| }
    port: 4111

- name: CheckAll
  run_when_succeeded: [Peers]   # wait for discovery before fanning out
  lookup: a|Peers|
  lookup_partition: true
  actions:
    - name: Livez
      http: { url: a|body::url|/livez }

lookup_partition: true splits the result into succeeded and failed, so a pod that does not answer is reported rather than silently dropped — the difference between "all good" and "two of three replied".

Notes

Prefer the DNS routes. They discover the same pods with no API access, no credentials and no RBAC. Move up to /k8s/pods only when you need label selectors or pod metadata.
ready_only defaults to true. An unready pod has explicitly said not to send it traffic.
port_name: http beats a hard-coded port — the name survives a port change.
Exclusion beats inclusion. /k8s/pods excludes anything named canary whatever the selector matched; when the two disagree, "do not touch this one" wins.
Discovery is cached for 10s by default (cache_ttl_secs). Without it, a discovery action on a request path would issue an API call per request and become a denial-of-service against your own API server. Set 0 to disable.
/k8s/broadcast validates the path rather than taking it as given — an unchecked path would let a caller aim the fan-out at any route on every pod at once. Adapt that assert before exposing it publicly.
lookup_concurrency: 10 caps the blast radius on a large Deployment.

Customisation

Different app: ?app=<label value>, or edit the label_selector default.
Something other than a health check: swap the Livez action for any action — a DB write, a queue publish, a webhook.
Aggregate rather than list: add a post_transforms step after CheckAll to count or reduce the per-pod results.

Configuration

discovery.yml

name: KubernetesServiceDiscovery
description: >
  Discover your own Kubernetes pods and act on every one of them — list them with
  label selectors, health-check each replica, and broadcast a call to the whole
  Deployment. Self-hosted only: the agent discovers pods from inside the cluster.

docs: true

# SELF-HOSTED ONLY. Every route here asks the cluster about itself, which only
# works from an agent running inside it:
#   * the `kubernetes` backend authenticates with the pod's own ServiceAccount
#   * the `dns` backend resolves in-cluster Service names
# Managed (hosted) Air Pipe has no route to your cluster's API server or DNS.
#
# The `kubernetes` backend needs a ServiceAccount with `list` on pods. The
# README has the Role and RoleBinding; without it the action returns a 403 that
# names the fix.

interfaces:

  # GET /k8s/pods
  # GET /k8s/pods?app=api
  #
  # Every ready pod matching a label, with the metadata the API server knows:
  # node, readiness, image and labels. `port_name` takes the port from a NAMED
  # container port, which survives a port change in a way a hard-coded number
  # does not.
  k8s/pods:
    output: http
    method: GET
    summary: List pods by label
    description: Ready pods matching a label selector, with node, image and labels.
    tags: [kubernetes, discovery]
    response_example:
      - name: api-7d9f8b6c5d-x2ktp
        address: 10.42.1.3
        port: 8080
        url: http://10.42.1.3:8080
        source: kubernetes
        ready: true
        namespace: default
        node: worker-1
        image: example/api:1.4.0
        labels:
          app: api

    actions:
      # Default to this Deployment's own label so the route works before you
      # change anything. Override per call with ?app=<label value>.
      - name: Selector
        input: a|params|
        hide_data_on_success: true
        post_transforms:
          - add_attribute:
              app: a|params::app->default(airpipe)|

      - name: Pods
        run_when_succeeded: [Selector]
        discover:
          kubernetes:
            label_selector: app.kubernetes.io/name=a|Selector::app|
            port_name: http
            ready_only: true
          # Never target a canary from an automated route, whatever the selector
          # matched. Exclusion beats inclusion, so this wins.
          exclude:
            name: canary

  # GET /k8s/peers
  #
  # The same pods, found WITHOUT touching the API server. A headless Service
  # (clusterIP: None) resolves to one address per ready pod, so this needs no
  # ServiceAccount, no RBAC and no credentials at all.
  #
  # Prefer this wherever addresses are enough — it is the cheapest thing in the
  # pack, in privilege and in latency.
  k8s/peers:
    output: http
    method: GET
    summary: Peers via DNS (no RBAC)
    description: Every ready pod behind the headless Service, discovered by DNS.
    tags: [kubernetes, discovery, dns]

    actions:
      - name: Peers
        discover:
          dns:
            name: a|ap_var::AIRPIPE_MESH_SERVICE->default(airpipe-mesh)|
            # Drop our own address: a pod does not need to call itself. The
            # deployment supplies POD_IP from the downward API.
            exclude_self: a|env::POD_IP->default()|
          port: 4111

  # GET /k8s/health
  #
  # The point of discovery: find the members, then DO something to each one.
  # `lookup_partition` splits the result into succeeded/failed, so a pod that
  # does not answer is reported rather than silently dropped.
  k8s/health:
    output: http
    method: GET
    summary: Health-check every replica
    description: Discovers peers and calls /livez on each, reporting which answered.
    tags: [kubernetes, discovery, health]
    response_example:
      succeeded:
        - data:
            Livez:
              data:
                status: ok
      failed: []

    actions:
      - name: Peers
        discover:
          dns:
            name: a|ap_var::AIRPIPE_MESH_SERVICE->default(airpipe-mesh)|
            exclude_self: a|env::POD_IP->default()|
          port: 4111
        hide_data_on_success: true

      - name: CheckAll
        run_when_succeeded: [Peers]
        lookup: a|Peers|
        lookup_partition: true
        # Cap the blast radius on a large Deployment: 10 in flight at a time.
        lookup_concurrency: 10
        actions:
          - name: Livez
            http:
              url: a|body::url|/livez
              timeout: 3s
            post_transforms:
              - extract_value: body

  # POST /k8s/broadcast   { "path": "/livez" }
  #
  # Fan an HTTP call out to every pod — the shape you want for cache
  # invalidation, config reload, or any "tell all replicas" operation that has no
  # shared bus behind it.
  #
  # The path is validated rather than taken as given: an unchecked path here
  # would let a caller aim this at any route on every pod at once.
  k8s/broadcast:
    output: http
    method: POST
    summary: Broadcast a call to every pod
    description: Calls the same path on every discovered pod and reports per-pod results.
    tags: [kubernetes, discovery, fan-out]
    request_example:
      path: /livez
    response_example:
      succeeded:
        - data:
            Call:
              data:
                status: ok
      failed: []

    actions:
      - name: CheckBody
        input: a|body|
        hide_data_on_success: true
        assert:
          http_code_on_error: 400
          error_message: "path is required and must start with /"
          tests:
            - value: path
              is_not_null: true
              description: "Path to call on every pod, e.g. /livez"
            - value: path
              starts_with: "/"

      - name: Peers
        run_when_succeeded: [CheckBody]
        discover:
          dns:
            name: a|ap_var::AIRPIPE_MESH_SERVICE->default(airpipe-mesh)|
            exclude_self: a|env::POD_IP->default()|
          port: 4111
        hide_data_on_success: true

      - name: FanOut
        run_when_succeeded: [Peers]
        lookup: a|Peers|
        lookup_partition: true
        lookup_concurrency: 10
        actions:
          - name: Call
            http:
              url: a|body::url|a|CheckBody::path|
              timeout: 5s
            post_transforms:
              - extract_value: body

Kubernetes Service Discovery

What's included​

Endpoints​

Requirements​

Setup​

1. A headless Service​

2. RBAC — only for /k8s/pods​

3. Pod identity​

4. Variables​

Quick start​

How it works​

Notes​

Customisation​

Configuration​

discovery.yml​