Container Service Discovery

Category: Engineering & DevOps · 🔒 Self-hosted only

This page is generated from the Air Pipe marketplace. Browse it live to install into your organization.

Find the containers behind a service — by DNS, by Docker/Compose label, or from an explicit list — then do something to every one of them.

Scale a Compose service to three and the third container is discovered automatically. No orchestrator API, no service registry, no list of addresses to maintain.

Self-hosted only. Every route asks your own host or network about itself: the docker backend talks to the Docker Engine socket, and the dns backend resolves names on your container network. Managed (hosted) Air Pipe has no route to either.

On Kubernetes? Use the Kubernetes Service Discovery pack — same shape, plus label selectors and pod metadata through the API server.

What's included

File	Purpose
`discovery.yml`	Four routes: peers by DNS, containers by label, an explicit list, and a health fan-out

Endpoints

Route	Method	Backend	Needs the Docker socket
`/containers/peers`	GET	`dns`	➖
`/containers/list`	GET	`docker`	✅
`/containers/static`	GET	`static`	➖
`/containers/health`	GET	`dns` + fan-out	➖

Start with /containers/peers. Under Compose it discovers the whole service with no privilege at all.

Requirements

A self-hosted Air Pipe agent, engine 1.40.4+ (the member list and the port both come from variables)
For /containers/list only: access to the Docker Engine socket

The three ways, and what each costs

DNS — no privilege

Compose's embedded DNS returns one address per container for a scaled service name:

docker compose up -d --scale api=3
curl -s localhost:4111/containers/peers | jq '.data.Peers.data'

The same route works anywhere with DNS — Swarm, Nomad, a round-robin A record, or Kubernetes against a headless Service. Nothing to mount, nothing to authorise.

Docker labels — needs the socket

Compose stamps every container it starts with com.docker.compose.service, which makes that label the natural handle:

curl -s "localhost:4111/containers/list?service=api" | jq '.data.Containers.data'

You get the image and the full label set, which DNS cannot give you.

⚠️ The Docker socket is host-root-equivalent. Anything that can reach /var/run/docker.sock can start a privileged container mounting the host filesystem — it is effectively root on that machine. Mount it read-only, only where you actually need this route, and prefer the DNS route where it will do.

Unix only: Windows exposes the daemon on a named pipe, and the backend reports that rather than failing obscurely.

docker-compose.yml
services:
  airpipe:
    image: airpipeio/agent:1.42.1
    # The image runs as a non-root user, and the socket is mode 660 owned by the
    # host's `docker` group — so mounting it is NOT enough on its own. Add that
    # group, or every call fails with "Permission denied (os error 13)":
    #
    #   getent group docker | cut -d: -f3
    #
    group_add:
      - "999"                                          # your host's docker gid
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro   # only if you need /containers/list
      - ./configs:/app/configs:ro

Static — no orchestrator at all

An explicit list, returned in the same item shape as every other backend. That is the point: start static, move to real discovery later, and the actions consuming it never change.

Setup

Container identity (optional)

exclude_self stops a service calling itself. A container's own address is only known at runtime, so export it in the entrypoint:

entrypoint:
  - /bin/sh
  - -c
  - 'export CONTAINER_IP="$$(hostname -i)"; exec /usr/local/bin/airpipe server --address 0.0.0.0 --port 4111 --config-dir /app/configs'

Leave it unset and the route still works — you simply appear in your own results.

Variables

Variable	Default	Purpose
`SERVICE_NAME`	`api`	DNS name to resolve — a Compose service name
`SERVICE_PORT`	`4111`	Port applied to discovered members
`MEMBERS`	`127.0.0.1:4111`	The static list, comma-separated: `10.0.0.1:4111, 10.0.0.2:4111`
`HEALTH_PATH`	`/livez`	Path called on each member by the health route

Quick start

curl -s localhost:4111/containers/peers  | jq '.data.Peers.data'
curl -s "localhost:4111/containers/list?service=api" | jq '.data.Containers.data'
curl -s localhost:4111/containers/health | jq '.data.CheckAll.data'

GET /containers/list?service=api
[
  {
    "name": "myproject-api-1",
    "address": "172.19.0.4",
    "port": 4111,
    "url": "http://172.19.0.4:4111",
    "source": "docker",
    "image": "example/api:1.4.0",
    "labels": { "com.docker.compose.service": "api" }
  }
]

GET /containers/health

{ "succeeded": [ { "data": { "Health": { "data": { "status": "ok" } } } } ], "failed": [] }

How it works

Discovery returns an array, and lookup: runs the nested actions once per member with that member as their input:

- name: Peers
  discover:
    dns: { name: api, exclude_self: a|env::CONTAINER_IP->default()| }
    port: 4111

- name: CheckAll
  run_when_succeeded: [Peers]   # wait for discovery before fanning out
  lookup: a|Peers|
  lookup_partition: true
  actions:
    - name: Health
      http: { url: a|body::url|/livez }

lookup_partition: true splits the result into succeeded and failed, so a container that does not answer is reported rather than silently dropped.

Notes

Every backend returns the same item shape — { name, address, port, url, source, image, labels, … } — so switching backend does not change the actions that consume it.
Fields a backend cannot know are omitted, not faked. DNS has no image, so an image filter matches nothing there rather than silently matching everything.
Exclusion beats inclusion. /containers/list drops anything named canary whatever the label matched.
Discovery is cached for 10s (cache_ttl_secs). Without it, a discovery action on a request path would hit the Docker socket once per request. Set 0 to disable.
static takes the whole list from one variable, comma-separated, or a YAML sequence with one entry per member. Either shape is split after interpolation, so a variable holding a:80, b:80 becomes two members and an entry whose variable is unset drops out instead of becoming a phantom target.
port takes a number or a variable. It is resolved after interpolation, so a|ap_var::SERVICE_PORT->default(4111)| works; before 1.40.4 a marker here stopped the agent from starting. A per-member port can also go in the member itself (host:port), which the static list accepts.
lookup_concurrency: 10 caps the blast radius on a large fleet.

Customisation

Another label scheme: swap com.docker.compose.service for any label your tooling sets.
Multiple networks: add network: <name> under docker: to choose which address is reported.
Something other than a health check: replace the Health action with any action — the fan-out does not care what it runs.

Configuration

discovery.yml

name: ContainerServiceDiscovery
description: >
  Discover the containers behind a service — by Compose/Docker label, by DNS, or
  from an explicit list — and act on every one of them. Self-hosted only: the
  agent discovers containers on the host or network it runs in.

docs: true

# SELF-HOSTED ONLY. Every route here asks your own host or network about itself:
#   * the `docker` backend talks to the Docker Engine socket
#   * the `dns` backend resolves names on your container network
# Managed (hosted) Air Pipe has no route to either.
#
# For Kubernetes, use the Kubernetes Service Discovery pack instead — it adds
# label selectors and pod metadata through the API server.

interfaces:

  # GET /containers/peers
  #
  # START HERE. Under Docker Compose, a scaled service name resolves to one
  # address per container, so this discovers the whole service with NO socket
  # access and no privilege at all:
  #
  #   docker compose up -d --scale api=3
  #
  # The same route works in any environment with DNS — Swarm, Nomad, a plain
  # round-robin A record, or Kubernetes against a headless Service.
  containers/peers:
    output: http
    method: GET
    summary: Peers via DNS (no privilege)
    description: Every container behind a service name, discovered by DNS.
    tags: [containers, discovery, dns, compose]
    response_example:
      - name: 172.19.0.4
        address: 172.19.0.4
        port: 4111
        url: http://172.19.0.4:4111
        source: dns

    actions:
      - name: Peers
        discover:
          dns:
            name: a|ap_var::SERVICE_NAME->default(api)|
            # Drop our own address so a service does not call itself. In Compose
            # the container's own IP is only known at runtime — the README shows
            # how to export it.
            exclude_self: a|env::CONTAINER_IP->default()|
          port: a|ap_var::SERVICE_PORT->default(4111)|

  # GET /containers/list
  # GET /containers/list?service=api
  #
  # The Docker Engine view: every RUNNING container carrying a label, with its
  # image and full label set. Compose stamps every container it starts with
  # `com.docker.compose.service`, so that label is the natural handle.
  #
  # Needs the Docker socket — see the warning in the README before enabling it.
  containers/list:
    output: http
    method: GET
    summary: List containers by label
    description: Running containers matching a Docker/Compose label, with image and labels.
    tags: [containers, discovery, docker, compose]
    response_example:
      - name: myproject-api-1
        address: 172.19.0.4
        port: 4111
        url: http://172.19.0.4:4111
        source: docker
        image: example/api:1.4.0
        labels:
          com.docker.compose.service: api

    actions:
      - name: Selector
        input: a|params|
        hide_data_on_success: true
        post_transforms:
          - add_attribute:
              service: a|params::service->default(api)|

      - name: Containers
        run_when_succeeded: [Selector]
        discover:
          docker:
            labels:
              com.docker.compose.service: a|Selector::service|
          port: a|ap_var::SERVICE_PORT->default(4111)|
          # Never target a canary from an automated route. Exclusion beats
          # inclusion, so this wins over any label match above.
          exclude:
            name: canary

  # GET /containers/static
  #
  # No orchestrator at all — an explicit list, in the SAME item shape as every
  # other backend. That is the point: a config can start here and move to real
  # discovery later without touching the actions that consume it.
  containers/static:
    output: http
    method: GET
    summary: An explicit list
    description: A fixed set of endpoints, in the same shape the other backends return.
    tags: [containers, discovery, static]

    actions:
      - name: Members
        discover:
          # The whole list from ONE variable, comma-separated — so adding a
          # member is editing a variable, not editing the config:
          #
          #   MEMBERS = 10.0.0.1:4111, 10.0.0.2:4111
          #
          # A YAML sequence works too, one entry per member. Either way an entry
          # whose variable is unset simply drops out, so a half-filled list still
          # discovers the members it does have.
          static: a|ap_var::MEMBERS->default(127.0.0.1:4111)|
          port: a|ap_var::SERVICE_PORT->default(4111)|

  # GET /containers/health
  #
  # The point of discovery: find the members, then DO something to each one.
  # `lookup_partition` splits the result into succeeded/failed, so a container
  # that does not answer is reported rather than silently dropped.
  containers/health:
    output: http
    method: GET
    summary: Health-check every container
    description: Discovers containers and calls a health path on each, reporting which answered.
    tags: [containers, discovery, health]
    response_example:
      succeeded:
        - data:
            Health:
              data:
                status: ok
      failed: []

    actions:
      - name: Peers
        discover:
          dns:
            name: a|ap_var::SERVICE_NAME->default(api)|
            exclude_self: a|env::CONTAINER_IP->default()|
          port: a|ap_var::SERVICE_PORT->default(4111)|
        hide_data_on_success: true

      - name: CheckAll
        run_when_succeeded: [Peers]
        lookup: a|Peers|
        lookup_partition: true
        # Cap the blast radius on a large fleet: 10 in flight at a time.
        lookup_concurrency: 10
        actions:
          - name: Health
            http:
              url: a|body::url|a|ap_var::HEALTH_PATH->default(/livez)|
              timeout: 3s
            post_transforms:
              - extract_value: body

Container Service Discovery

What's included​

Endpoints​

Requirements​

The three ways, and what each costs​

DNS — no privilege​

Docker labels — needs the socket​

Static — no orchestrator at all​

Setup​

Container identity (optional)​

Variables​

Quick start​

How it works​

Notes​

Customisation​

Configuration​

discovery.yml​