Waiting a long time

Some work is not finished when you ask for it. A video render, a model inference job, a bulk import: you submit it, get an id, and the answer arrives minutes or hours later.

Holding the request open for that is the wrong instrument. The connection stays occupied, a proxy or client timeout cuts it off long before an hour is up, and a restart loses the run entirely, halfway through, after the first API call has already been made.

So past a threshold Air Pipe suspends the run instead of waiting in it. What has completed is written to a durable store, the caller is answered immediately, and the scheduler resumes the rest when it comes due.

A fixed wait

- name: Confirm
  json_output: '{"confirmed": true}'

- name: Remind
  run_when_succeeded: [Confirm]
  delay: "1d"

- name: SendFollowUp
  run_when_succeeded: [Remind]
  email:
    to: a|body::email|
    subject: How did it go?

Under AIRPIPE__MAX_DELAY_SECS (default 300s) a delay simply sleeps — that is ordinary pacing between steps and nothing changes. Above it the run is suspended, and the caller gets:

{ "deferred": true, "run_uuid": "01a0463b-…", "resume_in_secs": 86400 }

with HTTP 202. A day later the scheduler picks the run up, restores what had already run, and continues from SendFollowUp. Confirm is not run again — its result comes back out of the store, so a|Confirm::id| resolves exactly as it did the day before.

Waiting for something to finish

A delay wakes once, at the time you asked for. That is fine for "tomorrow" and wrong for "when the render is done", because you are guessing: too short and you read an unfinished job, too long and you wait for nothing.

Poll instead. An assert that fails while the job is still running turns retry into a poll:

- name: Submit
  http:
    url: https://api.example.com/jobs
    method: POST
    body: a|body|

- name: AwaitResult
  run_when_succeeded: [Submit]
  http:
    url: https://api.example.com/jobs/a|Submit::body.id|
  assert:
    http_code_on_error: 504
    error_message: the job did not finish in time
    tests:
      - value: body.status
        is_equal_to: completed
  retry:
    attempts: 40
    delay: 60000

A job still running answers 200 with a status that is not completed — a success as far as HTTP is concerned — so the assert is what decides to go round again. retry here means "until the answer changes", not only "after an error".

The waits add up past the in-process window almost immediately, so from then on the run is suspended between polls rather than sleeping through them. Forty attempts a minute apart is a forty-minute ceiling that costs a database row, not an open request. When the ceiling is reached the action fails with its http_code_on_error, so a job that never finishes surfaces as an error instead of an empty result.

If the upstream reports failure as a status of its own (failed, errored), add a test for it so a dead job stops the poll immediately instead of retrying to the ceiling.

Waiting until a specific time

A duration is what delay takes, but a deadline is often what you have. until(seconds) is the gap between now and a datetime, and the trailing s makes it a duration:

- name: HoldUntilDue
  delay: "a|body::due_at->until(seconds)|s"

Same behaviour as any other long wait — parked if it is past the in-process window, resumed when it comes due.

Waiting for a person

Sometimes the thing you are waiting for is not a time at all. An approval, a signature, a third party's webhook: you cannot predict when it arrives, and a delay long enough to cover it would be a guess in both directions.

wait_for_callback parks the run until someone calls in:

- name: Ticket
  json_output: '{"token": "a|uuid|"}'

- name: AskApprover
  run_when_succeeded: [Ticket]
  email:
    to: [email protected]
    subject: Approve this refund
    body: |
      Approve: https://api.example.com/_ap/resume/a|org_uuid|/a|Ticket::token|

- name: Approval
  run_when_succeeded: [AskApprover]
  wait_for_callback:
    token: a|Ticket::token|
    timeout: "7d"

- name: Refund
  run_when_succeeded: [Approval]
  http:
    url: https://api.example.com/refunds
    method: POST
    body: '{"order": "a|body::order_id|", "approved_by": "a|Approval::approved_by|"}'

POST /_ap/resume/<org_uuid>/<token> resumes the run, and whatever was posted becomes the waiting action's output — so a|Approval::approved_by| above reads a field from the approver's own request body.

The token is the credential

You mint it, because you are the one who has to put it in a link. Use a|uuid|, not something predictable like an order number: anyone holding the token can resume the run, exactly as with a password-reset link.

The organisation is in the URL and is not a secret — it already appears in every managed route. It is there so a token guessed in one organisation cannot reach a run in another, which matters because every managed tenant shares one table of parked runs. a|org_uuid| resolves it, so a config can build its own link.

An unknown token, an already-used one, an expired one, and one belonging to another organisation all answer the same bare 404. Telling them apart would tell someone guessing which tokens exist.

A deadline that passes is a failure

timeout is required and cannot be unbounded — a wait nobody ever answers would hold a row forever. When it expires the waiting action fails, so everything gated on it with run_when_succeeded does not run.

That is deliberate. If a timed-out approval completed quietly, the refund above would go out unapproved.

What it costs

	held open	survives a restart	returns when ready
delay under the window	yes, briefly	no	n/a — fixed wait
delay over the window	no	yes	no — wakes once
retry + assert	only until the window	yes, once suspended	yes
wait_for_callback	no	yes	yes — the moment someone calls

A resume is picked up on a 30-second tick, so it can be up to half a minute late. That is immaterial for hours and days; it is not a precise timer.

Requirements

Suspending needs somewhere durable to write.

• Managed — already there, nothing to configure.
• Self-hosted — set AIRPIPE__DATABASE_URL. The scheduler already uses this database, and the suspended runs live alongside its own tables.

How many runs may wait at once

A parked run is a row now and a whole interface execution later, so the number one organisation may hold is capped.

• Managed — max_deferred_runs, 1,000 by default, raisable for your organisation without a deploy. Ask if your workload genuinely needs more.
• Self-hosted — AIRPIPE__MAX_DEFERRED_RUNS, 100,000 by default. The database is yours; this is a runaway guard, not a commercial limit.

At the ceiling a new suspension is refused, with an error saying so. It is not queued silently — a caller told 202 for a run that was never stored would wait for something that is never going to happen.

Without it a long delay is refused, with an error saying so. It is deliberately not shortened to fit the in-process window: that would continue the run at the wrong time and report nothing unusual.

Where suspended runs live

In ap_deferred_runs, one row per parked run, which you can inspect like any other table:

-- what is waiting, and when it comes due
SELECT run_uuid, config_name, interface_name, resume_at, attempts
FROM ap_deferred_runs
ORDER BY resume_at;

-- how close an organisation is to its ceiling
SELECT org_uuid, count(*) FROM ap_deferred_runs GROUP BY org_uuid ORDER BY 2 DESC;

-- runs waiting on a person rather than a clock
SELECT run_uuid, config_name, resume_at AS deadline
FROM ap_deferred_runs WHERE callback_token IS NOT NULL;

Rows are claimed with FOR UPDATE SKIP LOCKED, so several engines share one queue without two of them resuming the same run. A claim defers the row rather than deleting it, so an engine that dies mid-resume returns the work instead of losing it.

Seeing them on managed Air Pipe

A managed organisation has no database of its own to query, so the same two answers are a screen instead. Waiting Runs, under the organisation menu in the app, lists what is parked — config, interface, when it comes due, how many attempts it has already made — and cancels a run that should not resume. The list is scoped to your organisation and paginated at 100 a page.

Both are gated by the role scopes you already use for executions: execution:list to see the list, execution:delete to cancel. Cancelling deletes the parked run, so the rest of that workflow never happens and its callback token stops working. It does not undo the actions that already ran before the run parked — there is nothing to roll back to.

See also

• The Async Job Polling pack — the whole pattern, with a fake job API you can run against with no credentials.
• Scheduling — for work that starts on a clock rather than continuing after a wait.