S3 Object Storage

Category: Backend & APIs

This page is generated from the Air Pipe marketplace. Browse it live to install into your organization.

Private file storage on S3-compatible object storage, where the file never passes through Air Pipe. Air Pipe holds the metadata and mints short-lived presigned URLs; the bytes go client ↔ S3 directly.

Works with anything S3-compatible — AWS S3, Cloudflare R2, MinIO, Backblaze B2. Only S3_ENDPOINT and S3_REGION change.

Requires an Air Pipe engine with operation: put on s3_generate_presigned_url (added in 1.37.0). Earlier engines can only presign downloads.

Why presigned URLs rather than proxying the upload

Routing uploads through your API means every large file is bounded by your request timeout, your memory, and your bandwidth bill. Presigning avoids all three:

No size ceiling — a 2 GB upload never touches Air Pipe.
No proxy cost or latency — the client talks to S3 directly.
The bucket stays private — no public objects. Every download is a fresh link that expires (5 minutes here), so a leaked URL stops working.
Signatures are method-scoped — a download link cannot be replayed as an upload.

Endpoints

Method	Route	Purpose
POST	`/files/upload-url`	Record a pending object, return a presigned PUT (15 min).
POST	`/files/confirm`	Mark the object stored once the client's upload succeeded.
GET	`/files/download`	Return a presigned GET (5 min) for a stored object.
GET	`/files`	List stored objects, newest first (`?owner=`, `?limit=`).
POST	`/api/seed`	Create the `s3_objects` table (from `seed.yml`).

Setup

Create a bucket and an access key with read/write on it. Keep the bucket private — this pack works by minting links, not by public objects.
Set the managed variables:
- DATABASE_URL — Postgres connection string
- S3_BUCKET, S3_ENDPOINT, S3_REGION, S3_ACCESS_KEY_ID, S3_SECRET_KEY
For Cloudflare R2, S3_ENDPOINT is https://<account-id>.r2.cloudflarestorage.com and S3_REGION is auto.
Seed the table: POST /api/seed.
Set CORS on the bucket to allow PUT from your web origin, or the browser upload in step 2 below is blocked before it starts.

The upload flow

# 1. ask for an upload URL
curl -s -X POST https://<your-host>/files/upload-url \
  -H 'content-type: application/json' \
  -d '{"filename":"report.pdf","content_type":"application/pdf","owner":"alice"}'

{ "file_id": "019f97…", "upload_url": "https://…X-Amz-Signature=…", "expires_in": 900 }

# 2. PUT the bytes STRAIGHT to S3 — this request does not go to Air Pipe
curl -s -X PUT "<upload_url>" \
  -H 'content-type: application/pdf' \
  --data-binary @report.pdf

# 3. confirm, so the object flips from pending to stored
curl -s -X POST https://<your-host>/files/confirm \
  -H 'content-type: application/json' \
  -d '{"file_id":"019f97…","size_bytes":12345}'

Downloading

curl -s "https://<your-host>/files/download?file_id=019f97…"

{ "file_id": "019f97…", "filename": "report.pdf", "download_url": "https://…", "expires_in": 300 }

The link is valid for 5 minutes. Mint a new one per request rather than storing them.

How it works

upload-url — validates the filename, generates a uuid and namespaces the object key as <uuid>/<filename> so two uploads of the same name can never collide, inserts the row as pending, then presigns a PUT.
confirm — flips pending → stored and records the size. The UPDATE … WHERE status = 'pending' plus a [0].id is_not_null assert means confirming an unknown or already- confirmed id is a clean 404, not a silent no-op.
download — looks the object up (must be stored), presigns a GET.
files — lists stored objects; ?owner= is applied only when non-empty, so the same query serves both filtered and unfiltered listings.

Housekeeping

Rows left pending are uploads that were never completed — the URL expired, or the client went away. They hold no bytes. Sweep them on a schedule:

DELETE FROM s3_objects WHERE status = 'pending' AND created_at < NOW() - INTERVAL '1 day';

The Scheduled Cleanup pack shows that pattern with a schedule: block.

Tuning the TTLs

upload_ttl_secs (900) and download_ttl_secs (300) are in global.variables. The upload window only needs to cover starting the transfer, and a download link should be short — long enough to click, short enough that sharing it is useless.

Configuration

config.yml

name: S3ObjectStorage

docs: true

# Private file storage on S3-compatible object storage, with the file itself never
# passing through Air Pipe.
#
# The pattern is presigned URLs on both sides:
#   1. the client asks for an upload URL — Air Pipe records the pending object and
#      mints a short-lived presigned PUT
#   2. the client PUTs the bytes STRAIGHT to S3 (no size limit, no proxying, no
#      request-timeout risk on large files)
#   3. the client confirms, and Air Pipe marks the object stored
#   4. downloads are short-lived presigned GETs minted per request, so the bucket
#      itself stays private and links expire
#
# Works with anything S3-compatible: AWS S3, Cloudflare R2, MinIO, Backblaze B2 —
# only `endpoint` and `region` change.
#
# Required managed variables:
#   DATABASE_URL      — Postgres connection string (object metadata)
#   S3_BUCKET         — bucket name
#   S3_ENDPOINT       — e.g. https://s3.eu-west-1.amazonaws.com
#                       or https://<account>.r2.cloudflarestorage.com for R2
#   S3_REGION         — e.g. eu-west-1 (use "auto" for R2)
#   S3_ACCESS_KEY_ID  — access key
#   S3_SECRET_KEY     — secret key

global:
  variables:
    upload_ttl_secs: 900     # 15 minutes to start the upload
    download_ttl_secs: 300   # 5 minutes for a download link

  databases:
    main:
      driver: postgres
      conn_string: "a|ap_var::DATABASE_URL|"

interfaces:

  # POST /files/upload-url
  # Body: { "filename": "report.pdf", "content_type": "application/pdf", "owner": "alice" }
  # Returns a presigned PUT the client uploads to directly.
  files/upload-url:
    output: http
    method: POST
    summary: Mint a presigned upload URL and record the pending object
    description: >
      Records the object as pending and returns a short-lived presigned PUT URL.
      The client uploads straight to S3, so the file never transits Air Pipe.
    tags: [s3, storage, upload, presigned]
    request_example: |-
      { "filename": "report.pdf", "content_type": "application/pdf", "owner": "alice" }
    response_example: |-
      { "file_id": "019f97…", "upload_url": "https://…X-Amz-Signature=…", "expires_in": 900 }

    actions:
      - name: ValidateBody
        input: a|body|
        hide_data_on_success: true
        assert:
          http_code_on_error: 400
          error_message: "filename is required"
          tests:
            - value: filename
              is_not_null: true
              is_not_empty: true

      # The object key is namespaced by a fresh uuid so two uploads of the same
      # filename can never collide or overwrite each other.
      - name: NewObject
        run_when_succeeded: [ValidateBody]
        json_output: |-
          { "ready": true }
        post_transforms:
          - add_attribute:
              file_id: a|uuid|

      - name: Record
        run_when_succeeded: [NewObject]
        database: main
        query: |-
          INSERT INTO s3_objects (id, object_key, filename, content_type, owner, status)
          VALUES (
            a|NewObject::file_id->single_quote|,
            a|NewObject::file_id->single_quote| || '/' || a|ValidateBody::filename->single_quote|,
            a|ValidateBody::filename->single_quote|,
            a|ValidateBody::content_type->default('application/octet-stream')->single_quote|,
            a|ValidateBody::owner->default('')->single_quote|,
            'pending'
          )
          RETURNING id, object_key
        post_transforms:
          - extract_value: "[0]"

      # operation: put is what makes browser-direct upload possible (engine 1.37.0+).
      - name: Presign
        run_when_succeeded: [Record]
        json_output: |-
          { "file_id": "a|Record::id|", "expires_in": a|var::upload_ttl_secs| }
        post_transforms:
          - s3_generate_presigned_url:
              operation: put
              bucket: a|ap_var::S3_BUCKET|
              file_key: a|Record::object_key|
              expiration_secs: 900
              endpoint: a|ap_var::S3_ENDPOINT|
              region: a|ap_var::S3_REGION|
              access_key_id: a|ap_var::S3_ACCESS_KEY_ID|
              secret_access_key: a|ap_var::S3_SECRET_KEY|
              key: upload_url

  # POST /files/confirm
  # Called once the client's PUT succeeds. Body: { "file_id": "…", "size_bytes": 12345 }
  files/confirm:
    output: http
    method: POST
    summary: Mark an uploaded object as stored
    description: Flips a pending object to stored once the client's direct upload finished.
    tags: [s3, storage, upload]
    request_example: |-
      { "file_id": "019f97…", "size_bytes": 12345 }

    actions:
      - name: ValidateBody
        input: a|body|
        hide_data_on_success: true
        assert:
          http_code_on_error: 400
          error_message: "file_id is required"
          tests:
            - value: file_id
              is_not_null: true
              is_not_empty: true

      - name: MarkStored
        run_when_succeeded: [ValidateBody]
        database: main
        query: |-
          UPDATE s3_objects
             SET status      = 'stored',
                 size_bytes  = a|ValidateBody::size_bytes->default(0)|,
                 uploaded_at = NOW()
           WHERE id = a|ValidateBody::file_id->single_quote|
             AND status = 'pending'
          RETURNING id, filename, size_bytes
        assert:
          http_code_on_error: 404
          error_message: "no pending object with that file_id"
          tests:
            - value: "[0].id"
              is_not_null: true
        post_transforms:
          - extract_value: "[0]"

      - name: Respond
        run_when_succeeded: [MarkStored]
        json_output: |-
          { "file_id": "a|MarkStored::id|", "filename": "a|MarkStored::filename|", "status": "stored" }

  # GET /files/download?file_id=…
  # Mints a short-lived presigned GET. The bucket stays private.
  files/download:
    output: http
    method: GET
    summary: Mint a short-lived presigned download URL
    description: >
      Returns a presigned GET for a stored object. Links expire, so the bucket
      itself is never public.
    tags: [s3, storage, download, presigned]

    actions:
      - name: ReadParams
        input: a|params|
        hide_data_on_success: true

      - name: Lookup
        run_when_succeeded: [ReadParams]
        database: main
        query: |-
          SELECT id, object_key, filename, content_type
            FROM s3_objects
           WHERE id = a|ReadParams::file_id->default('')->single_quote|
             AND status = 'stored'
        assert:
          http_code_on_error: 404
          error_message: "no stored object with that file_id"
          tests:
            - value: "[0].id"
              is_not_null: true
        post_transforms:
          - extract_value: "[0]"

      - name: Presign
        run_when_succeeded: [Lookup]
        json_output: |-
          {
            "file_id": "a|Lookup::id|",
            "filename": "a|Lookup::filename|",
            "expires_in": a|var::download_ttl_secs|
          }
        post_transforms:
          - s3_generate_presigned_url:
              operation: get
              bucket: a|ap_var::S3_BUCKET|
              file_key: a|Lookup::object_key|
              expiration_secs: 300
              endpoint: a|ap_var::S3_ENDPOINT|
              region: a|ap_var::S3_REGION|
              access_key_id: a|ap_var::S3_ACCESS_KEY_ID|
              secret_access_key: a|ap_var::S3_SECRET_KEY|
              key: download_url

  # GET /files?owner=alice&limit=25
  files:
    output: http
    method: GET
    summary: List stored objects
    description: Lists stored objects, newest first, optionally filtered by owner.
    tags: [s3, storage]

    actions:
      - name: ReadParams
        input: a|params|
        hide_data_on_success: true

      - name: ListFiles
        run_when_succeeded: [ReadParams]
        database: main
        query: |-
          SELECT id, filename, content_type, owner, size_bytes, status, uploaded_at, created_at
            FROM s3_objects
           WHERE status = 'stored'
             AND (a|ReadParams::owner->default('')->single_quote| = ''
                  OR owner = a|ReadParams::owner->default('')->single_quote|)
           ORDER BY created_at DESC
           LIMIT a|ReadParams::limit->default(25)|

seed.yml

name: S3ObjectStorageSeed
description: Creates the s3_objects table (idempotent). Safe to re-run — resets to empty.

docs: true

# Required managed variables:
#   DATABASE_URL — Postgres connection string

global:
  databases:
    main:
      driver: postgres
      conn_string: "a|ap_var::DATABASE_URL|"

interfaces:

  # POST /api/seed
  api/seed:
    output: http
    method: POST

    actions:
      - name: CreateTable
        database: main
        hide_data_on_success: true
        query: |
          CREATE TABLE IF NOT EXISTS s3_objects (
            id            TEXT         PRIMARY KEY,
            object_key    TEXT         NOT NULL UNIQUE,
            filename      TEXT         NOT NULL,
            content_type  TEXT         NOT NULL DEFAULT 'application/octet-stream',
            owner         TEXT         NOT NULL DEFAULT '',
            size_bytes    BIGINT       NOT NULL DEFAULT 0,
            status        TEXT         NOT NULL DEFAULT 'pending',
            uploaded_at   TIMESTAMPTZ,
            created_at    TIMESTAMPTZ  NOT NULL DEFAULT NOW()
          );

      - name: CreateIndexes
        database: main
        run_when_succeeded: [CreateTable]
        hide_data_on_success: true
        query: |
          CREATE INDEX IF NOT EXISTS idx_s3_objects_created_at ON s3_objects (created_at DESC);

      - name: Reset
        database: main
        run_when_succeeded: [CreateIndexes]
        hide_data_on_success: true
        query: "TRUNCATE TABLE s3_objects;"

      - name: Respond
        run_when_succeeded: [Reset]
        json_output: |-
          { "status": "seeded" }

schema.sql

-- Object metadata. The bytes live in S3; only the record lives here.
CREATE TABLE IF NOT EXISTS s3_objects (
  id            TEXT         PRIMARY KEY,
  object_key    TEXT         NOT NULL UNIQUE,
  filename      TEXT         NOT NULL,
  content_type  TEXT         NOT NULL DEFAULT 'application/octet-stream',
  owner         TEXT         NOT NULL DEFAULT '',
  size_bytes    BIGINT       NOT NULL DEFAULT 0,
  -- pending: upload URL minted, bytes not confirmed yet. stored: upload confirmed.
  status        TEXT         NOT NULL DEFAULT 'pending',
  uploaded_at   TIMESTAMPTZ,
  created_at    TIMESTAMPTZ  NOT NULL DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS idx_s3_objects_created_at ON s3_objects (created_at DESC);
CREATE INDEX IF NOT EXISTS idx_s3_objects_owner      ON s3_objects (owner);
CREATE INDEX IF NOT EXISTS idx_s3_objects_status     ON s3_objects (status);

S3 Object Storage

Why presigned URLs rather than proxying the upload​

Endpoints​

Setup​

The upload flow​

Downloading​

How it works​

Housekeeping​

Tuning the TTLs​

Configuration​

config.yml​

seed.yml​

schema.sql​