Transforms

Reshape an action's data after it's fetched with a post_transforms block — an ordered list of transform functions. Extract, rename, filter, aggregate, hash, encrypt, and more, without writing code.

actions:
  - name: Fetch
    http:
      url: https://jsonplaceholder.typicode.com/users
    post_transforms:
      - extract_value: ".body"
      - keep_attributes: [id, name, email]
      - filter:
          conditions:
            $.name:
              contains: a|params::q|

A few of what's available: extract_value / extract_with_jq, rename_keys / remove_keys, keep_attributes, filter, group_by, math, generate_password, encrypt_value / decrypt_value, add_jwt. The full list with fields is in the Transforms reference.

Limits on jq programs

extract_with_jq programs are compiled once and reused, so a transform costs only the work the program does. Each run also has limits, so a program that would never finish, or would recurse without end, fails its own step and leaves every other request running:

Limit	Default	Variable (self-hosted)
Run time	10 seconds	AIRPIPE__JQ_TIMEOUT_MS
Recursion	half of the engine thread's stack, at least 1,500 levels	AIRPIPE__JQ_STACK_SHARE_PERCENT, AIRPIPE__WORKER_STACK_MB
Nesting of the program's text	1,000 levels	AIRPIPE__JQ_MAX_PROGRAM_DEPTH
Nesting of the result	127 levels, the same as any JSON the engine reads	AIRPIPE__JQ_MAX_RESULT_DEPTH
Memory	256 MB, plus 4 times what loading the step's input takes	AIRPIPE__JQ_MEMORY_FLOOR_MB, AIRPIPE__JQ_MEMORY_INPUT_MULTIPLE
Integers	exact up to 10,000 digits, then the nearest float, as in jq	AIRPIPE__JQ_MAX_INTEGER_DIGITS

These sit well above what real programs need. A 30 MB array of records goes through group_by or sort_by in under a second, and a sample of 542 public jq programs nests 52 levels deep at the 90th percentile. The memory allowance grows with the input, so a large payload gets the room its transform needs. A step that reaches a limit fails with a message naming it, for example jq error: the program ran for longer than 10000 ms and was stopped.

A marker inside a string in a program, like select(.owner == "a|jwt::sub|"), reaches jq as a value: whatever the caller sends is compared as text, quotes and backslashes included, and cannot change what the program does. A marker outside a string is inserted into the program as jq, and the engine logs a warning for each one at startup.

On a self-hosted engine, set a variable in the engine's environment (or the --dotenv file it is started with) and restart it. AIRPIPE__WORKER_STACK_MB is read before the engine reads its arguments, so set that one in the environment itself. A value the engine cannot use is logged as a warning and the default applies.