jq Transform

Execute a jq expression to transform and combine input data. This transform uses gojq, a pure Go implementation of jq.

Usage

node:
  config:
    from_file: config.json
  env:
    from_file: env.yaml
  result:
    to_file: result.json
    transform:
      jq: |
        {
          name: $config.name,
          env: $env.environment,
          merged: ($config * $env)
        }

Variables

The jq expression has access to all nodes as variables.

The `$self` Variable

In hybrid nodes (nodes with both from_* and transform), the input data is available via $self:

node:
  config:
    from_file: base.json
    to_file: output.json
    transform:
      jq: $self * {extra: "added"}

For multi-document inputs, access documents via $self.docName:

node:
  config:
    from_file:
      - base.json
      - override.json
    to_file: output.json
    transform:
      jq: $self.base * $self.override

Note: $self is only available in hybrid nodes. Using it in a transform-only node will result in an error.

Single-Document Nodes

For nodes with one document (single file, single URL, inline data), access directly:

# Node with single file
$config

# Access nested fields
$config.database.host

# Access array elements
$users[0].name

Multi-Document Nodes

For nodes with multiple documents (file arrays, globs), access specific documents by name:

node:
  inputs:
    from_file:
      - base.json           # Document "base"
      - override.json       # Document "override"
  result:
    transform:
      jq: |
        {
          base: $inputs.base,
          override: $inputs.override,
          merged: ($inputs.base * $inputs.override)
        }

Glob Documents

Glob documents are named by filename stem:

node:
  configs:
    from_glob: "configs/*.json"   # database.json, cache.json
  result:
    transform:
      jq: |
        {
          db: $configs.database,
          cache: $configs.cache
        }

Transform Node Data

Access intermediate node data:

# Access intermediate node data
$merged

# Access nested fields from intermediate node
$merged.settings.port

Output Format

The jq expression must produce a JSON object. This output is:

Parsed by panconf
Re-encoded to the format specified by the output file extension (JSON, YAML, TOML)
Available to dependent nodes via $nodeName

Examples

Basic Transformation

Extract and rename fields:

node:
  source:
    from_file: full-config.json
  minimal:
    to_file: minimal.json
    transform:
      jq: |
        {
          appName: $source.application.name,
          version: $source.application.version,
          port: $source.server.port
        }

Deep Merge with jq

Use jq's * operator for recursive merge:

node:
  base:
    from_file: base.yaml
  override:
    from_file: override.yaml
  merged:
    to_file: merged.json
    transform:
      jq: $base * $override

Multi-Document Merge

Merge all documents from a multi-document node:

node:
  configs:
    from_file:
      - defaults.yaml
      - env.yaml
      - local.yaml
  merged:
    to_file: merged.json
    transform:
      jq: $configs.defaults * $configs.env * $configs.local

Array Operations

Transform and filter arrays:

node:
  data:
    from_file: users.json
  summary:
    to_file: summary.json
    transform:
      jq: |
        {
          count: ($data | length),
          names: [$data[].name],
          adults: [$data[] | select(.age >= 18)],
          avgAge: (($data | map(.age) | add) / ($data | length))
        }

Conditional Logic

Use jq's if-then-else:

node:
  config:
    from_file: config.yaml
  processed:
    to_file: processed.json
    transform:
      jq: |
        {
          logLevel: (if $config.debug then "debug" else "info" end),
          secure: ($config.environment == "production"),
          settings: (
            if $config.environment == "production"
            then {ssl: true, timeout: 30}
            else {ssl: false, timeout: 60}
            end
          )
        }

Using Intermediate Nodes

Chain jq with other transforms:

node:
  base:
    from_file: base.yaml
  env:
    from_file: production.yaml

  # Intermediate: merge configs
  merged:
    transform:
      deepMerge:
        - "{{.base}}"
        - "{{.env}}"

  # Final: transform merged data with jq
  final:
    to_file: config.json
    transform:
      jq: |
        {
          config: $merged,
          meta: {
            generated: true,
            keys: ($merged | keys)
          }
        }

Complex Transformations

Restructure nested data:

node:
  services:
    from_file: services.yaml
  manifest:
    to_file: manifest.json
    transform:
      jq: |
        {
          version: "1.0",
          services: [
            $services | to_entries[] | {
              name: .key,
              config: .value,
              enabled: (.value.enabled // true)
            }
          ]
        }

Common jq Functions

Function	Example	Description
`*`	`$a * $b`	Recursive merge (objects)
`+`	`$a + $b`	Shallow merge / concatenate
`keys`	`$config \| keys`	Get object keys
`values`	`$config \| values`	Get object values
`length`	`$items \| length`	Array/string length
`map(f)`	`$items \| map(.name)`	Transform array elements
`select(f)`	`$items[] \| select(.active)`	Filter elements
`add`	`$numbers \| add`	Sum array elements
`sort_by(f)`	`$items \| sort_by(.name)`	Sort array
`group_by(f)`	`$items \| group_by(.type)`	Group array elements
`unique`	`$tags \| unique`	Remove duplicates
`flatten`	`$nested \| flatten`	Flatten nested arrays
`to_entries`	`$obj \| to_entries`	Object to key-value pairs
`from_entries`	`$pairs \| from_entries`	Key-value pairs to object
`ascii_upcase`	`$name \| ascii_upcase`	Uppercase string
`ascii_downcase`	`$name \| ascii_downcase`	Lowercase string
`split(s)`	`$csv \| split(",")`	Split string to array
`join(s)`	`$items \| join(",")`	Join array to string

For a complete reference, see the jq manual.

Comparison with Other Transforms

Aspect	jq	gotmpl_inline	deepMerge
Merge objects	`$a * $b`	Manual	Automatic
Filter arrays	`select()`, `map()`	`range` loops	Not supported
Conditionals	`if-then-else`	`if`/`else` blocks	Not supported
String manipulation	Limited	Full (Sprig)	Not supported
Learning curve	jq syntax	Go templates	Simple
Best for	Data transformation	Text generation	Simple merging

Use jq when:

You need complex data transformations
You're working primarily with JSON-like data structures
You want powerful array filtering and mapping
You're already familiar with jq

Use gotmpl_inline when:

You need text generation or string manipulation
You want access to Sprig functions
You prefer Go template syntax

Use deepMerge or treeMerge when:

You just need to combine configs
No transformation is needed

Error Handling

Common errors and their causes:

Error	Cause	Solution
"failed to parse jq expression"	Invalid jq syntax	Check expression syntax
"jq execution error"	Runtime error (e.g., null access)	Use `// default` for optional fields
"jq expression produced no output"	Expression returned nothing	Ensure expression produces a value

Handling Optional Fields

# Provide default value if field is null or missing
{
  name: ($config.name // "default"),
  port: ($config.port // 8080)
}

Null Coalescing

# Check if field exists
{
  hasName: ($config.name != null),
  name: ($config.name // "unknown")
}

jq Transform#

Usage#

Variables#

The $self Variable#

Single-Document Nodes#

Multi-Document Nodes#

Glob Documents#

Transform Node Data#

Output Format#

Examples#

Basic Transformation#

Deep Merge with jq#

Multi-Document Merge#

Array Operations#

Conditional Logic#

Using Intermediate Nodes#

Complex Transformations#

Common jq Functions#

Comparison with Other Transforms#

Error Handling#

Handling Optional Fields#

Null Coalescing#