Developer Guide

This guide covers the internal architecture of panconf and how to contribute.

Architecture Overview

panconf/
├── cmd/panconf/
│   └── main.go              # CLI entry point (Cobra), DAG ordering
├── internal/
│   ├── config/
│   │   └── config.go        # Config file parsing + validation
│   ├── input/
│   │   └── input.go         # Input loading (files, URLs, globs, cmds, exprs)
│   ├── output/
│   │   ├── encoder.go       # Encoder interface
│   │   ├── json.go          # JSON encoder
│   │   ├── yaml.go          # YAML encoder
│   │   └── toml.go          # TOML encoder
│   ├── strategy/
│   │   ├── strategy.go      # Strategy interface
│   │   ├── cmd.go           # External command transform
│   │   ├── deepmerge.go     # Deep merge (permissive)
│   │   ├── treemerge.go     # Tree merge (strict)
│   │   ├── gotemplate.go    # Go template from file
│   │   ├── gotemplate_inline.go  # Inline Go template
│   │   ├── jq.go            # jq expression transform
│   │   └── funcs.go         # Custom template functions (toYaml, toToml)
│   ├── template/
│   │   └── template.go      # Template data types + expansion
│   ├── merge/
│   │   └── tree.go          # YAML tree merge logic
│   └── json5/
│       └── json5.go         # JSON5 parser
├── go.mod
└── go.sum

Package Descriptions

cmd/panconf

The CLI entry point using Cobra.

Commands:

panconf run - Generate configuration files once
panconf watch - Watch for changes and regenerate automatically
panconf init - Create a default panconf.json5 config file
panconf transform deepMerge <files...> - Run deepMerge directly
panconf transform treeMerge <files...> - Run treeMerge directly

Run Command Flow:

Discover config file (panconf.json5 → panconf.json → panconf.toml → panconf.yaml)
Parse and validate configuration
Load all input nodes (nodes with from_* fields) into multi-document NodeData
Build template data from loaded nodes
Build transform node dependency graph and topologically sort
For each transform node (in dependency order):
- Execute transform (merge, template, jq, or cmd)
- Parse result as YAML/JSON into map[string]any
- Store result as single-document NodeData (doc named "doc")
- If to_file specified: encode and write to disk

Watch Command Flow:

Same initial steps as run command
Set up fsnotify watcher for local files and glob matches
URL inputs are cached (downloaded once, not re-fetched)
On file change events:
- Debounce rapid changes (100ms delay)
- Reload changed inputs (preserve URL cache)
- Regenerate all outputs

internal/config

Configuration file parsing and validation.

Key Types:

type Config struct {
    Node map[string]NodeConfig
}

type NodeConfig struct {
    // Input sources (exactly one for input nodes)
    FromFile FileInput  // from_file: single or array of files
    FromURL  URLInput   // from_url: single or array of URLs
    FromGlob GlobInput  // from_glob: glob pattern(s)
    FromCmd  CmdInput   // from_cmd: single or array of commands
    FromExpr ExprInput  // from_expr: inline data

    // Output (optional)
    ToFile ToFileInput  // to_file: output path(s)

    // Transform (required for transform nodes)
    Transform *TransformConfig
}

Input Types (with custom YAML/JSON unmarshalers):

// FileInput supports shorthand and array forms
type FileInput struct {
    Items []NamedPath  // [{Name: "", Path: "file.json"}, ...]
}

// NamedPath represents a path with optional explicit name
type NamedPath struct {
    Name string  // Explicit document name (empty = derive from path)
    Path string  // File path or URL
}

// NamedCmd represents a command with optional explicit name
type NamedCmd struct {
    Name string   // Explicit document name
    Cmd  []string // Command arguments
}

// NamedExpr represents inline data with a name
type NamedExpr struct {
    Name string         // Document name
    Data map[string]any // Inline data
}

// ToFileEntry represents a document-to-file mapping
type ToFileEntry struct {
    DocRef string  // Document reference (e.g., "{{.node.doc}}")
    Path   string  // Output file path
}

Input Syntax Examples:

# Single file (doc named "doc")
from_file: config.json

# Multiple files (docs named by filename stem)
from_file:
  - base.json      # doc named "base"
  - override.json  # doc named "override"

# Explicit naming
from_file:
  - defaults: path/to/defaults.json
  - custom: path/to/custom.json

Node Classification:

Input node: has exactly one from_* field, no transform
Transform node: has a transform, optional to_file

Validation Rules:

Node must be either input or transform (not both, not neither)
Input: exactly one of from_file, from_url, from_glob, from_cmd, from_expr
Glob: must include file extension
Transform: exactly one of cmd, deepMerge, treeMerge, gotmpl, gotmpl_inline, jq
Output file extension: must be .json, .yaml, .yml, or .toml
No cycles in node dependencies

TOML Parsing Note: The BurntSushi/toml library doesn't support custom unmarshal methods, so TOML configs are first parsed to a generic map[string]any, marshaled to JSON, then unmarshaled using our custom JSON unmarshalers.

internal/input

Input source loading functions.

Functions:

// LoadFiles loads multiple files into a map of documents
func LoadFiles(items []config.NamedPath, cwd string) (map[string]template.DocData, error)

// LoadURLs loads multiple URLs into a map of documents
func LoadURLs(items []config.NamedPath) (map[string]template.DocData, error)

// LoadGlobs loads files matching glob patterns into a map of documents
func LoadGlobs(patterns []string, cwd string) (map[string]template.DocData, error)

// LoadCmds loads command outputs into a map of documents
func LoadCmds(items []config.NamedCmd, cwd string) (map[string]template.DocData, error)

// LoadExprs loads inline expressions into a map of documents
func LoadExprs(items []config.NamedExpr) (map[string]template.DocData, error)

Document Naming:

Single item without explicit name: document named "doc"
Multiple items without explicit names: documents named by filename stem (without extension)
Explicit name: {name: path} → document named by the key

Decode Helpers:

DecodeFile(path) - Decodes based on file extension
DecodeBytes(data, ext) - Decodes based on extension
Supported formats: .json, .yaml, .yml, .toml, .gotmpl, .tmpl

internal/output

Output encoding.

Interface:

type Encoder interface {
    Encode(data map[string]any, w io.Writer) error
}

Implementations:

JSONEncoder - Pretty-printed JSON
YAMLEncoder - YAML with 2-space indent
TOMLEncoder - TOML format

Factory:

GetEncoder(path string) (Encoder, error)  // Based on file extension

internal/strategy

Merge and transform strategies.

Interface:

type Strategy interface {
    Execute(args []StrategyArg) (map[string]any, error)
}

type StrategyArg struct {
    Path string         // File path (if from .path reference)
    Data map[string]any // In-memory data (if from direct reference)
}

Implementations:

DeepMerge:

Recursive merge of maps
Arrays replace (no concat)
Scalars replace
Type conflicts allowed (later wins)

TreeMerge:

Uses internal/merge YAML tree logic
Recursive merge of maps
Arrays replace
Scalars replace
Type conflicts error

Cmd:

type Cmd struct {
    Args []string
}
func (c *Cmd) Execute() ([]byte, error)  // Returns stdout (must be valid YAML/JSON)

GoTemplate:

type GoTemplate struct {
    TemplateFile string
    DataFiles    []string
    TemplateData template.TemplateData
}
func (g *GoTemplate) Execute() ([]byte, error)  // Returns rendered output

GoTemplateInline:

type GoTemplateInline struct {
    Template     string
    TemplateData template.TemplateData
}
func (g *GoTemplateInline) Execute() ([]byte, error)  // Returns rendered output

JQ:

type JQ struct {
    Expr         string
    TemplateData template.TemplateData
}
func (j *JQ) Execute() ([]byte, error)  // Returns JSON output

The jq transform passes all nodes as variables:

Single-doc nodes: $nodeName is the document data directly
Multi-doc nodes: $nodeName is an object with doc names as keys

Custom Template Functions (in funcs.go):

toYaml - Convert value to YAML string
toToml - Convert value to TOML string
toJson - Convert value to JSON string
nindent - Add newline and indent

internal/template

Template data types and expansion.

Types:

// DocData holds data for a single document
type DocData struct {
    Data map[string]any  // Parsed document data
    Path string          // File path (empty for inline/transform)
}

// NodeData holds all documents for a node
type NodeData struct {
    Docs      map[string]DocData  // docName → DocData
    Paths     map[string]string   // docName → path (convenience)
    IsSingle  bool                // true if single document (shorthand enabled)
    SingleKey string              // the single doc's key (usually "doc")
}

// TemplateData maps node names to their data
type TemplateData map[string]NodeData

Helper Functions:

// NewNodeData creates a NodeData from a map of documents
func NewNodeData(docs map[string]DocData) NodeData

// toTemplateValue converts NodeData for use in templates
// - Single-doc: returns the document data directly (shorthand)
// - Multi-doc: returns map of doc names to data
func (n NodeData) toTemplateValue() any

Template Access Patterns:

For single-document nodes:

# Shorthand access (recommended)
"{{.config}}"              # Document data
"{{.config.fieldName}}"    # Nested field
"{{.meta.config.path}}"    # File path (via .meta)

For multi-document nodes:

# Access specific document
"{{.inputs.base}}"      # Document named "base"
"{{.inputs.override}}"  # Document named "override"

# In deepMerge, {{.nodeName}} expands to all docs in sorted order
deepMerge:
  - "{{.inputs}}"       # Expands to all documents

Expansion Functions:

// Expand expands a template string with data, env vars, and tilde
func Expand(s string, data TemplateData) (string, error)

// ExpandSlice expands each string in a slice
func ExpandSlice(ss []string, data TemplateData) ([]string, error)

internal/merge

YAML tree merge logic (for treeMerge transform).

Types:

type YAMLTree struct {
    root      *yaml.Node
    sourceMap map[string]string  // Key path → source file
}

Functions:

NewYAMLTree() - Creates empty tree
Merge(keyPath, content, sourcePath) - Merges content at path
Bytes() - Serializes to YAML
Node() - Returns root node

Merge Rules:

Base	Overlay	Result
Map	Map	Merge keys
Array	Array	Replace
Scalar	Scalar	Replace
Empty Map	Any	Convert
Mismatch	Mismatch	Error

Data Flow

Node Loading

panconf.yaml          config.Load()           input.Load*()
     │                     │                       │
     ▼                     ▼                       ▼
┌─────────────┐      ┌─────────────┐        ┌──────────────┐
│ from_file:  │─────▶│ FileInput   │───────▶│ map[string]  │
│   - a.json  │      │ {Items: []} │        │   DocData    │
│   - b.json  │      └─────────────┘        │ "a" → data   │
└─────────────┘                             │ "b" → data   │
                                            └──────────────┘
                                                   │
                                                   ▼
                                            ┌──────────────┐
                                            │   NodeData   │
                                            │ Docs: {...}  │
                                            │ IsSingle: F  │
                                            └──────────────┘

Transform Execution

transform.deepMerge      resolveDataRefArgs()      strategy.Execute()
        │                         │                         │
        ▼                         ▼                         ▼
┌────────────────┐       ┌────────────────┐       ┌────────────────┐
│ ["{{.base}}",  │──────▶│ []StrategyArg  │──────▶│ map[string]any │
│   "{{.env}}"]  │       │ [{Data: ...},  │       │    (merged)    │
└────────────────┘       │  {Data: ...}]  │       └────────────────┘
                         └────────────────┘               │
                                                          ▼
                                                   ┌──────────────┐
                                                   │   NodeData   │
                                                   │ Docs: {"doc":│
                                                   │   merged}    │
                                                   │ IsSingle: T  │
                                                   └──────────────┘

Multi-Document Expansion in deepMerge

When {{.nodeName}} references a multi-document node in deepMerge:

// resolveDataRefArgs() in main.go
if node.IsSingle {
    // Single doc: return one StrategyArg
    return []StrategyArg{{Data: node.Docs[node.SingleKey].Data}}, true
}
// Multi-doc: return all docs in sorted order
keys := sortedKeys(node.Docs)
for _, k := range keys {
    result = append(result, StrategyArg{Data: node.Docs[k].Data})
}
return result, true

Output Encoding

map[string]any          GetEncoder()              file/stdout
      │                      │                         │
      ▼                      ▼                         ▼
┌──────────┐           ┌──────────┐            ┌──────────┐
│ merged   │──────────▶│ JSON/YAML│───────────▶│ written  │
│ data     │           │ /TOML    │            │ output   │
└──────────┘           └──────────┘            └──────────┘

Testing

Run Tests

# All tests
go test ./...

# Verbose
go test -v ./...

# Specific package
go test -v ./cmd/panconf/...

# With coverage
go test -cover ./...

# Specific test
go test -v -run TestMultiDoc ./cmd/panconf/...

Test Structure

Integration tests are in cmd/panconf/main_test.go:

func TestFeature(t *testing.T) {
    dir := t.TempDir()
    
    // Create test files
    writeFile(t, dir, "input.json", `{"key": "value"}`)
    writeFile(t, dir, "panconf.yaml", `
node:
  data:
    from_file: input.json
  out:
    to_file: out.json
    transform:
      deepMerge:
        - "{{.data}}"
`)
    
    // Run panconf
    if err := runPanconf(t, dir); err != nil {
        t.Fatalf("panconf failed: %v", err)
    }
    
    // Assert results
    result := readJSON(t, filepath.Join(dir, "out.json"))
    if result["key"] != "value" {
        t.Errorf("expected key=value, got %v", result["key"])
    }
}

Test Categories

Category	Tests
Config Discovery	JSON, TOML, YAML, JSON5, priority
Input Sources	from_file, from_glob, from_url, from_cmd, from_expr
Multi-Document	file arrays, explicit names, glob access, jq access
Output Formats	JSON, YAML, TOML, multiple outputs
Transforms	deepMerge, treeMerge, gotmpl, gotmpl_inline, jq, cmd
Templates	env vars, tilde
Validation	all error cases
Cross-Format	YAML↔JSON↔TOML
CLI Transform	deepMerge, treeMerge subcommands
DAG	intermediate nodes, cycle detection
Edge Cases	empty input, arrays, nesting, dir creation

Dependencies

github.com/spf13/cobra       # CLI framework
github.com/fsnotify/fsnotify # File system notifications (watch mode)
github.com/BurntSushi/toml   # TOML parsing/encoding
github.com/Masterminds/sprig # Template functions
github.com/itchyny/gojq      # jq implementation
gopkg.in/yaml.v3             # YAML parsing/encoding
encoding/json                # JSON (stdlib)

Contributing

Code Style

Follow standard Go conventions
Use gofmt for formatting
Keep functions focused and small
Add comments for non-obvious logic

Adding a New Input Type

Add new input type struct in internal/config/config.go
Implement custom UnmarshalYAML and UnmarshalJSON methods
Add field to NodeConfig struct
Add Load* function in internal/input/input.go
Handle in cmd/panconf/main.go loadNodes() function
Add validation rules in NodeConfig.Validate()
Write tests

Adding a New Output Format

Create internal/output/newformat.go
Implement Encoder interface
Add case in GetEncoder() switch
Add extension validation in config
Write tests

Adding a New Transform

Create internal/strategy/newtransform.go
Implement Strategy interface (or custom Execute method)
Add field to TransformConfig struct
Handle in cmd/panconf/main.go processTransformNode() switch
Update TransformConfig.TransformType() and Args() methods
Add CLI subcommand if appropriate
Write tests and documentation

Pull Request Checklist

Tests pass (go test ./...)
Code formatted (gofmt)
Documentation updated if needed
No new linter warnings

Developer Guide#

Architecture Overview#

Package Descriptions#

cmd/panconf#

internal/config#

internal/input#

internal/output#

internal/strategy#

internal/template#

internal/merge#

Data Flow#

Node Loading#

Transform Execution#

Multi-Document Expansion in deepMerge#

Output Encoding#

Testing#

Run Tests#

Test Structure#

Test Categories#

Dependencies#

Contributing#

Code Style#

Adding a New Input Type#

Adding a New Output Format#

Adding a New Transform#

Pull Request Checklist#