Learn

smokepod

Smoke test runner for CLI applications. You write .test files describing commands and expected output, then smokepod executes them in Docker containers or against local targets and reports pass/fail.

Overview

  • Language: Go
  • Repo: peteretelej/smokepod
  • Install: npx smokepod --help or go install github.com/peteretelej/smokepod/cmd/smokepod@latest
  • Status: active

Architecture

smokepod has a clean three-layer structure: CLI commands, a public library package, and internal parsing.

Package layout:

  • cmd/smokepod/main.go - CLI entry point using urfave/cli, defines four commands: run, validate, record, verify
  • pkg/smokepod/ - public Go library (importable by other projects)
    • smokepod.go - top-level API: Run(), RunWithOptions(), RunFile(), ValidateConfig()
    • config.go - YAML config parsing (Config, Settings, TestDefinition), validation, defaults
    • executor.go - test orchestrator with parallel/sequential execution, fail-fast, functional options pattern
    • target.go - Target interface (Exec + Close)
    • target_local.go - LocalTarget runs commands via local shell
    • target_docker.go - DockerTarget runs commands inside a testcontainers-go container
    • target_process.go - ProcessTarget communicates via JSONL stdin/stdout with a long-running process
    • docker.go - container lifecycle management using testcontainers-go
    • compare.go - output comparison with regex and stderr support
    • diff.go - unified diff generation for failure reporting
    • fixture.go - JSON fixture read/write for record/verify workflow
    • discovery.go - finds .test files by walking directories
    • reporter.go - JSON result reporting for run mode
    • verify_reporter.go - human-readable verify output with pass/fail/xfail/xpass
    • result.go - result types (Result, TestResult, SectionResult, CommandResult)
    • platform.go - platform detection for recorded fixtures
    • launch.go - test launching utilities
    • runners/ - test type runners
      • cli.go - CLI test runner: parses .test sections, executes commands, compares output
      • playwright.go + playwright_types.go - Playwright test runner for browser tests in Docker
      • types.go - shared runner types
  • internal/testfile/parser.go - .test file parser: sections, commands, expected output, regex/stderr suffixes, exit codes, metadata directives, xfail
  • internal/whitespace/ - whitespace normalization for output comparison
  • npm/ - npm package wrapper with platform-specific native binaries

Three execution modes:

  1. run - reads a smokepod.yaml config, spins up Docker containers via testcontainers-go, executes .test file commands inside them, compares output against inline expectations
  2. record - runs commands against a reference target (e.g. /bin/bash), captures stdout/stderr/exit code, saves to JSON fixture files
  3. verify - runs commands against a different target, compares output against previously recorded fixtures, reports diffs

Data flow for run mode:

  1. Parse smokepod.yaml into Config
  2. Executor.Execute() iterates test definitions (parallel or sequential)
  3. For each CLI test: parse the .test file, create a Target (Docker or local), instantiate a CLIRunner
  4. Runner iterates sections and commands, calls target.Exec(), compares output using compare.go
  5. Aggregate results into Result struct, report as JSON

The .test file format:

Sections start with ## name, commands with $ command, and expected output as plain text below. Suffixes modify matching: (re) for regex, (stderr) for stderr matching, (stderr,re) for both. Exit codes are asserted with [exit:N]. Sections can be marked (xfail) for expected failures. Metadata directives (# target:, # mode:) appear before the first section.

Key Design Decisions

Custom .test format instead of YAML/JSON tests. The format is designed to look like a terminal session, making tests readable without understanding any framework. Section headers, dollar-sign commands, and plain-text expectations are intuitive. The parser handles edge cases (multi-line commands, regex suffixes, exit codes) without making the common case complex.

Docker via testcontainers-go. Rather than requiring a pre-running service, smokepod manages container lifecycle itself. Tests declare an image in the YAML config, and testcontainers-go handles pulling, starting, executing, and tearing down. This means tests are fully self-contained and reproducible.

Dual distribution: Go binary + npm wrapper. The npm package (npm/) includes platform-specific native binaries so JavaScript projects can use npx smokepod without a Go toolchain. The npm/bin/run.js launcher detects the platform and executes the correct binary. This lowers the adoption barrier for non-Go projects.

Record/verify separation from run. The three modes serve different use cases. run is for “does my app behave as documented in the .test file?” while record/verify is for “does my replacement tool produce identical output to the original?” Keeping them separate avoids conflating inline expectations with fixture-based comparison.

Target interface abstraction. The Target interface (Exec(ctx, cmd) -> ExecResult) cleanly separates command execution from test logic. LocalTarget, DockerTarget, and ProcessTarget all implement it, so the same test runner works against local binaries, Docker containers, or long-running processes that speak JSONL.

xfail sections for known failures. Sections marked ## name (xfail) are expected to fail. If they pass unexpectedly (xpass), that is treated as a failure. This supports incremental compatibility work where you track known gaps without breaking CI.

Development

go build -o smokepod ./cmd/smokepod
go test ./...

# Run smoke tests against a Docker image
./smokepod run smokepod.yaml

# Record fixtures from a reference target
./smokepod record --target /bin/bash --tests tests/ --fixtures fixtures/

# Verify a new target against recorded fixtures
./smokepod verify --target ./my-tool --tests tests/ --fixtures fixtures/