Writing TOML rules

A TOML rule is a declarative spec for one of the eight compression stages. You write the spec, drop it under one of three search paths, and tok0 picks it up. There is no plugin API to learn, no Rust to write, no rebuild required.

The 30-second example

# brew-install.toml
[filter]
name = "brew-install"
commands = ["brew install", "brew upgrade"]
strip_patterns = [
  "^==>.*Downloading",
  "^==>.*Pouring",
  "^🍺.*",
]
head_lines = 5
tail_lines = 3
empty_message = "ok"

[[assertions]]
input = "==> Downloading\n==> Pouring foo\n🍺 foo installed"
expected_contains = "ok"
min_savings_pct = 60

That’s a complete, CI-tested compressor. The assertions block runs in CI on every change.

Where rules live

Rules are loaded from three locations, in priority order:

1. .tok0/filters/*.toml — project-local, requires explicit trust.
2. ~/.config/tok0/filters/*.toml — user-global.
3. Built-in (apps/cli/src/rules/*.toml) — embedded in the binary at build time.

The first match wins. Project-local rules let you compress an in-house tool without forking tok0.

The full schema

[filter]
# Required. Stable identifier — also the suggested filename.
name = "string"

# Required. List of command prefixes this rule applies to.
# Matched against the full command line, longest-prefix-wins.
commands = ["string", ...]

# Stage 1 (always applied automatically). No config needed.
# strip_ansi: built-in.

# Stage 2 — pattern replace.
[[replacements]]
pattern = "regex"
replacement = "string"

# Stage 3 — output match. If present, ONLY blocks matching at least one
# of these patterns survive.
output_match = ["regex", ...]

# Stage 4 — line select.
strip_patterns  = ["regex", ...]   # exclude
keep_patterns   = ["regex", ...]   # include (excludes win on conflict)

# Stage 5 — per-line truncation.
max_line_chars = 240               # default

# Stage 6 — head/tail window.
head_lines = 30
tail_lines = 15
elided_message = "… {n} lines elided …"   # {n} = number of dropped lines

# Stage 7 — hard cap.
max_chars = 8000

# Stage 8 — empty fallback.
empty_message = "ok"

# CI assertions. At least one is required for built-in rules.
[[assertions]]
input             = "raw output to feed the pipeline"
expected_contains = "substring that must appear in compressed output"
expected_excludes = "substring that must NOT appear"
min_savings_pct   = 60                # fails CI if compression < 60%

Authoring workflow

The recipe is the same whether you’re shipping a built-in or a project-local rule:

1. Capture a real fixture. Run the tool and pipe to a file:

   brew install jq 2>&1 > /tmp/brew-install.raw

2. Write the rule. Start with strip_patterns for the obvious noise. Iterate with tok0 rules test ./my-rule.toml /tmp/brew-install.raw.
3. Add assertions. Pin the savings floor so future edits can’t silently regress.
4. Drop it in place. Either commit to apps/cli/src/rules/ (PR welcome), ~/.config/tok0/filters/, or .tok0/filters/.

Inspecting loaded rules

tok0 rules list                # every rule tok0 sees, with its source
tok0 rules show brew-install   # the resolved rule TOML, post-overrides
tok0 rules test ./rule.toml fixture.txt

tok0 rules test is the read-eval-print loop for rule authoring. It re-runs every assertion and prints diffs.

When TOML is not enough

Use a native Rust compressor instead when you need:

• Multi-pass parsing (apply rule A, then re-process with rule B based on what survived).
• Structured input parsing (JSON reshaping, AST-aware filtering, binlog walking).
• Anything stateful — tracking nesting depth, paired begin/end markers.

These cases are rare. Most compressions look like “strip the obvious noise, keep the verdict” — exactly what the eight pipeline stages were designed for. See Architecture for where native compressors live.

Performance

Rules are cached based on file mtime. Loading 120 rules from disk happens once per process (cold start adds <1ms). After that, dispatch to the matching rule is a hash lookup. The pipeline itself is a series of regex applies — typical end-to-end compression cost is < 0.5 ms per command.