Recipes

This document defines how recipe QIP components are discovered from disk.

Root #

Given MIME type/subtype, recipe directory is:

WARC Recipes #

application/warc recipes run at the whole-site layer instead of one page at a time. You can use them for site-wide transforms, such as adding trailing-slash redirects, verifying there are no broken links, or using the path to modify body content.

Debugging broken links

warc-check-broken-links.wasm traps when an internal HTML link does not resolve. To inspect the failures, run the same archive through warc-extract-broken-links.wasm:

qip router warc ./site --view-source \
  | qip run components/application/warc/warc-extract-broken-links.wasm

The result is another application/warc archive. It keeps only response pages containing broken links and reduces each HTML body to the exact opening tags with broken href, src, action, data, or srcset values. An archive with no broken links contains only a warcinfo record; WARC 1.1 does not define a zero-record archive.

Rendering referenced content sizes

recipes/application/warc/25-add-content-size.wasm fills <qip-content-size> elements from the response body stored at an absolute site path:

<qip-content-size src="/example.wasm"></qip-content-size>

Bodies below 1,000 bytes render as bytes. Larger bodies render as decimal kilobytes with two fractional digits. The recipe updates the enclosing HTTP and WARC content lengths. During single-route development, the router adds direct static src dependencies to the subset WARC. An unresolved size path is an error instead of producing a plausible size.

The rewrite preserves the input record's WARC-Date, WARC-Record-ID, WARC-Target-URI, content type, and extension fields, then emits the record as WARC 1.1. Because the HTML and HTTP block changed, it removes stale WARC-Block-Digest and WARC-Payload-Digest fields. It also removes HTTP ETag, Content-MD5, and Digest validators rather than claiming they still describe the rewritten body. Records with no content-size replacement keep their metadata and payload.

Writing WARC transforms

An application/warc -> application/warc recipe receives standards-valid WARC 1.1 and must return standards-valid WARC 1.1. In particular:

The router validates the final archive after the full recipe chain. This keeps the trust boundary at export: malformed output traps or fails the command instead of being written to disk.

Turning URI lists into redirects

components/application/warc/warc-text-uri-list-to-redirect.wasm rewrites each text/uri-list HTTP response in a WARC into 302 Found. The first non-empty, non-comment line becomes the Location header; a UTF-8 BOM on the first line and surrounding whitespace are ignored. A URI list without a target traps.

The standard recipe list runs this component before the other WARC transforms. The router itself does not parse URI lists: it builds an ordinary WARC response and runs the configured recipe chain. Other HTTP hosts can use the same component without reproducing redirect behavior in host code.

Loading custom elements selectively

components/application/warc/warc-add-custom-element-scripts.wasm connects element routes to the pages that use them. It discovers top-level /elements/<tag-name>.js responses in the archive, detects matching custom-element tags in each HTML response, and inserts one external module script per used element:

<script type="module" src="/elements/qip-edit.js"></script>

The recipe ignores tag-shaped text in comments, script, style, textarea, and title content. Existing scripts are not inserted again. Nested routes such as /elements/lib/shared.js remain available to imports but are not treated as entrypoints.

Run this recipe late in the WARC chain so it sees elements introduced by earlier transforms. This repository links it as 99-add-custom-element-scripts.wasm. During single-route development, the router includes transformed top-level element modules in the subset WARC so discovery has the same inputs as a whole-site export.

Execution Context #

WARC recipes can run in two useful scopes. Pick the scope based on the question you are answering.

Use subset scope while developing recipe logic. Use whole-site scope before publishing so final archive semantics are still exercised.

# Fast single-path iteration:
qip router get ./site /docs/router

# Final whole-site run:
qip router warc ./site

Content Recipe CSV #

Content Recipe CSV is the canonical machine-readable description of an ordered Content component pipeline. It uses the ordinary text/csv media type. Each row records the component path and the effective contract on both sides of that step:

path,input_encoding,input_mime,input_capacity_bytes,output_encoding,output_mime,output_capacity_bytes
/text/markdown/commonmark.0.31.2.wasm,utf8,text/markdown,2097152,utf8,text/html,2097152
/text/html/html-to-accessibility-tree.wasm,utf8,text/html,262144,utf8,text/markdown,1048576

Rows execute in file order. The first data row is step 1; a separate step column would duplicate the record order and create another value that could disagree with it. A recipe contains at least one data row.

The canonical serialization is UTF-8 without a byte-order mark, uses LF line endings, and ends with one LF. Fields cannot contain CR or LF. A field is quoted only when it contains a comma or double quote, and a double quote in a quoted field is written twice. The header and column order are exact.

The columns have these meanings:

Adjacent rows follow the normal Content composition rules: output and input MIME types match exactly, equal encodings connect, and UTF-8 output may widen to a bytes input. A bytes output cannot narrow implicitly to UTF-8.

Capacity metadata does not make a recipe invalid when one output maximum is larger than the next input maximum. The actual intermediate value may fit. Tools may report that comparison as a warning or enforce it with semantics equivalent to --capacities-must-fit. At execution time, the loaded Wasm module's capacity exports remain authoritative if an artifact at a recorded path has changed.

The component finder catalog uses the same header and row schema. It is a curated graph of format converters, not an inventory of every Content component. Generic transforms, same-type transforms, and infrastructure such as Content Recipe CSV source generators stay out of the finder by remaining absent from that catalog.

Planning And Dry Runs #

qip dry run resolves and validates the same ordered component pipeline as qip run, but does not read input, call render, or write output:

qip dry run \
  components/text/markdown/commonmark.0.31.2.wasm \
  components/text/html/html-page-wrap.wasm

The report is intended to be useful in CI logs without another formatting step:

Pipeline compatible: 2 step(s)
1. components/text/markdown/commonmark.0.31.2.wasm — Content
   Input:  encoding=UTF-8, type=text/markdown, capacity=2.0 MiB (2097152 bytes)
   Output: encoding=UTF-8, type=text/html, capacity=2.0 MiB (2097152 bytes)
   Buffers: 4.0 MiB (4194304 bytes)
2. components/text/html/html-page-wrap.wasm — Content
   Input:  encoding=UTF-8, type=text/html, capacity=256.0 KiB (262144 bytes)
   Output: encoding=UTF-8, type=text/html, capacity=512.0 KiB (524288 bytes)
   Buffers: 768.0 KiB (786432 bytes)
   Note: step 2 (components/text/html/html-page-wrap.wasm): previous output capacity 2.0 MiB (2097152 bytes) exceeds this input capacity 256.0 KiB (262144 bytes); qip run remains valid when the actual intermediate output fits
Total declared buffer capacity: 4.8 MiB (4980736 bytes)
Warnings: 1

A compatible plan exits successfully. Invalid component contracts, uniforms, encoding or MIME composition, and module-policy violations return a non-zero exit status.

Runtime Failures #

When a recipe step fails, the host reports its one-based position and component path before the component error:

step 2 (components/bytes/zlib-decompress.wasm): rejected input

The same format applies to qip run pipelines and router recipe chains. This lets a log identify the failing component even when a recipe uses the same component more than once.

A component can reject expected input without trapping. Its render result sets the failure bit. The CLI does not read output from a rejected result. If the component supplies an input offset, the message is more specific:

step 3 (components/text/utf8-must-be-valid.wasm): rejected input at input offset 17

The input offset is diagnostic data. Recipe logic must not treat it as a stable error code. Every rejected result stops the recipe, and no output from that step is used.

A trap is different. The CLI reports render trapped and discards the Wasm instance because its memory may contain partial output or state. Capacity, content-type, and uniform failures use the same step N (component) prefix.

Use --capacities-must-fit to turn capacity warnings into errors:

qip dry run --capacities-must-fit \
  components/text/markdown/commonmark.0.31.2.wasm \
  components/text/html/html-page-wrap.wasm

The check requires each Content component's declared maximum output capacity to fit the next Content component's input capacity. This is useful in CI and when refining component contracts: without the flag, the pipeline remains valid when its actual intermediate values fit. Tile capacities are per-tile working buffers rather than whole-image Content capacities, so the Tile contract validates those separately.

The host first extracts a plain description for each recipe step: component kind, input and output encoding, optional MIME types, declared buffer capacities, and Tile halo or Interactive frame dimensions. A pure planner then validates those values and returns the ordered plan used by both commands. The dry-run output prints every step and the sum of its declared input/output buffer capacities. An in-place Tile buffer appears as both input and output but is counted once.

Composition is directional and based only on the ordered step descriptions. The planner does not inspect example input bytes or use browser/runtime heuristics, so the same component artifacts and uniforms produce the same plan or the same error.

The host validates arbitrary bytes once when they enter the UTF-8 domain. It then carries the UTF-8 guarantee through output_utf8_cap and input_utf8_cap stages without rescanning every intermediate value. Encoding a native string as UTF-8 also establishes the guarantee. If a bytes-producing stage breaks the chain, a later UTF-8 stage requires explicit host validation or a bytes-to-UTF-8 validator.

The planner applies these rules:

There is no generic bytes-to-pixels rule. Image tiling is available only through that explicit bridge, which keeps text, opaque binary data, and pixel buffers from being guessed into one another.

The encoding relationship is small:

Content encodings

raw bytes
└── valid UTF-8

Allowed widening:  UTF-8 ──> raw bytes
Rejected narrowing: raw bytes -X-> UTF-8

Explicit Tile bridge (not subtyping)

image/bmp raw bytes
        │ host decodes
        v
RGBA32Float tiles (width × height × 4 channels, in-place)
        │ host encodes
        v
image/bmp raw bytes

RGBA32Float pixels are physically held in linear memory, but they are not an opaque Content bytes value. Their dimensions, channels, coordinates, tile size, and halo are part of the Tile contract. Only the host's explicit image bridge may cross that boundary.

For example, this plan decodes SVG Content to BMP, applies an in-place Tile filter, then passes BMP Content to the ICO encoder:

qip dry run \
  components/image/svg+xml/svg-rasterize-to-bmp-b8g8r8a8-srgb.wasm \
  components/rgba/brightness.wasm -u brightness=0.1 \
  components/image/bmp/bmp-to-ico.wasm

The middle step reports RGBA32Float tile for its input and output encoding; the adjacent Content steps report image/bmp raw bytes at the bridge.

Capacity maxima do not make two steps incompatible by themselves: an upstream component may declare a larger output buffer while producing an actual value that fits the next input buffer. Dry run reports this as a warning because only execution can determine the intermediate byte count.

Host And URLs #

qip router warc controls canonical route host via --host <host>. We prefer setting this explicitly for production builds so recipe logic that reads target URLs sees stable, deploy-intended origins.

Example:

qip router warc ./site --host https://qip.dev

Adding Routes #

WARC recipes can synthesize or rewrite archive records, which means they can add output routes (for example /sitemap.xml) when they emit additional WARC records.

The search target table relates sections without storing a document tree:

target,url,label
2-0,/docs/abc,ABC documentation
2-3,/docs/abc#portable,ABC documentation — Portable

The part before - identifies the page and the part after it identifies a section. Search can therefore combine terms found in different sections of the same page. It links to a fragment when one section matches the whole query, or to the page when the matches are spread across sections.

Posting shards keep the indexed term first and sort by that field:

term,target,weight
component,2-3,4
portable,2-3,12

The mandatory header means every posting begins after a newline. A browser can find a prefix with "\n" + prefix, scan the contiguous matching rows, and avoid parsing unrelated rows. The build step folds page-title, heading, and body importance into the integer weight; the browser only adds weights after grouping targets by page.

Ordering #

Filename format:

Disabled filename format:

Examples:

Tie-Breaking #

Validation #

Host should reject recipe entries if:

Host should ignore non-.wasm files in the recipes tree.

Scope #