Sequence
What has to change, and in what order
Three orchestrator changes stand between the engine we have and the product we are describing.
Three engine changes, one that's already free
Read directly out of DagOrchestrator and PluginInvokerPool. Ordered so each unblocks the next.
C1 — Per-step parameters
small · blocks the DSLEvery step in a pipeline currently receives the same task payload, with only the id rewritten. There is no way today to say "convert to PDF in step one, then save as PDF/A in step two" — the with: block on the recipe DSL cannot exist until this changes.
Fix: carry per-step options through the plan builder and merge them into each step's payload before it runs.
C2 — Lazy plugin warm-up
medium · cost and startupEvery pod currently warms all 49 plugins at boot, including LibreOffice and OCR, whether or not they are ever called in that pod's lifetime. That sets memory to the sum of the whole catalogue and slows startup enough to matter for autoscaling.
Fix: create an invoker on first use and evict it when idle — the same pattern the Fn control plane already uses for its worker pool.
C3 — Distributed step leasing
large · deliberately lastOne pod currently owns an entire pipeline for its whole duration — a large archival run is pinned to one machine, and a pod restart mid-run relies on checkpoint replay to catch up rather than another pod simply picking up the next ready step.
Fix: move pipeline state out of the pod and have pods lease ready steps from it instead of executing a batch locally. Only this change lets a single pipeline scale horizontally or idle to zero between runs. It is deliberately last because everything else in this roadmap demos without it.
Fan-out over a file selection needs no orchestrator change. A step is staged from its own task's input directory, so a run over twelve files is simply twelve pipeline instances, each with its own id, checkpoints and progress stream. This falls entirely on the new Run object — see below — not on the engine.
The objects that don't exist yet
Everything the product needs beyond the engine reduces to three missing records.
| Object | Why it's missing | What it unlocks |
|---|---|---|
| Workspace | Entitlement today is one signed key per instance — there is no notion of an account that owns anything. | Files, recipes and runs get an owner. |
| Library | Storage keys are scoped to a single task; a file exists only as an input to one operation and has no life afterward. | "Select from files you already uploaded" becomes possible. |
| Run | A task is one operation on one input set; there is no persisted object meaning "this recipe, over these files, currently 7 of 12 done." | The object the whole UI binds to — the single most important addition. |
None of these require new infrastructure to prototype. They fit next to the existing SQLite-backed queue on the serverless surface, and the interfaces should be written so that swapping in a real database for the hosted surface is a provider change, not a rewrite.
Sequence
Each milestone ends in something that can be put in front of someone.
Catalogue is live data
Publish the plugin schema and cost model as JSON; expose the format-routing table; add a test asserting every template's plugin references resolve.
demo: every plugin, cost and parameter as JSONPer-step parameters (C1)
Options flow through the plan builder and merge per step. Proved with a pipeline that converts, then re-saves at a different quality.
demo: one pipeline, two steps, different settingsWorkspace and library
Re-key storage to a workspace-owned layout, bind the signed key to a workspace, build file CRUD, and promote run outputs back into the library.
demo: upload once, reuse across operationsRuns and the live view
Persist runs, fan out over selected files, bridge the existing Redis progress stream to the browser.
demo: twelve files through an eight-step pipeline, liveRun builder and recipes
Format picker, surface selector, file picker, generated options form. "Save as recipe" serialises the run; the editor validates against the M1 schema.
demo: build by clicking, open as YAML, edit, re-runLazy warm-up, then step leasing (C2, C3)
Pods stop warming the whole catalogue; pipeline state moves to the store and pods lease ready steps.
demo: a pipeline that scales horizontally and idles to zeroM1 and M2 are the pair worth starting on immediately: M1 is read-only and unblocks this documentation along with the run builder and the DSL validator, and M2 removes the only structural blocker standing between the engine we have and the recipe grammar on the previous page. M3 through M5 are the product surface. M6 is scheduling work that deserves a design pass of its own once the rest is proven.