Decision 01
The engine already exists
Cloud runs on the DAG workflow orchestrator in Documentize.Plugin.Manager, not on a new execution stack.
The decision
Cloud is built on Plugins/Documentize.Plugin.Manager and its plugin tree, rather than on a new execution stack or on the Fn worker model.
The decisive argument is that the pipeline product we are describing needs a dependency graph, per-step file staging, crash resume and admission control — and all four already exist in the DAG orchestrator. Building them again on top of the single-operation worker model would mean reimplementing an engine that sits in the same repository.
What that gives us on day one
Every row is running code, verified in the source tree.
| Mechanism | What it does | Where |
|---|---|---|
| YAML DAG templates | 50+ shipped pipelines. Steps with dependsOn, canRunInParallel, priority, applyToAllParts. | Workflow/Template/*.yaml Workflow/YamlTemplateManager.cs |
| Format routing | A table mapping (from, to) to a pipeline. This is the “pick a format” mechanic, already built. | Conversion/conversion-routing.json Conversion/ConversionResolver.cs |
| Plugin manifests | 49 plugins, each declaring kind, concurrency, timeout, CPU and memory cost, and pool support. | Plugins/*/plugin.json |
| Typed parameters | A declared schema per plugin — the single source for CLI flags, JSON binding and help text. | Plugin.Shared/CliOptionAttribute.cs |
| Resource governor | Pod-level weighted admission across all concurrent invocations. Blocks for backpressure instead of dropping work. | ResourceGovernor.cs |
| Four invocation modes | Hot pooled process, cold process, in-process and util — selected per plugin from its manifest. | Invocation/PluginInvokerFactory.cs |
| Split, fan out, merge | Large documents split into page ranges, steps run per part in parallel, results merged. | Workflow/WorkflowPlanBuilder.cs |
| Crash resume | A checkpoint written after each successful step, so a re-delivery skips completed work. | Invocation/S3CheckpointService.cs |
| Live progress | Plugins report percentage and message; a per-step monitor publishes to Redis for the UI. | Plugin.Shared/StatusReporter.cs Workflow/ProgressMonitoringService.cs |
How a step actually executes
Worth understanding, because two of the three planned changes follow from it.
The orchestrator topologically sorts the plan into batches and runs each batch in parallel. Before a step runs, its input directory is staged: a root step is seeded from the task's input directory, and a dependent step receives the output files of everything it depends on. After the last step, the output is promoted to the canonical task directory and the checkpoints are cleared.
Steps therefore chain through the file system, and one process owns a whole pipeline for its duration. That is simple and fast, and it is also why distributing a pipeline across pods is a real piece of work rather than a configuration change.
Because a pipeline is keyed by task id and staged from that task's own directory, running a recipe over twelve files is simply twelve pipeline instances, each with its own checkpoints and progress. Fan-out over a file selection needs no orchestrator change at all.
One caution about the templates
The template folder contains two YAML shapes. Templates using steps: with pluginName: are real — they deserialize into the workflow model and name plugins that exist in the registry.
Templates using streamingSteps: with an executablePath: pointing at binaries such as SentimentAnalyzer.exe are a design sketch: those executables are not in the repository. The cloud DSL is built on the steps dialect, and the streaming templates should be moved out of the shipped set so nobody plans against them.