Domino Orchestrator

Use domino when a project needs a simple serial workflow runner configured by
Hydra-compatible YAML. Prefer it for ordered data jobs, scripts, ETL-like
pipelines, research workflows, or project automation where a shared context
dictionary is enough.

Do not use domino for DAG scheduling, parallel execution, retries, async
execution, persistent state, or distributed orchestration unless those features
are added explicitly.

Design Philosophy

Domino is a lightweight serial workflow orchestrator. Its job is to make workflow
structure explicit, readable, and configurable. A good Domino workflow shows the
ordered operational steps in YAML, while Python provides reusable unit behavior.

Core principle:

Workflow structure belongs in config. Reusable unit behavior belongs in Python.

Prefer Direct Callables

Use the real callable whenever Domino can load it.

Good:

callable: "package.module:function"
callable: "package.client:api.method"

Avoid wrapping a callable only to rename it, forward kwargs, adapt a dotted
method, or preserve an old import path.

A wrapper is justified only when it adds real behavior: validation,
transformation, error handling, persistence, or a reusable policy that cannot be
expressed cleanly through kwargs.

Keep Step Functions Atomic

A Domino step should represent one conceptual operation at the workflow level.

Good examples:

• Fetch source data.
• Build an output path.
• Write raw data.
• Merge records.
• Validate a gap.
• Write a manifest.
• Summarize a result.

Avoid a single Python function that internally performs a whole workflow such as
fetch -> transform -> merge -> validate -> persist.

Use Kwargs For Variation

If two workflow steps differ only by parameters, keep the same callable and pass
different kwargs.

Common examples include date ranges, entity ids, field lists, provider options,
calendar policies, output labels, and storage partitions.

Do not create separate wrapper functions or config files for differences that
are only arguments.

Prefer One Config For One Workflow Shape

If two runnable jobs have the same ordered steps and differ only by runtime
values, they should usually share one config.

Use config overrides for runtime differences. Split configs only when workflow
shape, ownership boundary, or operational intent is genuinely different.

Make Component Workflows Self-Contained

A runnable config should contain the complete ctx and complete ordered workflow
needed to run that component.

Avoid generic entrypoints that depend on many overrides or hidden composition
layers when the project has clear components. Generic entrypoints are acceptable
for small one-off projects, but they should not become an indirection layer over
real workflows.

Keep Policies Reusable, Not Source-Specific

When a workflow needs a reusable policy, model the policy directly.

Good:

callable: "quality:warn_if_gap"
kwargs:
  expected_dates: "calendar_provider:expected_dates"

Avoid separate policy functions for each data source unless the behavior is
genuinely different. Prefer passing policy inputs or provider callables through
kwargs.

Be Explicit With Third-Party APIs

Some third-party callables expose generic signatures such as **kwargs, so
Domino cannot infer parameters from ctx.

In that case, pass required arguments explicitly in YAML. Do not wrap the
third-party API only to create a nicer signature.

Avoid Compatibility Wrappers By Default

Compatibility wrappers preserve old boundaries and make future refactors harder.

Use them only when maintaining a stable public API is more important than
clarity. In internal or early-stage projects, update configs and imports
directly.

Test Architecture, Not Only Behavior

If a project has repeatedly drifted toward hidden workflows or wrapper layers,
add regression tests that scan active configs and source files for forbidden
patterns.

Useful architecture tests may guard against:

• Generic workflow entrypoints returning after removal.
• Local callable loaders duplicating Domino.
• Wrapper callables replacing direct callables.
• Split configs that differ only by kwargs.
• Monolithic sync or run functions hiding workflows.

Decision Checklist

Before adding a Domino step, function, or config, ask:

• Can Domino call the real function directly?
• Is this function hiding multiple workflow steps?
• Is the difference only kwargs?
• Does this config have the same workflow shape as an existing config?
• Is this wrapper adding behavior, or only forwarding?
• Would a future reader understand the workflow from YAML alone?
• Should a regression test prevent this pattern from returning?

If the answers point to direct callable + kwargs + explicit YAML, prefer that
path.

Setup In A Project

Install domino in the target project environment:

python -m pip install git+https://github.com/Junxiao-Zhao/Domino

If using a local checkout of this repository:

python -m pip install -e /path/to/domino

Use this project layout unless the target project already has conventions:

conf/workflow/
  my_workflow.yaml
workflow_steps/
  steps.py

Run workflows from the target project root:

domino --config-path conf/workflow --config-name my_workflow
python -m domino --config-path conf/workflow --config-name my_workflow

Hydra overrides can follow the known flags:

domino --config-path conf/workflow --config-name my_workflow ++ctx.trade_date=20260524

Config Pattern

Create a YAML config with ctx and ordered workflow sections:

work_dir: /tmp/project-work

ctx:
  work_dir: ${work_dir}
  trade_date: 20260524

workflow:
  load_data:
    callable: "workflow_steps/steps.py:load_data"
    kwargs: {}
    return_key: data

  transform:
    callable: "workflow_steps/steps.py:transform"
    kwargs:
      mode: fast
    return_key: transformed

  save:
    callable: "workflow_steps/steps.py:save"
    kwargs: {}
    return_key: null

Workflow steps execute in YAML declaration order.

Each step supports:

• callable: required module:target string; target may be a dotted attribute path.
• kwargs: optional explicit keyword arguments.
• return_key: optional key, list of keys, or null.

The module part can be an importable Python module or a .py file path.
Relative file paths are resolved from the current working directory. The
target part can be a single callable attribute or a dotted attribute path.

Step Function Pattern

Define step functions as plain Python functions. Domino builds call arguments
from explicit step kwargs first, then fills missing signature parameters from
ctx.

Recommended pattern:

def load_data(work_dir, trade_date, **kwargs):
    return {"work_dir": work_dir, "trade_date": trade_date}


def transform(data, mode="default", **kwargs):
    return {"data": data, "mode": mode}


def save(transformed, ctx, **kwargs):
    print(ctx["work_dir"], transformed)

Always consider adding **kwargs to workflow step functions. Domino passes all
explicit step kwargs through unchanged, including keys not declared in the
function signature. A **kwargs fallback prevents unrelated config keys from
raising TypeError when configs are shared or evolve.

Argument resolution:

1. Start with all explicit step kwargs.
2. If a declared parameter is missing and named ctx, pass the whole context.
3. If a declared parameter is missing and exists in ctx, use ctx[name].
4. Otherwise omit the parameter from the call.

Use Python default parameters for optional inputs.

Return Values

Domino writes non-None return values back into ctx.

• return_key: null: store the result under the workflow step name.
• return_key: some_key: store the result under ctx["some_key"].
• return_key: [a, b]: require the result to be a sequence of the same length,
  then store each item under the matching key.
• Return None to store nothing.

Example:

workflow:
  split:
    callable: "workflow_steps/steps.py:split"
    kwargs: {}
    return_key: [left, right]

def split(value, **kwargs):
    return [value[:1], value[1:]]

Verification

Before handing off a target project workflow, run:

domino --config-path conf/workflow --config-name my_workflow

If the target project uses tests, add at least one smoke test that runs the
workflow or calls the step functions with representative ctx values.

Common failures:

• DominoLoadError: the callable module path, module name, target path is
  wrong, or the target is not callable.
• DominoConfigError: workflow config shape or return_key handling is invalid.
• TypeError: a step function received explicit kwargs it does not accept; add
  **kwargs or remove the extra config key.