Author → normalise → plan

FAST-HEP workflows are not executed directly from the raw author.yaml file.

Instead, workflows pass through several compilation stages before runtime execution begins.

This pipeline allows FAST-HEP to:

The compilation pipeline is one of the core architectural ideas behind FAST-HEP.

Overview#

flowchart TD

    subgraph Compile["Compilation and planning"]
        Author["author.yaml"]
        Profiles["profiles and registries"]
        Normalised["normalised workflow"]
        Dependency["dependency inference"]
        Plan["execution plan"]

        Author --> Normalised
        Profiles --> Normalised
        Normalised --> Dependency
        Dependency --> Plan
    end

    subgraph Execute["Runtime execution"]
        Runtime["runtime execution"]
        Outputs["artifacts and outputs"]

        Runtime --> Outputs
    end

    Plan --> Runtime

The main boundary is between the execution plan and runtime execution: FAST-HEP can validate, normalise, resolve profiles, and infer dependencies before any data processing starts.

Stage 1: Author workflow#

The author workflow is the user-facing declarative YAML.

Example:

analysis:
  stages:
    - id: BasicVars
      op: hep.define
      params:
        variables:
          - name: Muon_Pt
            expr: "sqrt(Muon_Px ** 2 + Muon_Py ** 2)"

At this stage, the workflow is:

For example, operation implementations are not yet resolved directly. The workflow only references operation kinds such as:

op: hep.define

Stage 2: Profile resolution#

Profiles extend the workflow language.

Example:

use:
  profiles:
    - fasthep_carpenter:registry
    - fasthep_render:registry

During profile resolution, FAST-HEP loads:

This step determines which implementations are available to the workflow.

Stage 3: Normalisation#

The workflow is then converted into a normalised representation.

Normalisation expands implicit structures into a more explicit intermediate form.

Typical normalisation tasks include:

The resulting workflow is easier for later planning stages to reason about.

The normalised representation is generally intended for machines and debugging tools rather than direct authoring.

Stage 4: Dependency inference#

FAST-HEP then analyses the workflow graph.

Operations declare or infer:

Example:

expr: "sqrt(Muon_Px ** 2 + Muon_Py ** 2)"

allows FAST-HEP to infer dependencies on:

Muon_Px
Muon_Py

and determine that the operation produces:

Muon_Pt

This dependency graph helps determine:

Stage 5: Planning#

The planner lowers the workflow into an executable plan.

The plan contains more explicit execution details such as:

At this stage, the workflow has become significantly more runtime-oriented.

Stage 6: Runtime execution#

The runtime executes the generated plan.

The execution backend may vary:

Importantly, the author workflow itself does not usually need to change when switching execution backends.

Artifacts and debugging#

FAST-HEP intentionally exposes intermediate artifacts where practical.

These may include:

This improves:

Why separate authoring from execution?#

Traditional analysis code often mixes:

FAST-HEP attempts to separate these concerns.

The author workflow describes:

what the analysis means

The planner and runtime determine:

how the analysis executes

This separation helps make workflows:

On this page