Workflow language

FAST-HEP workflows are described declaratively using YAML files.

Rather than writing analysis execution logic directly in Python scripts, users describe:

The FAST-HEP ecosystem then compiles and executes the workflow.

The primary entry point is typically: author.yaml

Goals#

The workflow language aims to make analyses:

The language intentionally separates:

Design philosophy#

The workflow language intentionally favors:

FAST-HEP separates:

This separation allows analyses to evolve while keeping workflow infrastructure relatively stable.

Workflow compilation#

A workflow is not executed directly from the raw YAML.

Instead, FAST-HEP performs several stages:

flowchart LR

    Author["author.yaml"]
    Normalised["normalised workflow"]
    Plan["execution plan"]
    Runtime["runtime execution"]
    Artifacts["artifacts and outputs"]

    Author --> Normalised
    Normalised --> Plan
    Plan --> Runtime
    Runtime --> Artifacts

These stages help provide:

The compilation pipeline is described further in:

High-level structure#

A FAST-HEP workflow is usually written as an author.yaml file.

The exact contents depend on the analysis, but most workflows follow the same broad shape:

version: 1.0

use:
  profiles:
    - registry
    - fasthep_carpenter:registry
    - fasthep_curator:registry
    - fasthep_render:registry
    - fasthep_workshop:registry

data:
  datasets:
    - name: data
      eventtype: data
      files:
        - data/CMS/Zmumu/data.root
    - name: dy
      files:
        - data/CMS/Zmumu/dy.root
  defaults:
    eventtype: mc
    tree_primary: events

sources:
  events:
    kind: root_tree
    tree: events
    stream_type: event_stream

styles:
  dimuon_mass:
    op: hep.render.data_mc
    axes:
      x:
        name: dimu_mass
        label: 'Di-muon Mass, $M_{\mu\mu}$ [GeV/c$^{2}]$'
        limits: [60, 120]
      y:
        name: events
        label: Events
        scale: log
    data_mc:
      data: data
      signals: [dy]
      backgrounds: [qcd, single_top, ttbar, wjets, ww, wz, zz]
      stack: true
      ratio: true

observers:
  - kind: hep.schema_snapshot
    at: [read.events, stage.BasicVars]
    out: schema

analysis:
  stages:
    - id: BasicVars
      op: hep.define
      params:
        variables:
          - name: Muon_Pt
            expr: "sqrt(Muon_Px ** 2 + Muon_Py ** 2)"
          - name: IsoMuon_Idx
            expr: "(Muon_Iso / Muon_Pt) < 0.10"
          - name: NIsoMuon
            reduce:
              op: count_nonzero
              over: IsoMuon_Idx

    - id: DiMuons
      op: hep.di_object_mass
      params:
        collection: Muon
        mask: IsoMuon_Idx

    - id: DiMuonMass
      op: hep.hist
      params:
        dataset_axis: true
        axes:
          - name: dimu_mass
            source: DiMuon_Mass
            type: regular
            bins:
              low: 60
              high: 120
              nbins: 60
        weight_expr: EventWeight
      render:
        style: dimuon_mass
        when: final

This example describes:

  1. which extension profiles are active
  2. which datasets are available
  3. how event data are read
  4. how derived variables are calculated
  5. how physics objects are combined
  6. how histograms are filled
  7. how plots are rendered
  8. which diagnostics are collected

The YAML describes the analysis intent. FAST-HEP then turns this into an executable plan.

Declarative execution#

FAST-HEP workflows describe analysis intent rather than execution mechanics.

For example:

- id: BasicVars
  op: hep.define
  params:
    variables:
      - name: Muon_Pt
        expr: "sqrt(Muon_Px ** 2 + Muon_Py ** 2)"

This says:

It does not say:

From this declaration, FAST-HEP can infer that the operation needs the input fields:

Muon_Px
Muon_Py

and produces the output field: Muon_Pt

That dependency information is used during compilation and planning. The same workflow description can then be executed by different backends without changing the analysis logic.

Profiles and extensions#

The workflow language is extensible through profiles.

Profiles register additional:

Example:

use:
  profiles:
    - fasthep_carpenter:registry
    - fasthep_render:registry

Profiles are typically provided by installable Python packages.

This enables analyses and experiments to extend FAST-HEP without modifying core workflow infrastructure.

Sources and streams#

Data are introduced into workflows through sources.

Examples include:

Sources produce streams of structured event data.

Example:

sources:
  events:
    kind: root_tree
    tree: Events

Sources are generally implemented in:

Operations#

Operations transform data or produce derived artifacts.

Each operation is identified by an operation kind, e.g. op: hep.define

Operations are registered through profiles and can be provided by different packages.

Derived variables#

Example:

- id: BasicVars
  op: hep.define
  params:
    variables:
      - name: Muon_Pt
        expr: "sqrt(Muon_Px ** 2 + Muon_Py ** 2)"

This operation derives a new field from existing event data.

Histogramming#

Example:

- id: DiMuonMass
  op: hep.hist
  params:
    axes:
      - name: dimu_mass
        source: DiMuon_Mass
        type: regular
        bins:
          low: 60
          high: 120
          nbins: 60

This operation fills a histogram from event-level quantities.

Rendering#

Rendering is usually attached declaratively to produced artifacts:

render:
  style: dimuon_mass
  when: final

The rendering backend is resolved through registered render operations and styles.

Writing outputs#

Some operations or sinks may persist outputs to disk.

Example use cases include:

FAST-HEP attempts to keep artifact generation explicit and inspectable.

Observers and diagnostics#

Observers inspect workflow execution and generated data products.

Example:

observers:
  - kind: hep.schema_snapshot
    at: [read.events, stage.BasicVars]
    out: schema

This observer captures schema information during execution.

Observers are commonly used for:

Observers generally do not modify event data directly.

Hooks and lifecycle events#

Hooks allow workflows and extensions to react to lifecycle events during execution.

Unlike normal operations, hooks are typically triggered by execution phases rather than appearing directly as analysis stages.

Examples include:

Hooks are commonly used for:

This separation helps keep analysis logic distinct from runtime orchestration concerns.

Outputs and artifacts#

Workflows may produce:

Generated artifacts are typically written into a build directory.

FAST-HEP attempts to make execution outputs inspectable and reproducible.

Execution backends#

The workflow language is designed to remain largely independent of execution backend details.

Possible backends include:

Backends are responsible for execution orchestration rather than workflow semantics.

Next steps#

Related concept pages:

On this page