FAST-HEP is developed as a collection of focused packages.
Most users do not need to install or interact with these packages individually. The fasthep meta package provides curated installations that bring together the components needed for common use cases.
The package split becomes useful when you want to understand where a capability comes from, use a component independently, or extend FAST-HEP with your own implementations.
The main packages have distinct responsibilities:
flowchart TD
CLI["<b>CLI</b><br/>user interface"]:::input
Flow["<b>Flow</b><br/>compile, plan,<br/>orchestrate"]:::flow
Carpenter["<b>Carpenter</b><br/>HEP data processing"]:::capability
Curator["<b>Curator</b><br/>inspection + evidence"]:::capability
Render["<b>Render</b><br/>visualisation + reports"]:::capability
Distributed["<b>Distributed</b><br/>Dask + HTCondor execution"]:::capability
CLI --> Flow
Carpenter -.->|"registered capabilities"| Flow
Curator -.->|"registered capabilities"| Flow
Render -.->|"registered capabilities"| Flow
Distributed -.->|"backend implementation"| Flow
classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
Flow provides the workflow compilation, planning, and runtime machinery. Carpenter, Curator, Render, Distributed, and external packages contribute capabilities through registries and explicit contracts.
This is not a closed set: experiments, projects, and individual analyses can provide capabilities through the same interfaces.
fasthep-flow#Flow is the workflow compiler and runtime.
It provides the general infrastructure for:
Flow is designed to remain largely domain-independent. HEP-specific analysis functionality is provided by other packages rather than built into the workflow engine.
fasthep-carpenter#Carpenter provides common HEP data-processing and analysis capabilities.
These include:
Carpenter provides these capabilities to Flow through registries and operation contracts, allowing them to evolve independently of the workflow engine.
fasthep-curator#Curator provides machine-readable inspection, validation, diagnostic, and evidence-collection capabilities that enrich Flow’s core provenance records.
These include:
Flow owns the core plan, run, artifact, and writer provenance records. Curator adds evidence that helps explain the data and execution environment without owning scientific transformations or human-readable presentation.
fasthep-render#Render provides visualisation and reporting capabilities.
These include:
Rendering is kept separate from the analysis operations that produce the underlying scientific products.
fasthep-cli#CLI provides the user-facing fasthep command.
For example:
fasthep run workflow.yamlThe CLI provides a common interface to capabilities and workflows provided by the other FAST-HEP packages rather than implementing workflow or analysis behaviour itself.
fasthep-distributed#Distributed implements distributed execution capabilities against Flow’s backend contract.
It currently provides:
dask-jobqueueFlow owns the backend-neutral execution semantics and the local baseline. Distributed owns the infrastructure-specific implementation. Dask-to-HTCondor has been exercised successfully through Flow at CERN lxplus. Slurm provisioning is available at the Dask layer, but it is not yet enabled or site-tested as a Flow execution route.
fasthep#The fasthep meta package is the recommended installation entry point.
For a typical HEP environment:
pip install "fasthep[hep]"It provides curated combinations of FAST-HEP packages so users do not need to manage the toolkit components individually.
The base fasthep package intentionally installs no capability stack. Use
fasthep[basic] for Flow and the CLI or fasthep[hep] for the HEP analysis
stack. Install fasthep-distributed separately when Dask execution is needed.
fasthep-toolbench#fasthep-toolbench contains small shared utilities used across the toolkit, including download, package-discovery, and terminal helpers.
Most users will encounter Toolbench indirectly through other FAST-HEP packages.
fasthep-workshop#fasthep-workshop contains the runnable examples and tutorials used throughout the FAST-HEP documentation and training material.
It also demonstrates how analysis repositories can provide their own workflows, profiles, operations, and supporting code.
Workshop documentation · GitHub
fasthep-dev#fasthep-dev is the integration workspace used when developing multiple FAST-HEP packages together.
It provides a common environment for cross-package development, integration testing, and coordinated ecosystem validation.
Most FAST-HEP users do not need fasthep-dev.
For most HEP users, the answer is FAST-HEP as a whole: install
fasthep[hep] and use the fasthep command-line interface. A smaller
Flow-and-CLI environment is available as fasthep[basic].
Individual packages become relevant when you want to go deeper:
| If you want to… | Start with |
|---|---|
| describe, compile, plan, or execute workflows | fasthep-flow |
| read/write HEP data or add analysis operations | fasthep-carpenter |
| inspect data or add validation and provenance evidence | fasthep-curator |
| create plots and reports | fasthep-render |
| understand or extend the command-line interface | fasthep-cli |
| execute plans with Dask or provision HTCondor workers | fasthep-distributed |
| learn FAST-HEP through runnable examples | fasthep-workshop |
| develop several FAST-HEP packages together | fasthep-dev |
FAST-HEP packages use the same extension mechanisms available to experiment and analysis packages. The toolkit is therefore a collection of useful implementations built around Flow rather than a fixed set of capabilities built into it.
For more about this architecture, see Profiles and registries and Analysis repositories.