Packages

FAST-HEP is developed as a collection of focused packages.

Most users do not need to install or interact with these packages individually. The fasthep meta package provides curated installations that bring together the components needed for common use cases.

The package split becomes useful when you want to understand where a capability comes from, use a component independently, or extend FAST-HEP with your own implementations.


Toolkit overview#

The main packages have distinct responsibilities:

flowchart TD
    CLI["<b>CLI</b><br/>user interface"]:::input

    Flow["<b>Flow</b><br/>compile, plan,<br/>orchestrate"]:::flow

    Carpenter["<b>Carpenter</b><br/>HEP data processing"]:::capability
    Curator["<b>Curator</b><br/>inspection + evidence"]:::capability
    Render["<b>Render</b><br/>visualisation + reports"]:::capability
    Distributed["<b>Distributed</b><br/>Dask + HTCondor execution"]:::capability

    CLI --> Flow

    Carpenter -.->|"registered capabilities"| Flow
    Curator -.->|"registered capabilities"| Flow
    Render -.->|"registered capabilities"| Flow
    Distributed -.->|"backend implementation"| Flow

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;

Flow provides the workflow compilation, planning, and runtime machinery. Carpenter, Curator, Render, Distributed, and external packages contribute capabilities through registries and explicit contracts.

This is not a closed set: experiments, projects, and individual analyses can provide capabilities through the same interfaces.


Core toolkit#

fasthep-flow#

Flow is the workflow compiler and runtime.

It provides the general infrastructure for:

Flow is designed to remain largely domain-independent. HEP-specific analysis functionality is provided by other packages rather than built into the workflow engine.

Documentation · GitHub


fasthep-carpenter#

Carpenter provides common HEP data-processing and analysis capabilities.

These include:

Carpenter provides these capabilities to Flow through registries and operation contracts, allowing them to evolve independently of the workflow engine.

Documentation · GitHub


fasthep-curator#

Curator provides machine-readable inspection, validation, diagnostic, and evidence-collection capabilities that enrich Flow’s core provenance records.

These include:

Flow owns the core plan, run, artifact, and writer provenance records. Curator adds evidence that helps explain the data and execution environment without owning scientific transformations or human-readable presentation.

Documentation · GitHub


fasthep-render#

Render provides visualisation and reporting capabilities.

These include:

Rendering is kept separate from the analysis operations that produce the underlying scientific products.

Documentation · GitHub


fasthep-cli#

CLI provides the user-facing fasthep command.

For example:

fasthep run workflow.yaml

The CLI provides a common interface to capabilities and workflows provided by the other FAST-HEP packages rather than implementing workflow or analysis behaviour itself.

Documentation · GitHub


fasthep-distributed#

Distributed implements distributed execution capabilities against Flow’s backend contract.

It currently provides:

Flow owns the backend-neutral execution semantics and the local baseline. Distributed owns the infrastructure-specific implementation. Dask-to-HTCondor has been exercised successfully through Flow at CERN lxplus. Slurm provisioning is available at the Dask layer, but it is not yet enabled or site-tested as a Flow execution route.

GitHub


Installation and supporting packages#

fasthep#

The fasthep meta package is the recommended installation entry point.

For a typical HEP environment:

pip install "fasthep[hep]"

It provides curated combinations of FAST-HEP packages so users do not need to manage the toolkit components individually.

The base fasthep package intentionally installs no capability stack. Use fasthep[basic] for Flow and the CLI or fasthep[hep] for the HEP analysis stack. Install fasthep-distributed separately when Dask execution is needed.


fasthep-toolbench#

fasthep-toolbench contains small shared utilities used across the toolkit, including download, package-discovery, and terminal helpers.

Most users will encounter Toolbench indirectly through other FAST-HEP packages.


Examples and development#

fasthep-workshop#

fasthep-workshop contains the runnable examples and tutorials used throughout the FAST-HEP documentation and training material.

It also demonstrates how analysis repositories can provide their own workflows, profiles, operations, and supporting code.

Workshop documentation · GitHub


fasthep-dev#

fasthep-dev is the integration workspace used when developing multiple FAST-HEP packages together.

It provides a common environment for cross-package development, integration testing, and coordinated ecosystem validation.

Most FAST-HEP users do not need fasthep-dev.


Which package should I use?#

For most HEP users, the answer is FAST-HEP as a whole: install fasthep[hep] and use the fasthep command-line interface. A smaller Flow-and-CLI environment is available as fasthep[basic].

Individual packages become relevant when you want to go deeper:

If you want to…Start with
describe, compile, plan, or execute workflowsfasthep-flow
read/write HEP data or add analysis operationsfasthep-carpenter
inspect data or add validation and provenance evidencefasthep-curator
create plots and reportsfasthep-render
understand or extend the command-line interfacefasthep-cli
execute plans with Dask or provision HTCondor workersfasthep-distributed
learn FAST-HEP through runnable examplesfasthep-workshop
develop several FAST-HEP packages togetherfasthep-dev

FAST-HEP packages use the same extension mechanisms available to experiment and analysis packages. The toolkit is therefore a collection of useful implementations built around Flow rather than a fixed set of capabilities built into it.

For more about this architecture, see Profiles and registries and Analysis repositories.

On this page