FAST-HEP encourages analysis-specific Python code to live in an installable Python package.
An analysis repository can then bring together:
while giving the analysis-specific code a clear identity as a versioned software component.
flowchart LR
Repository["<b>Analysis repository</b>"]:::input
Workflow["<b>workflow.yaml</b><br/>scientific workflow"]:::input
Package["<b>Python package</b><br/>analysis capabilities"]:::capability
Profile["<b>Analysis profile</b><br/>compose environment"]:::capability
Flow["<b>Flow</b><br/>compile + execute"]:::flow
Repository --> Workflow
Repository --> Package
Package --> Profile
Workflow --> Flow
Profile --> Flow
classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
This is particularly important for reproducibility: analysis-specific code is part of the software environment that produced a scientific result.
A FAST-HEP analysis might use a structure such as:
my-analysis/
├── workflow.yaml
├── pyproject.toml
├── README.md
├── src/
│ └── my_analysis/
│ ├── profiles/
│ │ └── registry.yaml
│ ├── operations/
│ └── __init__.py
└── tests/The exact structure is not prescribed. An analysis only needs the components relevant to it.
The important recommendation is that analysis-specific Python code lives in an installable package rather than relying on scripts or directories added manually to PYTHONPATH.
The package can be installed normally with tools such as pip or pixi, including as an editable installation during development.
Many analyses need functionality beyond the operations provided by the standard FAST-HEP toolkit.
For example, an analysis may provide:
These capabilities can use the same registry and profile mechanisms as the rest of FAST-HEP.
flowchart TD
HEP["<b>HEP profile</b><br/>FAST-HEP capabilities"]:::capability
Experiment["<b>Experiment profile</b><br/>experiment capabilities"]:::capability
Analysis["<b>Analysis profile</b><br/>analysis-specific capabilities"]:::capability
HEP --> Experiment --> Analysis
classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
An analysis can therefore extend an existing environment rather than modifying Flow or the FAST-HEP packages that it depends on.
See Profiles and registries for how these environments are composed.
Python allows code to be made importable simply by manipulating PYTHONPATH. This is common in existing analysis environments, but provides little information about where that code came from or which version was used.
Installing analysis code as a package gives it a clearer identity within the software environment.
This makes it easier to:
For FAST-HEP, this also supports provenance.
Current provenance records already capture software versions associated with a workflow execution. As provenance support develops further, resolved implementations can be associated more directly with the packages that provided them, making it possible to reconstruct not only the requested workflow but the concrete software components used to execute it. Support for collecting this information automatically as part of FAST-HEP provenance is under development.
A reproducible analysis requires more than preserving its workflow.yaml.
The workflow may refer to operations provided by:
flowchart TD
FASTHEP["<b>FAST-HEP packages</b>"]:::capability
Experiment["<b>Experiment packages</b>"]:::capability
Analysis["<b>Analysis package</b>"]:::capability
Environment["<b>Executable scientific environment</b>"]:::runtime
FASTHEP --> Environment
Experiment --> Environment
Analysis --> Environment
classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
Together, these packages form part of the executable scientific environment.
Because FAST-HEP deliberately allows implementations to be replaced, preserving the workflow alone is insufficient for reproducibility: it is also necessary to know which implementations and software versions were actually used.
Treating an analysis as a software package also makes conventional software-engineering practices easier to adopt.
Analysis repositories can include:
None of these require special FAST-HEP mechanisms; standard Python tooling can be used alongside the workflow.
The fasthep-workshop repository contains runnable analyses and examples of FAST-HEP project organisation.
The recommended repository structure will continue to evolve as FAST-HEP tooling and provenance support mature.
The aim is not to impose a rigid directory layout, but to make analysis-specific software installable, identifiable, testable, and reproducible.