Getting Started with Scalable

This guide covers installation, baseline host requirements, and the bootstrap flow used to prepare local and HPC environments.

Installation

Install from PyPI using pip.

pip install scalable

For development or local source installs, clone the repository and install it from the checkout.

git clone https://github.com/JGCRI/scalable.git
pip install ./scalable

If installation reports that the scripts directory is not in PATH, add the reported directory to your shell profile.

WARNING: The script scalable_bootstrap.exe is installed in '/path/to/python/scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

For example:

echo "export PATH=\$PATH:/path/to/python/scripts" >> <shell_profile>
source <shell_profile>

This only needs to be done once per environment.

Compatibility Requirements

Required and supported tooling:

  • Local host: Docker

  • HPC scheduler: Slurm

  • HPC container runtime: Apptainer

Bootstrapping is designed for POSIX-like shells. On Windows, Git Bash is recommended.

Work Directory Setup

A dedicated work directory on the HPC host keeps dependencies, runtime assets, and outputs in a consistent layout. The scalable_bootstrap script prepares that directory and builds required worker containers.

Using key-based SSH authentication is strongly recommended because bootstrap may open multiple remote sessions. A setup guide is available on this website.

From a local working directory, run:

cd <local_work_dir>
scalable_bootstrap

Follow the interactive prompts. Bootstrap downloads and builds dependencies on both local and HPC systems, then opens an SSH session into the configured HPC work directory.

Inside the prepared environment, python3 starts an interactive session with Scalable dependencies available. You can also execute scripts directly.

Only files under the configured HPC work directory (and its subdirectories) are available in this execution model.

python3
python3 <filename>.py

If bootstrap is interrupted, rerun scalable_bootstrap. It resumes from the last valid step and skips completed setup where possible.

Next Steps

After setup:

  • Review the API for worker, caching, and function interfaces.

  • Run examples from Demos.

  • Use How-tos for targeted implementation guidance.

Report issues at https://github.com/JGCRI/scalable/issues.