Scalable Documentation¶
Scalable is a Python library for orchestrating multi-step workflows on HPC systems with minimal manual overhead. It combines Dask-based task execution, scheduler-aware worker provisioning, and optional containerized runtimes so workloads can run reproducibly across heterogeneous environments.
The diagram below shows the high-level architecture.
Scalable is a strong fit when your project needs one or more of the following:
Long-running or resource-intensive workflows on shared HPC infrastructure.
Pipeline-style execution where outputs from one stage feed downstream stages.
Automatic or programmatic scaling of workers and hardware allocations.
Scalable supports running functions in distinct software environments via container images. A multi-stage Dockerfile can define multiple worker profiles, each with different dependencies, models, or tools, and worker counts can be managed per profile when scaling out a cluster.
Contents¶
Getting Started
How-tos
Common Issues