Elastically Managing Workers of Multi-Worker Workloads on Accelerator Devices

This patent covers a method for elastically sharing a single accelerator device across multiple workers of a multi-worker workload, such as a distributed model training job. A device proxy layer intercepts accelerator API calls and decouples a worker’s memory state from the physical device, allowing workers to be time-sliced onto a shared accelerator with low overhead. The method exploits the fact that data-parallel replicas converge to identical parameter and optimizer state at fixed points in training, using content checksums to detect and skip redundant memory transfers between workers sharing a device. This work grew out of the Singularity project, which explored transparent preemption, migration, and elastic scaling for large-scale AI workloads.

Recommended citation: Sivathanu et al.
Download Paper