DART: Distributed Assignment of Research Tasks for Heterogeneous Compute Environments

Embarrassingly parallel workloads, such as parameter sweeps, Monte Carlo simulations, and hyperparameter searches, are pervasive in research computing, yet running a single experiment across a mix of laptops, lab servers, cloud virtual machines (VMs), and high-performance computing (HPC) clusters still demands ad hoc Secure Shell (SSH) tunnels, port forwarding, and scheduler-specific scripts. We present DART, a job distributor that turns any such mix of machines into a single shared queue. DART pairs a Dockerized job server with a one-line worker command-line interface (CLI); a hosted Hub automates reverse tunneling, worker authentication, and public endpoint provisioning, so collaborators on any network, including those behind network address translation (NAT) or using a personal laptop, can join an experiment with a single API-key and no firewall changes. This makes DART especially effective for researchers who combine Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS)-provisioned HPC resources [1] with institutional clusters and personal workstations, federating all resources under one queue with no per-site configuration. All communication is encrypted end-to-end, and access is controlled through API-key authentication and a personal identification number (PIN)-protected dashboard. Researchers create experiments, define parameter grids, and monitor jobs entirely from a web dashboard; a work-conserving scheduler dispatches jobs on demand, exploits resource heterogeneity, and automatically re-queues failures. In a real-world deployment of 37,720 jobs across 68 heterogeneous machines spanning an institutional HPC cluster, a national ACCESS resource, and personal computers, the observed run completed in approximately eight days. DART’s dynamic allocation achieved a simulation-derived 35× speedup over static partitioning (≈4 days vs. ≈139 days), reducing a multi-month workload to days.