--- title: Dist keywords: fastai sidebar: home_sidebar summary: "Utilities for distributed training." description: "Utilities for distributed training." nb_path: "nbs/03 - distributed.ipynb" ---
An all_gather layer with backward, useful for collecting model output embeddings from multiple gpus to allow large batch size loss calculation, e.g. for InfoNCE (SimCRL, CLIP).