--- title: Dist keywords: fastai sidebar: home_sidebar summary: "Utilities for distributed training." description: "Utilities for distributed training." nb_path: "nbs/03 - distributed.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %}

An all_gather layer with backward, useful for collecting model output embeddings from multiple gpus to allow large batch size loss calculation, e.g. for InfoNCE (SimCRL, CLIP).

{% raw %}

class GatherLayer[source]

GatherLayer(*args, **kwargs) :: Function

Gather tensors from all process, supporting backward propagation. https://github.com/open-mmlab/OpenSelfSup/blob/696d04950e55d504cf33bc83cfadbb4ece10fbae/openselfsup/models/utils/gather_layer.py

{% endraw %} {% raw %}
{% endraw %}