Topographic deep neural networks predict the functional organization of the primate ventral visual pathway

Recording of presentation at the Neuroscience 2021 annual meeting (held virtually). The abstract follows:

The primate ventral visual pathway is organized into functional maps, including pinwheel-like arrangements of orientation-tuned neurons in primary visual cortex (V1) and patches of category-selective neurons in higher visual cortex. While deep convolutional neural networks (DCNNs) trained for object recognition accurately predict neural representations throughout the ventral pathway, they have no spatial layout for features at a given retinotopic location and are thus unable to predict the rich topographic organization of visual cortex. Here, we close this gap by first assigning each DCNN unit a position in a 2D cortical sheet, then training the network to minimize a cost function with two components: one encouraging accurate object recognition, and another favoring correlated responses among nearby units in each model layer (Figure 1A, 1B). 

We find that training with this composite spatial loss produces brain-like topographic maps in both early and later model layers (Figure 1B). Early layers contain smooth orientation preference maps with pinwheels, clusters of units preferring the same spatial frequency, and color-preference domains resembling V1 “blobs”. In a later layer of the same model, we observe clusters of category-selective units, e.g., face patches, whose spatial organization largely matches that found in primate higher visual cortex. Our model thus leverages local response correlations, which have been linked to theories of wire-length minimization, to accurately predict neuron responses and functional organization throughout the ventral visual pathway. In support of the wire-length minimization hypothesis, we find that our topographic DCNN would require shorter connections than a standard DCNN to support connections between similarly-tuned neurons within early (38% reduction) and later (31% reduction) model layers (Figure 1D). These results suggest that the functional organization of visual cortex can be explained by two constraints: the need to perform object recognition and pressure for local populations of neurons to have correlated responses.