Published June 27, 2023 | Version v1
Conference paper Open

Learning distinct features helps, provably

  • 1. Faculty of Information Technology and Communication Sciences Tampere University, Finland
  • 2. Faculty of Information Technology, University of Jyväskylä, Finland & Programme for Environmental Information, Finnish Environment Institute, Jyväskylä, Finland
  • 3. Department of Electrical and Computer Engineering Aarhus University, Denmark


We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average L2-distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network’s units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.


This work has been supported by the NSF-Business Fin- land Center for Visual and Decision Informatics (CVDI) project AMALIA. The work of Jenni Raitoharju was funded by the Academy of Finland (project 324475). Alexandros Iosifidis acknowledges funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 957337.



Files (619.8 kB)

Name Size Download all
619.8 kB Preview Download

Additional details


Advanced machine learning methods for biomonitoring 324475
Academy of Finland
MARVEL – Multimodal Extreme Scale Data Analytics for Smart Cities Environments 957337
European Commission