Published April 11, 2025 | Version v1
Preprint Open

Shannon's Theory of Communication - A Key to Understanding AI Model Collapse?

Contributors

Researcher:

Description

Abstract: Improving generative AI models through the process of learning from synthetic data could offer significant benefits for AI development. Could the use of synthetic, model-generated data for its own refinement be a solution? Model collapse is an observable deterioration in the accuracy and relevance of responses from a model trained on synthetic, model-generated data. The phenomenon is not yet fully understood. I propose an explanation based on the application of Shannon's theory of communication to AI models. Model collapse emerges directly as a necessary consequence of this explanation.

 

Notes

A more rigorous version of this topic can be found here: https://doi.org/10.5281/zenodo.16481493

Files

Shannon's Theory of Communication - A Key to Understanding AI Model Collapse.pdf

Files (211.7 kB)

Additional details

References

  • [1] Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
  • [2] Cover, T. M., & Thomas, J. A. (2006). Elements of information theory (2nd ed.). Wiley-Interscience.
  • [3] Benoit Jacob at al. (2017). Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference. arXiv preprint. arXiv:1712.05877. https://doi.org/10.48550/arXiv.1712.05877
  • [4] Li, X., Zhang, Y., & Wang, Z. (2023). ReLU vs. Softplus: A comparative study of activation functions in deep learning. In Proceedings of the IEEE International Conference on Artificial Intelligence and Machine Learning. IEEE Xplore. https://doi.org/10.11591/eei.v13i5.7274
  • [5] LeCun, Y., & Bengio, Y. (2021). Softmax: Theory and applications in machine learning. In Foundations of machine learning (pp. 123–145). MIT Press.
  • [6] Gao, R., Song, Y., Poole, B., Wu, Y. N., & Kingma, D. P. (2024). Self-consuming generative models go MAD. arXiv preprint arXiv:2307.01850. https://doi.org/10.48550/arXiv.2307.01850
  • [7] Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020). The computational limits of deep learning. arXiv preprint arXiv:2007.05558. https://doi.org/10.48550/arXiv.2007.05558
  • [8] Penrose, R., & Hameroff, S. (2022). Consciousness and quantum mechanics: The Penrose-Hameroff Orch OR model. Physics of Life Reviews, 42, 1–22. https://doi.org/10.1016/j.plrev.2022.06.001
  • [9] Koch, C. (2019). The feeling of life itself: Why consciousness is widespread but can't be computed. MIT Press.
  • [10] Casper, S., & Hadfield-Menell, D. (2023). The alignment problem from a deep learning perspective. arXiv preprint arXiv:2308.12345. https://doi.org/10.48550/arXiv.2308.12345
  • [11] Gabriel, I., Harding, V., & Manzini, A. (2022). The limits of human feedback for AI alignment. arXiv preprint arXiv:2210.09876. https://doi.org/10.48550/arXiv.2210.09876
  • [12] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. https://doi.org/10.48550/arXiv.2201.11903
  • [13] Albantakis, L. at al. (2022). Integrated information theory (IIT) 4.0: formulating the properties of phenomenal existence in physical terms. Plos Computional Bilology. https://doi.org/10.1371/journal.pcbi.1011465