BERT-Based Semantic Entropy under Landauer's Principle: Quantifying Energy Cost of Meaning in NLP

This work introduces a novel framework that extends Landauer’s principle—originally describing the energy cost of bit erasure—to the semantic level of natural language understanding. By analyzing attention distributions in transformer models like BERT, we define a new metric called semantic entropy, which captures the cognitive and computational effort involved in language processing.

We propose a normalized energy cost model based on semantic entropy and evaluate it across three large corpora: news headlines, literary texts, and dialogues. Our experiments show a strong correlation between semantic entropy and human-perceived complexity, significantly outperforming traditional baselines such as TF–IDF and random-attention models.

The paper also explores hardware implications, including real energy measurements on GPUs and neuromorphic chips. Applications include energy-aware NLP systems, dynamic pricing of API calls based on processing cost, and potential brain-inspired computing benchmarks. We provide open-source code, detailed methodology, and a visual pipeline for reproducibility.

Author: PSBigBig (Independent Developer and Researcher)
Contact: hello@onestardao.com
Project Page: https://onestardao.com/papers