Approximation Is Not Learning

Universal approximation versus the promise that scale will reach the function

The universal approximation theorem proves a network capable of approximating any continuous function exists; it says nothing about whether training will find that network, whether finite data can teach it, or whether it would fit in a building, and the hype quietly swaps the first sentence for the other three.

Universal approximation is an existence result about representation capacity, not a claim about learnability. Cybenko (1989) and Hornik (1991) prove that for any continuous target function on a bounded domain, some network of sufficient width approximates it to any chosen accuracy. The proofs are non-constructive: they exhibit no procedure for finding that network, place no bound on its size, and assume the target function is already handed to you. A working system must instead find weights by training, from finite and noisy data, at a size that fits on real hardware, and must do well on inputs it never saw. None of those four obligations is the theorem's, and the gap between "a network can represent it" and "this training run learned it" is exactly where the pitch lives. State the mathematics fairly and it deflates the slide by itself.

Audiences:

• The founder sold capability as capacity — A vendor cites universal approximation to argue that because a network can represent any function, a large enough model will solve their problem if they buy enough compute. They are paying for a budget against a wall that is mathematical, not financial.
• The engineer who shipped the model — They trained a model that nailed the demo and degraded in production, and they half-suspect the failure means the network lacked the capacity to represent the task, so the fix must be a bigger network.
• The operator deciding whether to build — A small-business owner is told the technology can in principle do their task, and reads "in principle" as "soon, and for me," then budgets a project that the theorem never promised would converge or fit.

Note: written from Indonesian operator context. Frameworks apply broadly to other emerging-market and SME settings.