The Ceiling Holds: Testing AI/ML Architecture, Depth, and Training-Window Size Against an Empirically Measured Predictability Limit

A common, mostly unstated assumption behind a lot of modern AI/ML work is that more data

and a more powerful model can eventually predict almost anything. In the physical sciences this

was tested and found wanting: chaos theory showed that some systems, like weather, have a hard

limit on how far ahead they can be predicted no matter how good your model or how much data

you feed it, because the limit lives in the system’s own dynamics, not in the model. This research

program has already measured a limit like that for financial markets, directly and empirically,

with no model involved in the measurement itself. This paper runs two tests of what that limit

actually means for AI/ML in practice, plus a direct follow-up to the first test that closes an obvious

loophole. First, we give five very different kinds of models — a plain average, an intentionally

overfit decision tree, a reinforcement-learning agent, a generative adversarial network, and a

variational autoencoder — the exact same fair amount of training data, sized to stay safely within

that measured limit, and simply watch which one predicts real subsequent prices best. Since the

fancier three of those five were kept deliberately simple — no hidden layers, nothing a machine

learning engineer would call “deep” — we then test the obvious objection directly: we give the

reinforcement-learning agent, the GAN, and the variational autoencoder one genuine hidden

layer each, real nonlinearity rather than just a different training recipe, while keeping the exact

same tiny amount of training data, and check whether that changes anything. Second, we take the

original five models and start feeding them progressively more training data, extending further

and further past that limit, the way a real team chasing “more data” often does, and watch what

happens. In the first test, nothing beats the plain average; two of the fancier models essentially

become the plain average. Giving those same two models a real hidden layer doesn’t change that

either — they still just become the plain average, and the variational autoencoder is still unstable

rather than skillful, whether linear or given a hidden layer. In the second test, every model —

regardless of how it works internally — starts drifting away from reality once its training data

reaches well past the measured limit, and the drift gets worse the further past it they go. This is

the same pattern atmospheric scientists saw decades ago: an early, real-model-based estimate of a

predictability limit, a theoretical account soon after of why it should exist, and then decades of

dramatically better forecasting systems that kept running into the same wall anyway, no matter 

how much they improved in the meantime. We show this mainly as plots of predicted price

against real price, and back the training-data test up with a plain, direct measurement of how

large the errors actually get, as a percentage of price anyone can judge for themselves — not a

statistical test score, not a p-value, just a straightforward number.