Causal Representation Learning from Observational Video: Identifiability Assumptions and the Gap Between Synthetic and Real-World Settings

Causal representation learning (CRL) — the recovery of latent causal variables and their causal graph from high-dimensional
observations — has produced a substantial body of theoretical results in the past five years. CITRIS, CauCA, CRID, mechanismsparsity methods, and the recent grouping-based identifiability framework all establish conditions under which causal
structure can in principle be recovered. The gap between this theoretical progress and what has actually been demonstrated
on real-world observational video is wider than most accounts acknowledge. Almost all empirical CRL work uses synthetic 3Drendered video sequences with cleanly designed interventions, ground-truth latent factors, and assumption-friendly
generative processes. Real video provides none of these. This paper argues that the question is not whether CRL is
theoretically possible — it is, under specific assumptions — but which of those assumptions real observational video can
plausibly satisfy and which it cannot. We make three claims. First, the identifiability literature implicitly assumes intervention
access, multi-environment data, or strong sparsity priors that real video provides only partially and noisily. Second, three
properties of real video — object near-independence, occlusions and scene changes as quasi-interventions, and naturally
varying environmental contexts — offer plausible but unverified routes to satisfying identifiability assumptions. Third, objectcentric video models (Slot Attention, SAVi, SlotFormer) provide a necessary inductive bias but do not by themselves recover
causal structure; the bridge between object-centric representation and causal identification is the genuine empirical frontier.
We propose a research agenda focused on this bridge, on benchmarks that span synthetic-to-real, and on the methodological
honesty needed to make cumulative progress.