Crossing the Great Filter: Cultural Reflow and Inter-episodic State Inheritance in Zero- Reward Reinforcement Learning: A New Problem Class for Multi-generational Coordination Beyond Reward-Driven Episodic RL

Standard reinforcement learning (RL) operates under two rarely questioned assumptions: (1) episodic reset: the environment fully resets after each episode; (2) reward-driven learning: behavior must be reinforced by reward signals. These assumptions create a barrier for tasks requiring multi-generational sacrifice, where a single agent's lifetime is insufficient to complete the goal, and necessary actions yield zero reward and terminate the episode.

In this paper, we formalize Inter-episodic State Inheritance MDP (ISI-MDP), which splits environment state into agent-specific transient state and global persistent state inherited across episodes. We propose Event-Triggered Cultural Reflow (ETCR), which preserves and transmits zero-reward but goal-critical trajectories across generations via behavior cloning.

We evaluate our framework in the Sealed Corridor environment, a grid world where the goal requires multiple generations of sacrificial, zero-reward actions. Experimental results across 30 random seeds show that standard reward-driven optimization actively hinders cross-generational coordination: PPO falls into a zero-gradient trap, failing in 60% of runs and requiring over 2300 episodes on average. While pure off-policy epsilon-greedy methods (DQN) and evolution (No Reflow) can occasionally stumble through the filter, they are highly inefficient (>170 and >1000 episodes, respectively). In contrast, our ETCR framework triggers a rapid phase transition, accelerating breakthrough efficiency to a mean of 70.2 episodes with a 100% survival rate (p < 1e-10 vs. all baselines). These results suggest that reward-driven episodic RL faces inherent difficulties with multi-generational great filters, and inter-episodic state inheritance paired with cultural reflow offers a viable path for this class of tasks.