Semantic Context Alignment with Open-World Learning for Memory-Driven Human Activity Recognition

Agentic AI systems that combine perception with memory must repeatedly solve a context alignment problem:
deciding whether a current observation matches something already held in long-term memory, or whether it
is new. The mechanism used to make that decision, symbolic (lexical) or sub-symbolic (embedding-based),
determines whether the system’s memory generalises across vocabulary variation or fragments into redundant,
near-duplicate entries. This study investigates that choice inside an agentic memory architecture, using
Human Activity Recognition as the applied testbed rather than the object of study. Building on the
memory-driven architecture proposed by Shahabian Alashti et al. [1], a Jaccard-and-Bayesian symbolic
context alignment engine is replaced with a semantic engine that computes cosine similarity over Sentence
Transformer embeddings, while every other component of the pipeline, including the Bayesian posterior
and the 60% confidence-gating threshold, is held fixed. Both configurations are evaluated in a closed loop
on seen (Robot House Multi-View, N = 420) and unseen (Toyota Smarthome, N = 930) data, at roughly
three times the scale of the pilot study this project began as. Long-term memory improves accuracy over a
memory-free baseline on both datasets, confirming its functional role in the architecture. On seen data, the
semantic mechanism outperforms the symbolic baseline narrowly (61.60% versus 60.90%). On unseen data
the gap widens substantially (49.80% versus 37.40%), while the supervised classifier alone collapses from
94.00% to 38.30% once evaluated outside its training distribution. These results indicate that the advantage
of semantic over symbolic context alignment is concentrated in generalisation under distribution shift rather
than in-distribution accuracy, a finding with implications for the design of context alignment mechanisms in
memory-driven AI systems beyond activity recognition specifically.