Hybrid Spatially Regularized NMF and Marker-Guided Benchmarking: A Unified Framework for Interpretable Spatial Transcriptomics in Cancer

"Poster presented at TMC AI Summit 2026"
Background and Problem— High-resolution spatial
transcriptomics (ST) platforms, such as 10x Xenium, have revo-
lutionized the characterization of the Tumor Microenvironment
(TME) by preserving single-cell spatial coordinates. However,
the resulting high-dimensional data pose significant challenges
for representation learning. Standard dimensionality reduction
(DR) techniques, such as PCA and NMF, typically treat cells
as independent samples (“a-spatial” assumption). In complex
solid tumors like cholangiocarcinoma, this leads to fragmented
clustering and biological incoherence, hindering the identification
of subtle cell states. There remains a critical gap in rigorously
benchmarking DR strategies and developing scalable algorithms
that effectively balance geometric compactness with biological
fidelity.
Methods— We present a unified study utilizing a large-
scale 10x Genomics Xenium dataset (480-gene panel) derived
from 40 tumor microarray (TMA) cores across 25 intrahepatic
cholangiocarcinoma patients. Following rigorous quality control,
doublet removal, and log-normalization, 191,125 high-quality,
spatially resolved cells were retained from an initial pool of
212,000. Our approach is two-fold:
1) Rigorous Benchmarking: We systematically evaluated six
DR frameworks—including Linear (PCA, NMF), Non-
linear (VAE, AE), and Hybrid embeddings—via Pareto-
front analysis across varying latent dimensions (k = 5−40).
To quantify performance, we introduced two novel biologi-
cal metrics: Cluster Marker Coherence (CMC) and Marker
Exclusion Rate (MER), alongside a post-hoc reassignment
algorithm to correct biologically implausible labels.
2) Algorithmic Innovation (hSNMF): Leveraging insights
from our benchmark, we developed Hybrid Spatial NMF
(hSNMF). This novel framework extends NMF by in-
corporating a dual “contact-radius” graph (rc = 20μm,
rr = 80μm) to enforce local spatial smoothness on latent
factors prior to clustering, effectively bridging the gap
between transcriptomic similarity and physical proximity.
Results—Our benchmarking revealed fundamental trade-offs
in existing methods: while PCA maximized geometric separation
(Silhouette > 0.15), it suffered from high marker exclusion rates.
Standard NMF excelled at biological signal recovery but lacked
spatial cohesion. The proposed hSNMF framework outperformed
all non-spatial and spatial baselines. By jointly optimizing spatial
contiguity and molecular coherence, hSNMF achieved superior
spatial compactness (CHAOS < 0.002) and high spatial autocor-
relation (Moran’s I > 0.94) while maintaining distinct cluster
separability (Silhouette > 0.22).
Clinical Impact—This work establishes a validated computa-
tional roadmap for dissecting heterogeneity in cholangiocarci-
noma. By replacing “black-box” embeddings with interpretable,
spatially regularized factors, hSNMF enables the precise mapping
of continuous tissue structures and cell-cell interactions. This
scalable framework directly supports the discovery of spatially
distinct tumor niches, facilitating the development of targeted
therapies for complex solid tumors.
Index Terms—Spatial Transcriptomics, Generative AI, Dimen-
sionality Reduction, Cholangiocarcinoma, Xenium.