Semantic Axis Decomposition of Transformer Embeddings

This work introduces a novel method for interpreting sentence-transformer embeddings via semantic axis decomposition, capsule layering, and recursive reasoning. Top-N dimensions are selected using Random Forests and assigned human-interpretable meanings such as “emotionality”, “scientificness”, and “question intent”.

We go beyond scalar embeddings by stacking multi-layered capsule sub-coordinates and defining recursive semantic floors. Each floor is traversed using two parameters: alpha (influence magnitude) and theta (semantic shift direction), enabling low-cost, interpretable reasoning paths.

A new mechanism—floor pruning—discards semantically weak layers to retain only impactful shifts. Our proposed structure was validated in the Capsule AI Evolution experiment, achieving a +4.45% accuracy gain over baseline without fine-tuning the transformer.

This confirms that semantic floor traversal enables dynamic, modular reasoning and performance improvement through capsule-space navigation alone.

This is a conceptual and visual demonstration. Code and GUI tools are released separately.

Keywords: transformer, embeddings, interpretability, semantic floors, latent space, capsule layers, XAI, sentence-transformers, recursive reasoning

Interactive prototype:
https://github.com/kexi-bq/embedding-explainer

Experiment and evaluation code:
https://github.com/kexi-bq/capsule-ai-evolution

Capsule Selection

In this framework, capsule selection (e.g., dim_5.3) is not fixed in advance but is discovered through an evolutionary optimization process driven by Capsule AI.
The system applies random shifts along selected capsule dimensions and evaluates their impact on downstream task performance (e.g., classification accuracy).
If a particular capsule configuration improves results, it is rewarded and retained; otherwise, it is discarded.

Through multiple iterations, Capsule AI gradually identifies and accumulates effective capsule directions — allowing the embedding space to be optimized without fine-tuning the transformer model itself.

 Post-Publication Experiment: Capsule Reactivation of Neutral Dimensions

We conducted a follow-up experiment targeting neutral embedding dimensions — those that showed <0.001 change in classification accuracy when zeroed out.

Task:
Multi-class classification of 150 text phrases into 4 categories:
emotional, scientific, neutral, question
(see: Semantic_Test_Dataset.csv)

Baseline setup:

• Embeddings: all-MiniLM-L6-v2

• Classifier: LogisticRegression

• Train/test split: 70/30

Procedure:

1. Identified 383 “neutral” dims using accuracy-preserving zeroing

2. Applied Capsule Shift (dim += alpha) individually to each dim in test data

3. Measured accuracy impact per shift

Results:

• 87 out of 383 dims (22.7%) produced accuracy improvements

• Best delta: +2.22% (from 93.33% to 95.56%)

 This confirms that many “dead” dimensions are semantically reactivatable through capsule-based manipulation — even without retraining the model.

For correspondence:  
Aleksey Schetnikov  
Email: alex21259alex@gmail.com  
Telegram: @Alex_larinov