The Structured Latent Basis: Feature Engineering as Basis Selection
Description
We introduce the Structured Latent Basis (SLB) framework, a perspective that compares feature engineering, meta-learning, and representation learning through a common diagnostic question: does the chosen representation make a regularized linear predictor sufficient under a declared evaluation protocol? We make two formal observations and define an empirical program: 1. Conditional optimality: For a fixed feature matrix and \alpha>0, the regularized linear objective is strictly convex and Ridge gives its unique minimizer. 2. Approximation: An analytic cosine-decay component plus finitely many jumps admits a quantitative cosine-plus-sigmoid approximation bound. 3. Diagnostic protocol: Compare a regularized linear probe with specified nonlinear baselines after all learned preprocessing is fit inside the training fold. The current text implementation establishes the fold-local Ridge path and valid-token feature extraction; nonlinear baselines remain part of the prospective protocol, and the required embedding caches must be regenerated before numerical claims are reinstated. The unifying insight is narrow: feature construction and model fitting should be evaluated jointly. The basis can be hand-designed, inherited from pre-training, or learned across tasks. A companion paper studies learned feature maps with closed-form Ridge adaptation and explicitly relates that mechanism to R2-D2. The present paper does not claim that a suitable basis always exists at practical dimension or that Ridge is globally optimal among model classes. Keywords: feature engineering, basis selection, meta-learning, spectral methods, Ridge regression, few-shot learning, representation learning
Maturity: Working Paper. Target venue: Zenodo preprint; journal venue to be determined. Part of The Latent research program.
Related papers in this program: Universal.
Notes
Files
CHANGELOG.md
Additional details
References
- Almarwani, N., Aldarmaki, H., and Diab, M. (2019). Efficient Sentence Embedding using Discrete Cosine Transform. Proceedings of EMNLP-IJCNLP 2019, 3672–3678. DOI: 10.18653/v1/D19-1380.
- Bertinetto, L., Henriques, J. F., Torr, P. H. S., and Vedaldi, A. (2019). Meta-learning with differentiable closed-form solvers. International Conference on Learning Representations. https://openreview.net/forum?id=HyxnZh0ct7.
- Chen, T., and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of KDD 2016, 785–794. DOI: 10.1145/2939672.2939785.
- Gangeh, M. J., Farahat, A. K., Ghodsi, A., and Kamel, M. S. (2015). Supervised Dictionary Learning and Sparse Representation—A Review. arXiv:1502.05928. https://arxiv.org/abs/1502.05928.
- Goldblum, M., Reich, S., Fowl, L., Ni, R., Cherepanova, V., and Goldstein, T. (2020). Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks. Proceedings of ICML 2020, 3607–3616. https://proceedings.mlr.press/v119/goldblum20a.html.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. DOI: 10.1007/978-0-387-84858-7.
- Lee-Thorp, J., Ainslie, J., Eckstein, I., and Ontañón, S. (2022). FNet: Mixing Tokens with Fourier Transforms. Proceedings of NAACL-HLT 2022, 4296–4313. DOI: 10.18653/v1/2022.naacl-main.319.
- Mairal, J., Bach, F., Ponce, J., Sapiro, G., and Zisserman, A. (2008). Supervised Dictionary Learning. Advances in Neural Information Processing Systems 21. https://arxiv.org/abs/0809.3083.
- Müller-Eberstein, M., van der Goot, R., and Plank, B. (2022). Spectral Probing. Proceedings of EMNLP 2022, 7730–7741. DOI: 10.18653/v1/2022.emnlp-main.527.
- Nagy, T. (2026). The Smooth-Step Spectral Method: Unifying Smooth and Threshold Structure in Tabular Regression. Working paper.
- Rahimi, A., and Recht, B. (2007). Random Features for Large-Scale Kernel Machines. Advances in Neural Information Processing Systems 20, 1177–1184.
- Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP-IJCNLP 2019, 3982–3992. DOI: 10.18653/v1/D19-1410.
- Shuman, D. I., Narang, S. K., Frossard, P., Ortega, A., and Vandergheynst, P. (2013). The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains. IEEE Signal Processing Magazine, 30(3), 83–98. DOI: 10.1109/MSP.2012.2235192.
- Trefethen, L. N. (2013). Approximation Theory and Approximation Practice. SIAM. DOI: 10.1137/1.9781611975949.
- Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30.
Subjects
- Machine learning
- http://id.loc.gov/authorities/subjects/sh85079324