VSA, Analogy, and Dynamic Similarity
Description
Presentation to be given at the Workshop on Developments in Hyperdimensional Computing and Vector Symbolic Architectures, 16 March 2020, KirchoffInstitute for Physics at Heidelberg University, Heidelberg, Germany.
Extended Abstract
It has been argued that analogy is at the core of cognition [7, 1]. My work in VSA is driven by the goal of building a practical, effective analogical memory/reasoning system. Analogy is commonly construed as structure mapping between a source and target [5], which in turn can be construed as representing the source and target as graphs and finding maximal graph isomorphisms between them. This can also be viewed as a kind of dynamic similarity in that the
initially dissimilar source and target are effectively very similar after mapping.
Similarity (the angle between vectors) is central to the mechanics of VSA/HDC. Introductory papers (e.g. [8]) necessarily devote space to vector similarityand the effect of the primitive operators (sum, product, permutation) on similarity. Most VSA examples rely on static similarity, where the vector representations are fixed over the time scale of the core computation (which is usually a singlepass, feedforward computation). This emphasises encoding methods (e.g.
[12, 13]) that create vector representations with the similarity structure required by the core computation. Random Indexing [13] is an instance of the vector embedding approach to representation [11] that is widely used in NLP and ML. The important point is that the vector embeddings are developed in advance and then used as static representations (with fixed similarity structure) in the
subsequent computation of interest.
Human similarity judgments are known to be contextdependent (see [3] for a brief review). It has also been argued that similarity and analogy are based on the same processes [6] and that cognition is so thoroughly contextdependent that representations are created onthefly in response to task demands [2]. This seems extreme, but doesn’t necessarily imply that the base representations are contextdependent as long as the cognitive process that compares them is
contextdependent, which can be achieved by having dynamic representations that are derived from the static base representations by contextdependent transforms (or any functionally equivalent process).
An obvious candidate for a dynamic transformation function in VSA is substitution by binding, because the substitution can be specified as a vector and dynamically generated (see Representing substitution with a computed mapping in [8]). This implies an internal degree of freedom (a register to hold the substitution vector while it evolves) and a recurrent VSA circuit to provide the dynamics to evolve the substitution vector.
These essential aspects are present in [4], which finds the maximal subgraph isomorphism between two graphs represented as vectors. This is implemented as a recurrent VSA circuit with a register containing a substitution vector that evolves and settles over the course of the computation. The final state of the substitution vector represents the set of substitutions that transforms the static base representation of each graph into the best subgraph isomorphism to the static base representation of the other graph. This is a useful step along the path to an analogical memory system.
Interestingly, the subgraph isomorphism circuit can be interpreted as related to the recently developed Resonator Circuits for factorisation of VSA representations [9], which have internal degrees of freedom for each of the factors to be calculated and a recurrent VSA dynamics that settles on the factorisation. The graph isomorphism circuit can be interpreted as finding a factor (the substitution vector) such that the product of that factor with each of the graphs is the
best possible approximation to the other graph. This links the whole enterprise back to statistical modelling, where there is a long history of approximating matrices/tensors as the product of simpler factors [10].
References
1. Blokpoel, M., Wareham, T., Haselager, P., van Rooij, I.: Deep Analogical Inference as the Origin of Hypotheses. The Journal of Problem Solving 11(1), 1–24 (2018)
2. Chalmers, D.J., French, R.M., Hofstadter, D.R.: Highlevel perception, representation, and analogy: A critique of artificial intelligence methodology. Journal of Experimental & Theoretical Artificial Intelligence 4(3), 185–211 (1992)
3. Cheng, Y.: Contextdependent similarity. In: Proceedings of the Sixth Annual Conference on Uncertainty in Artificial Intelligence (UAI’90), pp. 27–30. Cambridge, MA, USA (1990)
4. Gayler, R.W., Levy, S.D.: A distributed basis for analogical mapping. In: Proceedings of the Second International Conference on Analogy (ANALOGY2009), pp. 165–174. New Bulgarian University, Sofia, Bulgaria (2009)
5. Gentner, D.: Structuremapping: A theoretical framework for analogy. Cognitive Science 7(2), 155–170 (1983)
6. Gentner, D., Markman, A.B.: Structure mapping in analogy and similarity. American Psychologist 52(1), 45–56 (1997)
7. Gust, H., Krumnack, U., Kühnberger, K.U., Schwering, A.: Analogical Reasoning: A core of cognition. KI  Künstliche Intelligenz 1(8), 8–12 (2008)
8. Kanerva, P.: Hyperdimensional computing: An introduction to computing in distributed representation with highdimensional random vectors. Cognitive Computation 1, 139–159 (2009)
9. Kent, S.J., Frady, E.P., Sommer, F.T., Olshausen, B.A.: Resonator Circuits for factoring highdimensional vectors. http://arxiv.org/abs/1906.11684 (2019)
10. Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Review 51(3), 455–500 (2009)
11. Pennington, J., Socher, R., Manning, C.D.: GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. Association for Computational Linguistics, Doha, Qatar (2014)
12. Purdy, S.: Encoding data for HTM systems. http://arxiv.org/abs/1602.05925 (2016)
13. Sahlgren, M.: An introduction to random indexing. In: Proceedings of the Methods and Applications of Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering (TKE 2005), Copenhagen, Denmark (2005)
Notes
Files
VSA_2020_presentation_gayler.pdf
Files
(157.4 kB)
Name  Size  Download all 

md5:6f5bfec46e1e5e360e9086e7c91cee16

157.4 kB  Preview Download 
Additional details
Related works
 Is derived from
 Software: https://github.com/rgayler/VSA_2020_presentation (URL)
 Is supplemented by
 Conference paper: https://github.com/rgayler/VSA_2020_presentation/raw/master/VSA2020_Gayler_abstract.pdf (URL)
References
 Blokpoel, M., Wareham, T., Haselager, P., van Rooij, I.: Deep Analogical Inference as the Origin of Hypotheses. The Journal of Problem Solving 11(1), 1–24 (2018)
 Chalmers, D.J., French, R.M., Hofstadter, D.R.: Highlevel perception, representation, and analogy: A critique of artificial intelligence methodology. Journal of Experimental & Theoretical Artificial Intelligence 4(3), 185–211 (1992)
 Cheng, Y.: Contextdependent similarity. In: Proceedings of the Sixth Annual Conference on Uncertainty in Artificial Intelligence (UAI'90), pp. 27–30. Cambridge, MA, USA (1990)
 Gayler, R.W., Levy, S.D.: A distributed basis for analogical mapping. In: Proceedings of the Second International Conference on Analogy (ANALOGY2009), pp. 165–174. New Bulgarian University, Sofia, Bulgaria (2009)
 Gentner, D.: Structuremapping: A theoretical framework for analogy. Cognitive Science 7(2), 155–170 (1983)
 Gentner, D., Markman, A.B.: Structure mapping in analogy and similarity. American Psychologist 52(1), 45–56 (1997)
 Gust, H., Krumnack, U., Kühnberger, K.U., Schwering, A.: Analogical Reasoning: A core of cognition. KI  Künstliche Intelligenz 1(8), 8–12 (2008)
 Kanerva, P.: Hyperdimensional computing: An introduction to computing in distributed representation with highdimensional random vectors. Cognitive Computation 1, 139–159 (2009)
 Kent, S.J., Frady, E.P., Sommer, F.T., Olshausen, B.A.: Resonator Circuits for factoring highdimensional vectors. http://arxiv.org/abs/1906.11684 (2019)
 Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Review 51(3), 455–500 (2009)
 Pennington, J., Socher, R., Manning, C.D.: GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. Association for Computational Linguistics, Doha, Qatar (2014)
 Purdy, S.: Encoding data for HTM systems. http://arxiv.org/abs/1602.05925 (2016)
 Sahlgren, M.: An introduction to random indexing. In: Proceedings of the Methods and Applications of Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering (TKE 2005), Copenhagen, Denmark (2005)