Vectoring Languages

Imagine that you are playing a word-guessing game with your friend. You chose the past tense of “read” (the one that sounds similar to “red”) as your answer for her to guess.

“Is the answer close to ‘said’?”, she asked. You said yes, since the two sound alike.

“Is the answer close to ‘real’?”, she asked. You said yes since the two have similar spellings.

“Is the answer close to ‘look’?”, she asked. You said yes since the two have similar meanings.

“Is the answer close to ‘book’?”, she asked. You said yes since the two often share appearances.

“Is the answer ‘read’?”, she asked. You said yes.

We use language every day so naturally that we sometimes do not notice how versatile we can recognize words. We can easily specify a word by comparing its similarity with other words, even if the comparisons we made are based on different reasons.

The outline of this article will go as follows. First, I will start by introducing vectoring and its similar concepts. Then I will take a look at related works and the concerns they propose. In the third section, I will construct our vectoring view of language by starting with some taxonomy and definitions, then proceed to show how a vectoring view can be a good approach to understanding language. Lastly, I will conclude with some inspirations we gain from vectoring, and present the path to a few potential future works.

Philosophers have been long discussing about the meaning and usage of specific words and how they resemble the talker’s internal stance. For instance, how the truth value is affected by the word “but” in a sentence, or when can we say “so-and-so means so-and-so”. I want to argue that the natural way humans use language is by considering multiple attributes of language embedded in its high-dimension vector property. Either consciously or subconsciously, we tend to consider how fluent and natural our sentence is when speaking or writing, how the words sound when composing lyrics, and for university students, how long each word or sentence is when writing semester reports. Recently, using vectors as word representation has empirically achieved great results in the form of large language models (LLMs). This outcome hints that the idea backing up what AI scientists have been working on may be much closer to the essence of language than what we expected. For the sake of simplicity, I will call the idea of treating words as high-dimensional vectors, and a language as a high-dimensional vector space, “vectoring”.

To make our metaphor with the vector space more concrete and to strengthen the connection between our philosophical research and the scientific basis of AI science, we will be loosely adapting some definitions from linear algebra. I believe the amount of linear algebra introduced in this article will not be enough to overwhelm readers without the corresponding background, but enough for discovering new aspects of language that were previously unnoticed.

Using vectors as a representation of words has a long history, dating back to research in computer science in the 1980s to the 1990s[19][20]. However, language in philosophy still remains anthropocentric, and benefits little from the breakthroughs of computer science research about language models. In this section, we will propose our approach of a vectoring perspective on languages, and we will show how this approach directly connects to recent AI science breakthroughs in later sections.

There are many aspects of language like the meaning of a word, the sound that happens when we utter a word, the aesthetic of poetry, and so on. Not only that we cannot consider all aspects of language at once, but we, as mortals, also cannot recognize all aspects of language. We are like blindfolded people trying to figure out what an elephant looks like just by touching it. Since this symbol is a high dimensional space with an unknown shape that changes dramatically depending on the perspective we look at it, we shall conveniently call it the vector space of a language, or $V_{L}$ for short.¹¹1 We say that a language is represented as a vector space $V_{L}$. I believe it is not that relevant to formally define the vector space in a mathematics fashion, since it is still difficult for us to understand what elements in this vector space stand for. I suggest that this vector space is over a scalar field $F$ representing the times of occurrences and that the addition operation stands for the idea when combining vectors without considering the sequential pattern, but won’t formally cover it in this article. This part will be left as future work for others.

An important concept of vectoring is the projection. Since we are not able to observe the $V_{L}$ space as a whole, we project $V_{L}$ into a subspace with a potentially lower dimension in order to understand it better. We can think of casting a shadow on the wall in order to better understand the shape of an object. An example of a projection of $V_{L}$ is to focus on the meaning of words. In linear algebra, a projection is a linear transformation $P$ in which applying the projection maps vectors in $V_{L}$ into a subspace $W$. Additionally, applying the projection twice leaves its image unchanged compared to applying it once. These harmonies with our analogy of the projection: trying to find the meaning of the meaning subspace itself essentially brings us the same subspace. ²²2In linear algebra, a projection $P$ is defined as a linear operator on $V$ such that $P:V\rightarrow V$ and that $P^{2}=P$. This means that the projection on $V_{L}$ *can* maintain its rank so that the subspace that projection $P$ maps to is the original space, nothing less.

Words, being the basic element in nearly all other research programs about similar topics, are just a set (since we are using a plural form of word) of vectors, and aren’t much different from utterances or phrases, which are also some set of vectors in $V_{L}$. However, a word is a good example of what a single vector may represent, and we will talk more about it later in (link to later section). For now, we will take a little detour to how vectoring is currently used in AI science, which is what we will call practical vectoring in this article.

The development of language models in computer science has come a long way, and by the underlying method behind the contemporary models, we can roughly divide the technology history into two stages. We will, however, later show that the two methods are essentially the same.

It is a nice timing to explain here a popular misunderstanding: the difference between vectoring and practical vectoring. While using similar wording and wanting to achieve similar goals, there are still large differences that shouldn’t be ignored when discussing vectoring.

Representation dimensions In vectoring, a word is represented as a high dimensional vector, which captures all degrees of a word in the language vector space. In practical vectoring, however, humans defined the dimension. The standard Word2Vec pre-trained models have 300 dimensions[27], and the optimal dimension count is experienced to be dependent on the quality and quantity of the training data.

Empirical data Currently, the majority of the training data utilized is in the form of text data, especially those written on the internet. The ease of data access leads to a distribution of language usage that differs from real-world usages, which may lead to inaccurate modeling of the language space $V_{L}$. For example, current large language models generally achieve higher accuracy in written language than in spoken language.

Structural restrictions Another difference between practical vectoring and vectoring is the structural restrictions that inevitably exist when implementing computer language models. It doesn’t matter whether you choose a Skip Gram structure for implementing Word2Vec or a Transformer structure for implementing large language models, the model structures all have fixed dimensions, and use floating point numbers to represent the vectors. What’s more important, is that the hidden layers of either structure hardly mean anything to a human, while in vectoring, we can explicitly define our projections for any kind, such as “the meaning of a word used in France in 1750”.

We will show how the differences make us free from the problems that are illustrated in Brenden M. Lake, and Gregory L. Murphy, 2023[4].

1. 1.

   The vector space of a language, or $V_{L}$ in this article, is a comprehensive model of a language $L$ in our interest. Due to its high dimensional nature, humans are not able to observe the vector space. Instead, we are projecting the vector space into a subspace with a lower dimension, and argue reasonably within the subspace.

2. 2.

   A Projection is a linear transformation $P$ where $P:V_{L}\rightarrow W$, in which $W$ denotes a subspace of $V_{L}$ that has a dimension not larger than $V_{L}$. ³³3In linear algebra, two projections on the same space with the same rank grant two isomorphic subspaces.

3. 3.

   When we say we are interested in an attribute of $V_{L}$, for example, the meaning of words, we are finding a subspace $W$ of $V_{L}$ where there exists a function that takes in a word as an input and gives a set of vectors as an output. Formally speaking, we are looking at an attribute subspace $W_{a}$ such that there exists a function $F(w)\rightarrow\{\{y\}|y\in W_{a}\}$.

4. 4.

   A Word $w$ is a single vector in $V_{L}$.

Given the above definitions, we can see that:

1. 1.

   We can denote the “finding of all meanings of a word” can be expressed as $F(w)\rightarrow\{\{y\}|y\in W_{meaning}\}$, whereas “finding the most common meaning of a word” can be expressed as $F(w)\rightarrow\{y|y\in W_{meaning}\}$.

2. 2.

   Existing theories are just a further projection of the attribute subspace $W_{a}$ when we find that the dimension of $W_{a}$ still poses difficulties for us to understand. Note that a projection of $W_{a}$ also means that it is a valid projection of $V_{L}$.

3. 3.

   The way we can perform a projection is by the explication of the taxonomy and definition. It needs to include what we are interested in, what we are not, and a clear perspective on how we should consider this attribute. (For example)

By adopting a vectoring view of language, we can see that it is easy to address some hard logical problems that other perspectives suffer from. For example, we are historically concerned about the truth value of the sentence “The king of France is wise”. On the one hand, we can say that when projecting to Russell’s theory of referring, we get a result that the sentence is nonsense. On the other hand, we can directly say that we are projecting this sentence to a subspace of a function that gives us the truth value of the input, formally: $F_{truth}(x)\rightarrow{True,False}$. Then we can simply regard the sentence as neither True nor False but not well-defined for the function since the subject “The king of France” is undefined.

In the work “Word Meaning in Minds and Machines” by Lake et al.(2023)[4], the author proposes five reasons why the word representations generated from large language models do not represent word meanings. I will argue that while they do indeed exist in practical vectoring (LLMs), it is not that significant when we take vectoring as a perspective. The five reasons are:

The vectoring view of language showed great empirical achievements through the success of recent large language models. We can see that regardless of the different definitions our current theories have for word meanings, large language models can give us the meanings we want. This opens up the possibility that there are some definitions of attributes of language lying above what we can capture using language theories but can be approximated using practical vectoring.

The vectoring view of language also tells us crucial clues on how we can proceed with our research of the language. Theories that focus on similar topics generate projections with high correlation. For example, the theory of meaning and the theory of referring. While these theories discuss important topics of language, newer theories tend to give less contribution to our overall understanding of language given the high correlation with previous research. Alternatively, if we can combine multiple perspectives from totally different approaches, we can better understand the hidden shape of language thanks to the low correlation. What’s more, by tying the philosophy of language together with the rapidly innovating large language models, we not only provide a sturdy theory basis for AI but also gain access to unlimited experiments on how the language works, in the form of training new models with novel assumptions of the vector space of language.