Thoughts on Words and their Meaning
I think that the mathematics of machine learning can give us some insight into deep questions in philosophy that previously seemed intractable. Up until a few years ago, philosophers didn't have the technology or the faith that neural networks were actually effective at doing their jobs. Now that the technology appears convincing, it could be an opportune time to rehash these deep questions from the perspective of deep learning.
After looking at some of the developments in transformer models, especially encoders, I wanted to explore if simpler, small scale models can give us any insight for answering questions about creativity, meaning and abstraction in language. This post will be about using simple techniques in machine learning to offer some answers to philosophical questions in language.
I was amazed at the recent caption-to-image techniques used by Open-AI to make DALL-E 2. DALL-E 2 uses an architecture called a transformer model to translate captions into various realistic images. Transformer models, loosely speaking, consist of two parts - an encoder and a decoder. An encoder tries to find an efficient representation of a larger space of words, and a decoder takes it from this abstract representation back into content. Mathematically, it's trying to find some functions that minimise this value over everything in the space:
$$ \text{Find } \phi, \psi \text{ such that } L = \sum _ { v \in \chi } || v - \phi \circ \psi ( v ) \text{ is minimised} || ^ 2. $$
One obvious question to start with, yet probably the most difficult to answer, is what gives a word its meaning? This is a problem that has been one of the main themes in the philosophy of language. One common stream of thought is that words are signs that provoke mental images. This is part of the ideational theory of meaning, most commonly associated with the British empiricist John Locke.