Review written by Paula Brooks (NEU, G6)
How similar are bears and bulls?
If you ask a biologist, she might say that they are pretty similar, since they are both four-legged mammals found in North America. However, if you ask an economist, he might say they are polar opposites, since they are used to describe distinct stock market conditions. The unique way in which individuals organize their semantic knowledge, or general information gained through life experiences, could cause two people to judge the similarity between two animals in very different ways.
Scientists have been trying to understand the structure of semantic knowledge for a long time, in large part because it may lead to deeper insights about human behavior. After all, almost everything we do requires relating concepts to one another. Judging how similar two concepts are is fundamental to organizing and using our knowledge since it allows us to generalize what we know to new situations and also enables us to understand others when we communicate (even though we do not share identical life experiences!). If we had a firm grasp on how to estimate the similarity between any arbitrary pair of objects, it would help our understanding of how these processes operate for us as people and potentially even help us improve the ability of all the automated agents around us, like Siri or Alexa, to perform as well as we do.
The structure of semantic knowledge is often modeled using a multidimensional space where similarity is measured by how close the representation of different concepts (represented as words) are to each other in this space. This type of multidimensional space is called a word embedding space. In the example above, bear would probably be closer to bull in the multidimensional space for the biologist’s semantic knowledge as compared to that of the economist. One way scientists can approximate this space is by having people rate items on various measures (e.g., furriness, largeness) and then plotting these measures on a graph. For instance, bear might be higher on the ‘furriness’ measure compared to bull, but these two animals might be close to each other along the ‘largeness’ axis. However, this is pretty inefficient if you want to understand the relationship between hundreds or thousands of items. Recently, scientists have been leveraging advances in machine learning and copious amounts of information found on the world wide web to create data-driven algorithms to describe human similarity judgments.
In a recent paper led by Dr. Marius Cătălin Iordan, a postdoctoral scholar in the Princeton Neuroscience Institute, researchers were able to use their knowledge of human behavior—namely, that context matters—to build better algorithms for distilling human semantic knowledge. But what does it really mean to say that “context matters”? Let’s take our prior example: someone who just visited the natural history museum would most likely judge bears and bulls as being similar looking mammals compared to bears and penguins; however, that same person would probably judge bears and bulls as being opposites after leaving an economics lecture. As we can see, the circumstances surrounding the encounter with bears and bulls (or any other pair of items) matters in making appropriate decisions about them. To translate this to Iordan et al.’s paper, the researchers sought to prioritize the relevance of the data used to train their classifiers by only using information that was relevant to a particular context.
In order to prioritize context when developing their algorithms, Iordan et al. generated different word embedding spaces by extracting word associations from different collections of texts. Specifically, they used nature-related Wikipedia articles to create a contextually-constrained nature embedding space and transportation-related Wikipedia articles to create a contextually-constrained transportation embedding space. Both of these training sets included 50-70 million words. For comparison purposes, they also generated different embedding spaces using more traditional, contextually-unconstrained models that used billions of unrelated words. Next, Iordan et al. had human participants provide a similarity judgment between every pair of objects in a small subset of the nature or transportation contexts. They then compared the similarity judgements predicted by their models with actual human judgments to test how well their models performed.
Perhaps unsurprisingly, the contextually-constrained models (those that took the category of “nature” or “transportation” into account) did the best at predicting the human similarity judgments in the respective category. Importantly, these models outperformed algorithms that were trained using significantly larger training data (billions, as opposed to millions) that were not contextually-constrained. This shows that, when it comes to algorithmic performance, more data is not always better. Iordan et al.’s approach also highlights the benefit of incorporating semantic (or situational) context into the training procedure of algorithms designed to predict human behavioral variables, as the current algorithm (trained exclusively on articles tailored to a particular context) outperforms other models that only take into account a handful of words that come before or after a particular word of interest within a piece of text, despite the latter being trained on two orders of magnitude more data.
Why does this matter?
From a more consumeristic standpoint, every time you use Siri or ask Alexa a question, you are trying to get a computer to understand what you are trying to say through human language. The more we understand how semantic knowledge is organized, the better this technology will get. Importantly, Iordan et al. show that quality is better than quantity when it comes to training models that effectively predict human judgments. This could translate to Siri using an ‘economics’ contextually-constrained algorithm when you ask her to remind you of the difference between bears and bulls after an economics lecture, that might be different from the contextually-constrained algorithm she would use when interacting with a child learning his ABCs through animal cartoons. Moreover, this approach highlights the benefit of incorporating existing knowledge, like the utility of contexts in learning and memory, when building computer algorithms to understand human similarity judgements without actually testing thousands of people on billions of objects.
As we can see, contexts can be incredibly powerful tools not just for navigating day-to-day life, but also for building better machine learning algorithms that explain human behavior. This research highlights the benefit of taking a step back when using fancy new tools and asking oneself, “How can my knowledge of human behavior improve this tool?”
Dr. Marius Cătălin Iordan, first author of this study and incoming Assistant Professor in the Brain and Cognitive Sciences & Neuroscience Departments at the University of Rochester, is excited about this line of research and added the following: “As people, we carve up our world into hundreds of semantic contexts or domains and the shades of meaning between them matter to us. We're excited about our new models because we see them being useful in designing the next generation of artificial systems (such as machine translation, text prediction, video and image captioning, spoken language understanding) that can intelligently interact with us across all of these domains simultaneously in a way that better captures subtle differences of meaning between them.”
This article was published in Cognitive Science in February 2022. Please follow this link to view the full version.