Review written by Paula Brooks (PNI)
Imagine that you are binge-watching Netflix. In spite of the algorithm’s calculations, you are getting bored by the show that was suggested and you are thinking about stopping before the end of the season. However, to your great surprise, a new character enters halfway through the season and you are hooked. The plot has gotten more interesting and the acting has suddenly improved. What just happened?
It is interesting to look at this experience through the lens of reinforcement learning, which is a branch of machine learning that focuses on how people make decisions to maximize reward. In this example, you experienced a large reward prediction error (RPE): you were expecting to stay bored by the show, when in reality you were surprised by the plot and became enthralled. Put differently, RPE is a value that reflects the difference between an expected value (e.g., being bored) and the received reward (e.g., becoming enthralled). Previous studies have shown that large RPE values are correlated with significant changes in dopamine, an important neurotransmitter associated with receiving reward and feelings of pleasure. More recently, a 2018 study showed that experiencing a large RPE during learning leads to better memory of the surprising event. However, it was unclear how exactly the memory of the surprising event was enhanced, until now.
A recent paper led by Dr. Nina Rouhani, who completed her doctoral degree in Psychology and Neuroscience at Princeton in Spring 2020, investigates the mechanism behind the enhanced memory of the surprising event linked with a large RPE. Specifically, Rouhani et al. sought to understand how large RPE values change the structure of memories leading to better memory for surprising events. They presented two potential hypotheses that could explain this effect. First, a surprising event could be better remembered because the large RPE value might strongly bind the memory to the context of the event. If so, then the memory of the surprising event would be more accessible when the context is encountered. On the other hand, the surprising event might serve to segment the larger event (e.g., binge watching Netflix that day) into a context from before the surprising event (e.g., the boring episodes) and after the surprising event (e.g., the more exciting episodes). If this were the case, then scenes from the more exciting episodes should be better remembered when you are already thinking about these episodes because there is less interference from scenes of the more boring episodes.
Rouhani et al. tackled this question in a series of four online behavioral experiments and a computational model. Instead of having participants binge watch television, the authors used a memory task. In this task, participants viewed a stream of scene images that were associated with low or high monetary rewards. The values of the scenes tended to be around the same value (e.g., 7 cents, 12 cents) until a surprising event occurred and the value changed dramatically for a while (e.g., 90 cents, 94 cents).
In experiments 1-3, participants completed a memory task where they had to indicate whether scene images were old or new. Importantly, the memory task was sensitive to priming effects by presenting most scenes in pairs that were either in or out of sequence. The idea was that if the order was preserved, then presenting the first scene should trigger the memory and lead to faster reaction time of the second scene. This is in fact what the researchers found. Further, the effect of RPE value was examined by having the pairs be either from the same reward value condition (low RPE) or from different reward value conditions (high RPE). The researchers found better memory for high RPE scenes that were primed, which suggests that these surprising events were linked to the preceding scene. These results were replicated in experiment 2, when trials of the surprising events were not primed to demonstrate that the high RPE value was not enough to drive the better memory for the high RPE scenes. However, this was not the case when the scenes in the pair during the memory test straddled the two sides of the surprising event, as occurred in experiment 3. The reaction times were slower if the first scene came before while the second scene came after the surprising event, even though the two scenes were tested in sequence. These results suggest that surprising events serve as an event boundary that separates different contexts (e.g., low reward versus high reward).
In experiment 4, participants were shown two scene images at a time and they had to indicate which scene came first during learning and how many other images were found between them. Rouhani et al. found similar results even though this was a different memory task from their prior experiments. Participants tended to answer correctly on this sequence memory task for pairs that included the surprising event and a preceding scene. However, participants had difficulty providing the correct sequence if the pair included a scene from before the surprising event and a scene after it. These results also support the interpretation that participants created two contexts straddling the surprising event. Put together, these studies suggest that surprising events and their resulting large RPE values create event boundaries in memories (i.e., before and after surprising events). To supplement these behavioral findings, Rouhani et al. created a computational model to better understand the mechanism that leads to surprising events creating event boundaries. Using this model, they were able to simulate their behavioral results by manipulating a parameter that increased the extent to which surprising events updated the context, which in turn would lead to an event boundary.
According to these results, when you encounter a surprising event while binge-watching a Netflix show, you create an event boundary in your memories of binge watching television that day. Specifically, the boring episodes become a separate context from the exciting episodes later in the season. These findings suggest that surprising events have the important role of helping to split memories into different chunks. Such splitting is useful because event boundaries can help organize memories so as to reduce the interference of memories from a different context (e.g., the boring episodes) when trying to recall memories from a given context (e.g., the exciting episodes).
Dr. Nina Rouhani, first author of this study and now a postdoctoral fellow at Caltech, added that she is “extending this work to answer whether such reward-prediction-error boundaries shape how we perceive, remember, and predict the actions of other people.”
This article was published in Cognition in October 2020. Please follow this link to view the full version.