The findings show that the brain does more work right when it meets information in a different codality, rather than waiting until the story moves on. Viewing times rose most sharply for the panel that presented the bridging event, indicating integration occurs at the moment of encounter. That timing detail matters because it changes how we think about designing stories and educational materials that mix text and images.
For anyone interested in learning, design, and inclusion, this research raises practical questions. How might textbooks, instructional guides, or accessible news feeds be arranged so transitions between pictures and words help rather than hinder understanding? Follow the full article to see the methods and examples, and to explore how adding codality to models of event cognition could expand opportunities for clearer communication and broader engagement.
Abstract
Despite the prevalence of multi-codal narratives—such as comics consisting of pictures and text—in everyday life, little is understood about how individuals integrate information across different codalities to form cohesive mental representations. In the present study, we explored when processing costs emerged during cross-codal integration of bridging event information in multi-codal narratives. Participants were presented with uni-codal (i.e., consisting only of pictures or of text) and multi-codal stories in which the bridging event information was presented in the other codality. We measured viewing times for the bridging event and end-state panels to reflect the cognitive costs associated with integration. The results showed a significant interaction between codality switch and panel position. While the presence of a codality switch increased viewing times for both panel positions, there was a much larger increase for the bridging event panel than the end-state panel. This result supports a hypothesis suggesting that during narrative understanding, information of a different codality is integrated into the mental representation of the narrative upon encountering it. Our results underscore the influence of codality on narrative integration, indicating that understanding may be affected by the format in which information is presented. Future models of event cognition should incorporate codality as a factor in narrative comprehension.
