基于情绪关联的多模态认知负荷量化

Emotion correlation-based multimodal cognitive load quantification

  • 摘要: 现代人机交互系统信息高度密集、任务复杂多变. 情绪是交互任务中的固有心理产物,为探讨人机交互任务中操作员脑力负荷的变化,依据认知负荷理论和成因分析,将认知负荷划分为记忆力负荷、专注力负荷、情绪情感,分别对应内部认知负荷、外部认知负荷、关联认知负荷. 以情绪作为内部认知负荷与外部认知负荷联系的桥梁,继而对整体人机交互的认知负荷进行分析. 同时基于多模态数据协同,提出了人机交互过程中操作人员脑力负荷动态预测模型. 为证明并校验所提出的模型,选取53名被试完成3种渐进式难度的心理学范式. 结果验证了不同类型负荷激发任务下多模态信号、不同等级负荷激发任务下多模态信号与被试报告的主观评价的主效应显著(P<0.05);总脑力负荷预测值与被试主观评价、子负荷预测值与子负荷量表主观评价呈显著正相关;平均脑力负荷预测值与对应多模态信号呈显著正相关. 所提模型对人机交互过程中操作人员脑力负荷的动态预测与评价具有应用价值.

     

    Abstract: Modern human–computer interaction (HCI) systems are characterized by high information density and complex tasks. In such systems, information is often presented in large volumes at rapid speeds, requiring users to process and comprehend it quickly. At the same time, tasks are intricate and multifaceted, involving multiple steps and decision-making processes. Monitoring the cognitive load in HCI tasks is of paramount importance because it can significantly enhance system performance and reduce the likelihood of operational errors. When users experience excessive cognitive load, their ability to process information and make decisions may be impaired, leading to decreased efficiency, increased error rates, and even system failures. By effectively monitoring cognitive load, system designers can identify potential bottlenecks and optimize the user experience to ensure smooth and efficient interaction. Cognitive load theory, supported by extensive causal research, provides a framework for understanding and categorizing cognitive load into three distinct types: internal, external, and associative. Internal cognitive load is closely tied to the brain’s information processing mechanisms, particularly memory resources. It reflects the mental effort required to encode, store, and retrieve information from memory. Conversely, external cognitive load stems from the presentation and difficulty of learning materials. It is influenced by how information is organized and displayed to the user, as well as the inherent complexity of the task. Both internal and external cognitive loads are critical factors that affect a user’s cognitive state during HCI tasks. However, directly measuring these two types of cognitive load is often challenging and time-consuming because it requires invasive or complex experimental setups. Given the limitations of direct measurement, this study proposes an innovative approach by leveraging the relationship between negative emotions and multimodal signals as a proxy for direct cognitive load measurement. Emotions, especially negative ones, are inherent psychological products of interactive tasks and are closely linked to cognitive processes. When users experience a high cognitive load, they often exhibit emotional responses such as frustration, anxiety, or stress. These emotional states can be reflected in various physiological and behavioral signals such as facial expressions, voice tone, eye movements, and physiological indicators such as heart rate and skin conductance. By capturing and analyzing these multimodal signals, we can indirectly infer the cognitive load experienced by users. To simulate the occurrence of cognitive load in HCI tasks, three repeatable visual stimuli were designed: digit memory, tracking, and combined memory-tracking tasks. The digit memory task focuses on memory by requiring participants to remember sequences of numbers. The tracking task emphasizes attention by asking participants to follow a moving target on the screen. The combined memory-tracking task integrates both memory and attention by requiring participants to remember numbers while simultaneously tracking a moving target. The difficulty of each task type increases progressively, and the overall stimulus difficulty was modeled as a linear increase. This design allowed for a systematic investigation of how cognitive load evolves as task complexity increases. A total of 53 participants completed the designed paradigm experiments. They were required to complete the NASA-TLX workload assessment and PANAS emotion assessment scales. Several multimodal and multichannel signals during the experiment were recorded for subsequent analysis, including physiological data, eye-tracking data, and behavioral responses. This study proposes a novel cognitive load measurement method to quantify the psychological stress generated by subjects during HCI. By analyzing the differences in emotion-related indices, conducting quantitative calculations, and performing classification verification, the study aims to establish a robust model for cognitive load assessment. The results demonstrated that the proposed progressive paradigm successfully induced a psychological load in the subjects. A significant correlation was found between the NASA-TLX scores and PANAS negative emotion scores, indicating that as the cognitive load increased, participants tended to experience more negative emotions. During the experiment, numerous multimodal and multichannel signals exhibited significant differences, suggesting that emotion-related signals undergo marked changes during cognitive load induction. According to the proposed calculation method, a significant correlation exists between the quantitative load and subjective load-level reports of the subjects. This finding validates the effectiveness of the proposed method in capturing cognitive load variations. Using different signals, the linear classification model achieved an accuracy of over 93%, which demonstrates its potential to accurately predict and assess cognitive load in real time. The ability to dynamically track the cognitive load can provide valuable feedback to system designers, enabling them to make timely adjustments and optimizations to the user interface and task design. Overall, the proposed model holds great promise for enhancing the user experience in HCI by providing a noninvasive and efficient way to monitor cognitive load.

     

/

返回文章
返回