Can students evaluate their understanding of cause-and-effect relations? The effects of diagram completion on monitoring accuracy

Exploring the use of metacognitive monitoring cues following a diagram completion intervention

Studying texts constitutes a significant part of student learning in health professions education. Key to learning from text is the ability to effectively monitor one's own cognitive performance and take appropriate regulatory steps for improvement. Inferential cues generated during a learning experience typically guide this monitoring process. It has been shown that interventions to assist learners in using comprehension cues improve their monitoring accuracy. One such intervention is having learners to complete a diagram. Little is known, however, about how learners use cues to shape their monitoring judgments. In addition, previous research has not examined the difference in cue use between categories of learners, such as good and poor monitors. This study explored the types and patterns of cues used by participants after being subjected to a diagram completion task prior to their prediction of performance (PoP). Participants' thought processes were studied by means of a think-aloud method during diagram completion and the subsequent PoP. Results suggest that relying on comprehension-specific cues may lead to a better PoP. Poor monitors relied on multiple cue types and failed to use available cues appropriately. They gave more incorrect responses and made commission errors in the diagram, which likely led to their overconfidence. Good monitors, on the other hand, utilized cues that are predictive of learning from the diagram completion task and seemed to have relied on comprehension cues for their PoP. However, they tended to be cautious in their judgement, which probably made them underestimate themselves. These observations contribute to the current understanding of the use and effectiveness of diagram completion as a cue-prompt intervention and provide direction for future research in enhancing monitoring accuracy.

Worth the Effort: the Start and Stick to Desirable Difficulties (S2D2) Framework

Desirable difficulties are learning conditions that are often experienced as effortful, but have a positive effect on learning results and transfer of knowledge and skills (Bjork & Bjork, 2011; Bjork, 1994). Learners often do not appreciate the beneficial effects of desirable difficulties, and the negative experiences of high effort and perceived low learning make them resistant to engage in desirable difficulties (Biwer et al., 2020a). This ultimately limits learning outcomes and academic achievement. With the increasing emphasis on self-regulation in education, characterized by higher learner agency and abundant choices in what, when, and how to study, the field of educational psychology is in need of theoretical and empirically testable assumptions that improve self-regulation in desirably difficult learning conditions with the aim to foster self-regulation abilities, learning outcomes, and academic achievement. Here, we present a framework that describes how to support self-regulation of effort when engaging in desirable difficulties: the “Start and Stick to Desirable Difficulties (S2D2)” framework. The framework builds on the Effort Monitoring and Regulation model (de Bruin et al., 2020). The aim of this framework is (1) to describe evidence for the central role of perceived effort and perceived learning in (dis)engagement in desirable difficulties, and (2) to review evidence on, and provide an agenda for research to improve learners’ self-regulated use of desirable difficulties to help them start and persist when learning feels tough, but is actually effective.

Do Video Modeling and Metacognitive Prompts Improve Self-Regulated Scientific Inquiry?

Guided inquiry learning is an effective method for learning about scientific concepts. The present study investigated the effects of combining video modeling (VM) examples and metacognitive prompts on university students’ (N = 127) scientific reasoning and self-regulation during inquiry learning. We compared the effects of watching VM examples combined with prompts (VMP) to watching VM examples only, and to unguided inquiry (control) in a training and a transfer task. Dependent variables were scientific reasoning ability, hypothesis and argumentation quality, and scientific reasoning and self-regulation processes. Participants in the VMP and VM conditions had higher hypothesis and argumentation quality in the training task and higher hypothesis quality in the transfer task compared to the control group. There was no added benefit of the prompts. Screen captures and think aloud protocols during the two tasks served to obtain insights into students’ scientific reasoning and self-regulation processes. Epistemic network analysis (ENA) and process mining were used to model the co-occurrence and sequences of these processes. The ENA identified stronger co-occurrences between scientific reasoning and self-regulation processes in the two VM conditions compared to the control condition. Process mining revealed that in the VM conditions these processes occurred in unique sequences and that self-regulation processes had many self-loops. Our findings show that video modeling examples are a promising instructional method for supporting inquiry learning on both the process and the learning outcomes level.

Selecting effectively contributes to the mnemonic benefits of self-generated cues

Self-generated memory cues support recall of target information more robustly than memory cues generated by others. Across two experiments, we tested whether the benefit of self-generated cues in part reflects a meta-mnemonic effect rather than a pure generation effect. In other words, can learners select better memory cues for themselves than others can? Participants generated as many possible memory cues for each to-be-remembered target as they could and then selected the cue they thought would be most effective. Self-selected memory cues elicited better cued recall than cues the generator did not select and cues selected by observers. Critically, this effect cannot be attributed to the process of generating a cue itself because all of the cues were self-generated. Further analysis indicated that differences in cue selection arise because generators and observers valued different cue characteristics; specifically, observers valued the commonality of the cue more than the generators, while generators valued the distinctiveness of a cue more than observers. Together, results suggest that self-generated cues are effective at supporting memory, in part, because learners select cues that are tailored to their specific memory needs.

Exploring psychometric properties of children' metacognitive monitoring

Two independent data sets assessing children's metacognitive monitoring abilities were used to explore the psychometric properties of classical and often-used monitoring measures in primary school age. Theoretically, monitoring is an overarching skill that helps individuals evaluate task mastery, strategy use, and correctness of performance. Monitoring skills are increasingly targeted when addressing individual differences in scholastic achievement and intervention approaches to foster students' self-regulated learning early on. In such contexts, knowledge about central psychometric properties is essential. Results of both studies revealed high internal consistency of prospective and retrospective monitoring judgments. When equivalent item sets (in terms of item difficulty) were considered (Study 1), split-half reliabilities were also satisfying. However, analyses revealed that the monitoring judgments' reliability depends on the reliability of the first-order task (recognition memory test). Retesting children of Study 2 after six months revealed considerable fluctuations in the monitoring measures. Among the included monitoring measures, reliabilities of within-person correlations (Gammas) between performance and confidence and recognition response times and confidence were poorest. Results are discussed in the context of the underlying theoretical construct and implications for research and practice.

What Were You Thinking? Medical Students' Metacognition and Perceptions of Self-Regulated Learning

PHENOMENON

As a component of self-regulated learning, metacognition is gaining attention in the medical education research community. Metacognition, simply put, is thinking about one's thinking. Having a metacognitive habit of mind is essential for healthcare professionals. This study identified the metacognitive competencies of medical students as they completed a conceptual learning task, and provided insight into students' perceptions of self-regulated learning in their curriculum. Approach: Eleven third-year medical students from a Dutch University were purposively sampled to participate in this qualitative study. The study design included a think-aloud assignment followed by a semi-structured interview. During the assignment, participants were instructed to think aloud while solving questions about medical physiological concepts such as blood flow, pressure, and resistance. Think-aloud data were collected through audiotaping and used to identify participants' metacognitive competencies. The assignment also served as a prompt for an interview in which participants were questioned about metacognitive knowledge, monitoring, experiences, and perceptions of self-regulated learning in their curriculum. All data were transcribed verbatim and analyzed iteratively using a template analysis. Findings: Students differed in their use of metacognitive skills, with an overall focus on monitoring and, to a lesser extent, on planning and evaluation. Additionally, differences were found in students' metacognitive knowledge and metacognitive experiences. There was apparent use of inefficient, superficial predictive cues. Regarding perceptions of self-regulated learning skills, some students felt no need to develop such skills as they perceived medical education as an exercise in memorizing facts. Others emphasized the need for more insight into their actual level of knowledge and competence. Insights: Pre-clinical medical students require explicit teaching of metacognitive skills to facilitate self-regulated learning. Educators should aim to integrate metacognition in the everyday discourse of the classroom to foster an environment in which students discuss their own learning.

Supplemental data for this article is available online at https://doi.org/10.1080/10401334.2021.1889559.

Does increasing social presence enhance the effectiveness of writing explanations?

Writing explanations has demonstrated to be less effective than providing oral explanations, as writing triggers less amounts of perceived social presence during explaining. In this study, we investigated whether increasing social presence during writing explanations would aid learning. University students (N = 137) read an instructional text about immunology; their subsequent task depended on experimental condition. Students either explained the contents to a fictitious peer orally, wrote their explanations in a text editor, or wrote them in a messenger chat, which was assumed to induce higher levels of social presence. A control group retrieved the material. Surprisingly, we did not obtain any differences in learning outcomes between experimental conditions. Interestingly, explaining was more effortful, enjoyable, and interesting than retrieving. This study shows that solely inducing social presence does not improve learning from writing explanations. More importantly, the findings underscore the importance of cognitive and motivational conditions during learning activities.

Improving Metacognition in the Classroom

Abstract. Students are often overconfident in educational settings and struggle to differentiate between well-learned and poorly-learned concepts. The present article reviews current research on strategies that help students assess their understanding, with a focus on research using authentic educational tasks and materials. We propose a framework for these strategies that we refer to as wait-generate-validate. The wait-generate-validate strategies can give students a more objective measure of their learning from lectures, understanding of course concepts, text comprehension, problem-solving ability, and test preparedness. These strategies have been shown to lead to more effective study decisions and greater learning. Lastly, we translate the reviewed research into practical tips for students and teachers and conclude with recommendations for future research regarding how students judge their learning in diverse educational contexts.

Developmental Improvements and Persisting Difficulties in Children's Metacognitive Monitoring and Control Skills: Cross-Sectional and Longitudinal Perspectives

This study investigated age‐dependent improvements of monitoring and control in 7/8‐ and 9/10‐year‐old children. We addressed prospective (judgments of learning and restudy selections) and retrospective metacognitive skills (confidence judgments and withdrawal of answers). Children (N = 305) completed a paired‐associate learning task twice, with a 1‐year delay. Results revealed improvements in retrospective, but not in prospective monitoring and control. Furthermore, control remained suboptimal, seemingly a consequence of overoptimistic monitoring. Both age groups showed stronger monitoring‐based control at the second compared to the first assessment. The comparison with a cross‐sectional sample (N = 144) revealed that improvements in retrospective monitoring can be mainly attributed to naturally occurring development, whereas retrospective control seemed to improve due to increased task familiarity.

To What Extent Do Situation-Model-Approach Interventions Improve Relative Metacomprehension Accuracy? Meta-Analytic Insights

This meta-analysis investigated the extent to which relative metacomprehension accuracy can be increased by interventions that aim to support learners’ use of situation-model cues as a basis for judging their text comprehension. These interventions were delayed-summary writing, delayed-keywords listing, delayed-diagram completion, self-explaining, concept mapping, rereading, and setting a comprehension-test expectancy. First, the general effectiveness of situation-model-approach interventions was examined. The results revealed that, across 28 effect sizes (comprising a total of 2,236 participants), situation-model-approach interventions exerted a medium positive effect (g = 0.46) on relative metacomprehension accuracy. Second, the interventions were examined individually. The results showed that, with the exception of self-explaining, each intervention had a significant positive effect on relative metacomprehension accuracy. Yet, there was a tendency for setting a comprehension-test expectancy to be particularly effective. A further meta-analysis on comprehension in the selected studies revealed that, overall, the situation-model-approach interventions were also beneficial for directly improving comprehension, albeit the effect was small. Taken together, the findings demonstrate the utility of situation-model-approach interventions for supporting self-regulated learning from texts.

Synthesizing Cognitive Load and Self-regulation Theory: a Theoretical Framework and Research Agenda

An exponential increase in the availability of information over the last two decades has asked for novel theoretical frameworks to examine how students optimally learn under these new learning conditions, given the limitations of human processing ability. In this special issue and in the current editorial introduction, we argue that such a novel theoretical framework should integrate (aspects of) cognitive load theory and self-regulated learning theory. We describe the effort monitoring and regulation (EMR) framework, which outlines how monitoring and regulation of effort are neglected but essential aspects of self-regulated learning. Moreover, the EMR framework emphasizes the importance of optimizing cognitive load during self-regulated learning by reducing the unnecessary load on the primary task or distributing load optimally between the primary learning task and metacognitive aspects of the learning task. Three directions for future research that derive from the EMR framework and that are discussed in this editorial introduction are: (1) How do students monitor effort? (2) How do students regulate effort? and (3) How do we optimize cognitive load during self-regulated learning tasks (during and after the primary task)? Finally, the contributions to the current special issue are introduced.

Mapping and Drawing to Improve Students’ and Teachers’ Monitoring and Regulation of Students’ Learning from Text: Current Findings and Future Directions

For (facilitating) effective learning from texts, students and teachers need to accurately monitor students’ comprehension. Monitoring judgments are accurate when they correspond to students’ actual comprehension. Accurate monitoring enables accurate (self-)regulation of the learning process, i.e., making study decisions that are in line with monitoring judgments and/or students’ comprehension. Yet, (self-)monitoring accuracy is often poor as the information or cues used are not always diagnostic (i.e., predictive) for students’ actual comprehension. Having students engage in generative activities making diagnostic cues available improves monitoring and regulation accuracy. In this review, we focus on generative activities in which text is transformed into visual representations using mapping and drawing (i.e., making diagrams, concept maps, or drawings). This has been shown to improve monitoring and regulation accuracy and is suited for studying cue diagnosticity and cue utilization. First, we review and synthesize findings of studies regarding (1) students’ monitoring accuracy, regulation accuracy, learning, cue diagnosticity, and cue utilization; (2) teachers’ monitoring and regulation accuracy and cue utilization; and (3) how mapping and drawing affect using effort as a cue during monitoring and regulation, and how this affects monitoring and regulation accuracy. Then, we show how this research offers unique opportunities for future research on advancing measurements of cue diagnosticity and cue utilization and on how effort is used as a cue during monitoring and regulation. Improving measures of cue diagnosticity and cue utilization can provide us with more insight into how students and teachers monitor and regulate students’ learning, to help design effective interventions to foster these important skills.

The Role of Mental Effort in Fostering Self-Regulated Learning with Problem-Solving Tasks

Problem-solving tasks form the backbone of STEM (science, technology, engineering, and mathematics) curricula. Yet, how to improve self-monitoring and self-regulation when learning to solve problems has received relatively little attention in the self-regulated learning literature (as compared with, for instance, learning lists of items or learning from expository texts). Here, we review research on fostering self-regulated learning of problem-solving tasks, in which mental effort plays an important role. First, we review research showing that having students engage in effortful, generative learning activities while learning to solve problems can provide them with cues that help them improve self-monitoring and self-regulation at an item level (i.e., determining whether or not a certain type of problem needs further study/practice). Second, we turn to self-monitoring and self-regulation at the task sequence level (i.e., determining what an appropriate next problem-solving task would be given the current level of understanding/performance). We review research showing that teaching students to regulate their learning process by taking into account not only their performance but also their invested mental effort on a prior task when selecting a new task improves self-regulated learning outcomes (i.e., performance on a knowledge test in the domain of the study). Important directions for future research on the role of mental effort in (improving) self-monitoring and self-regulation at the item and task selection levels are discussed after the respective sections.

Development of Children's monitoring and control when learning from texts: effects of age and test format

This study investigated elementary school children's development of monitoring and control when learning from texts. Second (N = 138) and fourth (N = 164) graders were tested in the middle (T₁) and end (T₂) of the school year. The study focused on the cross-sectional and longitudinal development of monitoring and control, and aimed to investigate the development of metacognition for two test formats. After reading expository texts, children completed a comprehension test consisting of open-ended and true-false questions. They monitored their test performance by making confidence judgments, and controlled performance by deciding whether to maintain or withdraw their given answers. Overall, monitoring and control accuracy was higher for open-ended questions than for true-false questions. For open-ended questions, results indicated higher metacognitive accuracy for fourth graders than second graders. No such age effects were found for monitoring and control for true-false questions. Longitudinally, children of both age groups improved their monitoring and control accuracy from T₁ to T₂, for open-ended and true-false questions. For both test types, improvement mainly occurred for the monitoring and controlling of incorrect, rather than correct answers. Additionally, the results indicated inter-individual stability of performance, but no stability of monitoring and control accuracy over time. The findings indicate that developmental as well as task-related factors affect children's metacognitive accuracy.

Metamemory viewed through the judgment lens

Metamemory research makes extensive use of judgments, such as judgments of learning (JOLs). In a JOL, people predict their chance of remembering a recently studied item in a memory test. There is a general agreement that JOLs rely on probabilistic cues that are combined in an inference process. Accuracy as measured by the gamma correlation between JOLs and actual performance is usually mediocre, suggesting limited metacognitive abilities. In judgment and decision-making research, Brunswik's lens model is often used to decompose judgmental accuracy: A matching index G measures how adequately people's cue weights match the optimal weights, two reliability indices assess the predictability of judgments and environment, respectively, and a nonlinear component measures systematic variance not captured by the cues. We employed the lens model equation for the first time to analyze four published and one new JOL data sets. There was considerable interindividual variance in metamemory monitoring. Although gamma was on average higher than the Pearson correlation, it still underestimated metacognitive ability in terms of matching (G). Also, the nonlinear component was considerably higher than in other judgment domains, pointing to substantial item-person-interactions that we interpret as idiosyncratic encoding strategies. An exploratory cluster analysis suggests different metacognitive strategies used by subgroups of participants. We suggest the lens model as a potentially promising tool in metacognition research.

Monitoring and regulation of learning in medical education: the need for predictive cues

CONTEXT

Being able to accurately monitor learning activities is a key element in self-regulated learning in all settings, including medical schools. Yet students' ability to monitor their progress is often limited, leading to inefficient use of study time. Interventions that improve the accuracy of students' monitoring can optimise self-regulated learning, leading to higher achievement. This paper reviews findings from cognitive psychology and explores potential applications in medical education, as well as areas for future research.

COGNITIVE PSYCHOLOGY

Effective monitoring depends on students' ability to generate information ('cues') that accurately reflects their knowledge and skills. The ability of these 'cues' to predict achievement is referred to as 'cue diagnosticity'. Interventions that improve the ability of students to elicit predictive cues typically fall into two categories: (i) self-generation of cues and (ii) generation of cues that is delayed after self-study. Providing feedback and support is useful when cues are predictive but may be too complex to be readily used.

APPLICATION TO MEDICAL EDUCATION

Limited evidence exists about interventions to improve the accuracy of self-monitoring among medical students or trainees. Developing interventions that foster use of predictive cues can enhance the accuracy of self-monitoring, thereby improving self-study and clinical reasoning. First, insight should be gained into the characteristics of predictive cues used by medical students and trainees. Next, predictive cue prompts should be designed and tested to improve monitoring and regulation of learning. Finally, the use of predictive cues should be explored in relation to teaching and learning clinical reasoning.

CONCLUSIONS

Improving self-regulated learning is important to help medical students and trainees efficiently acquire knowledge and skills necessary for clinical practice. Interventions that help students generate and use predictive cues hold the promise of improved self-regulated learning and achievement. This framework is applicable to learning in several areas, including the development of clinical reasoning.