Capturing the clinical decision-making processes of expert and novice diabetic retinal graders using a 'think-aloud' approach

BACKGROUND

Diabetic eye screening programmes have been developed worldwide based on evidence that early detection and treatment of diabetic retinopathy are crucial to preventing sight loss. However, little is known about the decision-making processes and training needs of diabetic retinal graders, particularly in low- and middle-income countries.

OBJECTIVES

To provide data for improving evidence-based diabetic retinopathy training to help novice graders process fundus images more like experts.

SUBJECTS/METHODS

This is a mixed-methods qualitative study conducted in southern Vietnam and Northern Ireland. Novice diabetic retinal graders in Vietnam (n = 18) and expert graders in Northern Ireland (n = 5) were selected through a purposive sampling technique. Data were collected from 21st February to 3rd September 2019. The interviewer used neutral prompts during think-aloud sessions to encourage participants to verbalise their thought processes while grading fundus images from anonymised patients, followed by semi-structured interviews. Thematic framework analysis was used to identify themes, supported by illustrative quotes from interviews. Mann-Whitney U tests were used to compare graders' performance.

RESULTS

Expert graders used a more systematic approach when grading images, considered all four images per patient and used available software tools such as red-free filters prior to making a decision on management. The most challenging features for novice graders were intra-retinal microvascular abnormalities and new vessels, which were more accurately identified by experts.

CONCLUSION

Taking more time to grade fundus images and adopting a protocol-driven "checklist" approach may help novice graders to function more like experts.

Using Virtual Patients to Explore the Clinical Reasoning Skills of Medical Students: Mixed Methods Study

BACKGROUND

Improving clinical reasoning skills-the thought processes used by clinicians to formulate appropriate questions and diagnoses-is essential for reducing missed diagnostic opportunities. The electronic Clinical Reasoning Educational Simulation Tool (eCREST) was developed to improve the clinical reasoning of future physicians. A feasibility trial demonstrated acceptability and potential impacts; however, the processes by which students gathered data were unknown.

OBJECTIVE

This study aims to identify the data gathering patterns of final year medical students while using eCREST and how eCREST influences the patterns.

METHODS

A mixed methods design was used. A trial of eCREST across 3 UK medical schools (N=148) measured the potential effects of eCREST on data gathering. A qualitative think-aloud and semistructured interview study with 16 medical students from one medical school identified 3 data gathering strategies: Thorough, Focused, and Succinct. Some had no strategy. Reanalysis of the trial data identified the prevalence of data gathering patterns and compared patterns between the intervention and control groups. Patterns were identified based on 2 variables that were measured in a patient case 1 month after the intervention: the proportion of Essential information students identified and the proportion of irrelevant information gathered (Relevant). Those who scored in the top 3 quartiles for Essential but in the lowest quartile for Relevant displayed a Thorough pattern. Those who scored in the top 3 quartiles for Relevant but in the lowest quartile for Essential displayed a Succinct pattern. Those who scored in the top 3 quartiles on both variables displayed a Focused pattern. Those whose scores were in the lowest quartile on both variables displayed a Nonspecific pattern.

RESULTS

The trial results indicated that students in the intervention group were more thorough than those in the control groups when gathering data. The qualitative data identified data gathering strategies and the mechanisms by which eCREST influenced data gathering. Students reported that eCREST promoted thoroughness by prompting them to continuously reflect and allowing them to practice managing uncertainty. However, some found eCREST to be less useful, and they randomly gathered information. Reanalysis of the trial data revealed that the intervention group was significantly more likely to display a Thorough data gathering pattern than controls (21/78, 27% vs 6/70, 9%) and less likely to display a Succinct pattern (13/78, 17% vs 20/70, 29%; χ²₃=9.9; P=.02). Other patterns were similar across groups.

CONCLUSIONS

Qualitative data suggested that students applied a range of data gathering strategies while using eCREST and that eCREST encouraged thoroughness by continuously prompting the students to reflect and manage their uncertainty. Trial data suggested that eCREST led students to demonstrate more Thorough data gathering patterns. Virtual patients that encourage thoroughness could help future physicians avoid missed diagnostic opportunities and enhance the delivery of clinical reasoning teaching.

The Development of Skill Knowledge in Conservation

In conservation, the practical ability to implement treatment on objects of cultural heritage requires skills as an essential part of conservation expertise and constitutes an fundamental part of learning in conservation education. The acquisition of practical skills, which include cognitive and practical elements, is governed by laws that are primarily explained by means of organizational psychology and medicine where they have evolved in the form of step models. Stages of explicit and implicit knowledge are distinguished as well as factual-theoretical and practical-performative knowledge. Forms of learning are, in addition to guided learning, the experiment, the emergency, case studies and implicit forms of problem solving. The development of the expert can also be traced by the development of practical skills in five stages from beginner to expert. The stages of skills acquisition can be explained by reference to the European Qualifications Framework for Lifelong Learning (EQF) and the skill levels defined by the European Confederation of Conservator-Restorers’ Organizations (E.C.C.O.).

Moving from Novice to Expertise and Its Implications for Instruction

Objective: To address the stages of expertise development, what differentiates a novice from an expert, and how the development and differences impact how we teach our classes or design the curriculum. This paper will also address the downside of expertise and discuss the importance of teaching expertise relative to domain expertise. Summary: Experts develop through years of experience and by progressing from novice, advance beginner, proficient, competent, and finally expert. These stages are contingent on progressive problem solving, which means individuals must engage in increasingly complex problems, strategically aligned with the learner's stage of development. Thus, several characteristics differentiate experts from novices. Experts know more, their knowledge is better organized and integrated, they have better strategies for accessing knowledge and using it, and they are self-regulated and have different motivations.

Medical Problem Solving

This essay reviews the origins, findings and influence of the monograph Medical Problem Solving: An Analysis of Clinical Reasoning. Majorfindings of the monograph are reviewed in the light of subsequent work and the results of selected studies of clinical cognition are related to the book's conclusions, thus sketching the growth of this field of research in the decade since publication. Several remaining methodologicalproblems and scholarly issues in the field are discussed, including: sampling cases and subjects, the definition of medical expertise, the role of verbal report in analyzing thinking, the level of clinical realism needed in research, and the relation of informationprocessing approaches to more quantitative approaches such as behavioral decision theory and social judgment theory.

Diagnostic Reasoning and Decision Making in the Context of Health Information Technology

Diagnostic reasoning and medical decision making have been focal areas of research in the fields of medical education, cognition, and artificial intelligence in medicine. Drawing on several decades worth of research, we propose an integrated summary of prior research on diagnostic reasoning and decision making—in terms of both historical development and theoretical shifts. We also characterize the changes in research and theory resulting from the incorporation and adoption of health information technology in the clinical work place. In this paper, we differentiate between the various forms of diagnostic reasoning and trace the evolution of the various models of reasoning, including knowledge-based, exemplar-based, and visual strategies. We also discuss the effect of clinical expertise on reasoning processes. Within the medical decision-making research, we delineate the various approaches highlighting decision-making errors that arise due to the nature of heuristics and biases and other factors. Although there has been significant progress in our understanding, there is still a need for greater theoretical integration of disparate empirical phenomena. Specifically, there is a need to reconcile the various characterizations of reasoning and to evaluate the similarity and differences in the context of current health care practice. Finally, we discuss the role of human factors research in the study of clinical environments and also in relation to devising approaches and methodologies for understanding, evaluating, and supporting the diagnostic reasoning and decision processes.

Unexpected downshifts in reward magnitude induce variation in human behavior

We investigated how changes in outcome magnitude affect behavioral variation in human volunteers. Our participants entered strings of characters using a computer keyboard, receiving feedback (gaining a number of points) for any string at least ten characters long. During a "surprise" phase in which the number of points awarded was changed, participants only increased their behavioral variability when the reward value was downshifted to a lower amount, and only when such a shift was novel. Upshifts in reward did not have a systematic effect on variability.

Cognitive emulation in expert system design

Cognitive emulation is an expert system design strategy which attempts to model system performance on human (expert) thinking. Arguments for and against cognitive emulation are reviewed. A major conclusion is that a significant degree of cognitive emulation is an inherent feature of design, but that an unselective application of the strategy is both unrealistic and undesirable. Pragmatic considerations which limit or facilitate the viability of a cognitive emulation approach are discussed. Particular attention is given to the conflict between cognitive emulation and established knowledge engineering objectives, detailed over 12 typical expert system features. The paper suggests circumstances in which a strategy of cognitive emulation is useful.

What is a deep expert system? An analysis of the architectural requirements of second-generation expert systems

First-generation expert systems have significant limitations, often attributed to their not being sufficiently deep. However, a generally accepted answer to “What is a deep expert system?” is still to be given. To answer this question one needs to answer “Why do first-generation systems exhibit the limitations they do?” thus identifying what is missing from first-generation systems and therefore setting the design objectives for second-generation (i.e. deep) systems. Several second-generation architectures have been proposed; inherent in each of these architectures is a definition of deepness. Some of the proposed architectures have been designed with the objective of alleviating a subset, rather than the whole set, of the first-generation limitations. Such approaches are prone to local, non-robust solutions. In this paper we analyze the limitations (under the categories: human-computer interaction, problem-solving flexibility, and extensibility) of the first-generation expert systems thus setting design goals for second-generation systems. On the basis of this analysis proposed second-generation architectures are reviewed and compared. The paper concludes by presenting requirements for a generic second-generation architecture.

Cognitive emulation in expert system design

Cognitive emulation is an expert System design strategy which attempts to model System performance on human (expert) thinking. Arguments for and against cognitive emulation are reviewed. A major conclusion is that a significant degree of cognitive emulation is an inherent feature of design, but that an unselective application of the strategy is both unrealistic and undesirable. Pragmatic considerations which limit or facilitate the viability of a cognitive emulation approach are discussed. Particular attention is given to the conflict between cognitive emulation and established knowledge engineering objectives, detailed over 12 typical expert System features. The paper suggests circum-stances in which a strategy of cognitive emulation is useful.

The coaching schematic: validation through expert coach consensus

Current research in coaching development infers that coaching is predominantly a decision-making process. The same and other research is not, however, informing the coaching development process due to a lack of a big picture approach. Consequently, there is a need for a model of coaching. In this paper, we offer such a model in the form of a schematic that reflects the coaching process from both a content and information-processing stance. To assess the validity of the schematic, 16 expert coaches were interviewed to elicit a complete description of their coaching process. The coaches were then shown a copy of the schematic and asked to comment on its design and content with respect to its accuracy in reflecting their coaching process. Following analysis of the interview, six general categories emerged: Roles, Goals, Typical Actions, Required Knowledge, Support for the Schematic, and Factors Influencing Development. The first four categories clearly displayed an implicit support for the schematic. Furthermore, all coaches offered explicit support for the schematic. Such strong support confirmed that the schematic was valid and could form the basis of focused interventions in coaching development.

The role of basic science knowledge and clinical knowledge in diagnostic reasoning: a structural equation modeling approach

PURPOSE

To examine four theories on the role of basic science knowledge and clinical knowledge in diagnostic reasoning.

METHOD

In 2000-01, the authors tested the basic science and clinical knowledge and diagnostic performances of 59 family physicians and 184 second- to sixth-year medical students at Maastricht University, The Netherlands. Structural equation modeling was used to analyze the data. Four theoretical models were tested. In the first model only basic science knowledge is involved in diagnostic reasoning; in the second model only clinical knowledge is related to diagnostic reasoning; in the third model, clinical knowledge is related to diagnostic reasoning, but basic science knowledge is integrated in clinical knowledge; and in the fourth model, both basic science knowledge and clinical knowledge independently influence diagnostic reasoning.

RESULTS

Forty-four (75%) of the family physicians and 184 (100%) of the students responded. The results indicated that the third model, which is based on the knowledge encapsulation theory, provided the best fit to the data, whereas the models that had directly related basic science knowledge with diagnostic performance did not fit the data adequately.

CONCLUSION

The results generally supported the third model by Schmidt and Boshuizen of knowledge encapsulation theory suggesting that basic science knowledge is activated in expert diagnostic reasoning through its relation with clinical knowledge.

A primer on aspects of cognition for medical informatics

As a multidisciplinary field, medical informatics draws on a range of disciplines, such as computer science, information science, and the social and cognitive sciences. The cognitive sciences can provide important insights into the nature of the processes involved in human- computer interaction and help improve the design of medical information systems by providing insight into the roles that knowledge, memory, and strategies play in a variety of cognitive activities. In this paper, the authors survey literature on aspects of medical cognition and provide a set of claims that they consider to be important in medical informatics.

Factors influencing analysis of complex cognitive tasks: a framework and example from industrial process control

We propose that considering four categories of task factors can facilitate knowledge elicitation efforts in the analysis of complex cognitive tasks: materials, strategies, knowledge characteristics, and goals. A study was conducted to examine the effects of altering aspects of two of these task categories on problem-solving behavior across skill levels: materials and goals. Two versions of an applied engineering problem were presented to expert, intermediate, and novice participants. Participants were to minimize the cost of running a steam generation facility by adjusting steam generation levels and flows. One version was cast in the form of a dynamic, computer-based simulation that provided immediate feedback on flows, costs, and constraint violations, thus incorporating key variable dynamics of the problem context. The other version was cast as a static computer-based model, with no dynamic components, cost feedback, or constraint checking. Experts performed better than the other groups across material conditions, and, when required, the presentation of the goal assisted the experts more than the other groups. The static group generated richer protocols than the dynamic group, but the dynamic group solved the problem in significantly less time. Little effect of feedback was found for intermediates, and none for novices. We conclude that demonstrating differences in performance in this task requires different materials than explicating underlying knowledge that leads to performance. We also conclude that substantial knowledge is required to exploit the information yielded by the dynamic form of the task or the explicit solution goal. This simple model can help to identify the contextual factors that influence elicitation and specification of knowledge, which is essential in the engineering of joint cognitive systems.

Barriers to acceptance of medical error: the case for a teaching program (695)

There is need for a teaching programme aiming to impart a tolerance of error to undergraduate medical students. The implementation of such a programme may have to challenge the institutional norms that encourage authoritarianism, intolerance of uncertainty and denial of error. Acceptance of error is a prerequisite for its candid reporting, and reporting of errors is a prerequisite for their analysis with a view to their prevention. A curriculum on medical error may, therefore, not only help medical students cope with their future mistakes, but also reduce their frequency. Teaching intervention aiming to promote an acceptance of medical error as both inevitable and reducible may also encourage students' epistemological development by making them realize that their doubts and uncertainties are shared by their peers and instructors.

Characterization of Problem Solving in Audiology

Characterizing diagnostic problem solving by students, intermediates, and experts may facilitate an understanding of how to help students attain required diagnostic skills. Studies in other specialties characterize experts as spending more time and effort in understanding a problem before they begin to solve it. That is, at the beginning of a problemsolving episode, experts work to limit the problem space. On the other hand, novices often plunge immediately into the solution process. In order to learn more about diagnostic problem-solving in audiology, a computer-based environment, Simon Says ©, was used to assist in the collection of problem-solving data. The results of two experiments revealed that accurate diagnosis in audiology appears to be related to the ability to set up patient problem contexts by analyzing presenting symptoms and history information and using the more focused context to interpret test data.

Recent cognitive perspectives on learning--implications for nurse education

Nurse educators must keep abreast of contemporary learning theory so that their teaching reflects current ideas of best practice. In view of this, it is important to report on recent developments in the field of learning. Of particular significance is the fact that behaviouristic explanations of learning have largely been replaced with cognitive perspectives which emphasise the complexity of the learning process. Memory, learning, problem solving and expertise have all been investigated from a cognitive stance. The highlights of this work include, firstly, the portrayal of learning as an active, constructivist, cumulative and self-regulated process leading to the development of understanding and complex, skilled performance. Secondly, the highly important role played by knowledge in learning has been identified and described. Lastly, novice-expert differences in problem solving and academic and practical performance more generally, are well understood as a result of investigations of expertise in many domains. In this paper, these three significant perspectives from cognitive psychology will be examined and their implications for the education of undergraduate nurses described. Developments in the field of nursing that reflect or challenge a cognitive outlook are also identified.

An appraisal of INTERNIST-I

INTERNIST-I was an expert system designed in the early 1970's to diagnose multiple diseases in internal medicine by modelling the behaviour of clinicians. Its form and operation are described, and evaluations of the system are surveyed. The major result of the project was its knowledge base which has been used in successor systems for medical education and clinical use. We also survey the effects of the project through these systems, and conclude that the most successful of them in the near future is likely to be Quick Medical Reference (QMR) when used as an "electronic textbook" of medicine.

Exploring the relationship between rationality and bounded rationality in medical knowledge-based systems

If our goal in Artificial Intelligence in Medicine (AIM) is to engineer systems health-care providers will both use and, in the process, improve their performance, we must concentrate on the development of causal theories of knowledge and problem solving. One broad direction in pursuing this goal is understanding the relationships between existing models of rationality and bounded rationality for similar tasks. Models of rationality refer to those approaches in which the optimal properties of the models are deductively provable, i.e. in which the processing is rational. Representative models of rationality used in AIM are deductive logical models, statistical models such as Bayesian inference models, and decision-analytic models. Models of bounded rationality are those which do not guarantee such optimal properties nor yield to deductive correctness proofs. These models have their roots in cognitive psychology. In this article we show how explicating the relationship between models of rationality and bounded rationality might be done in the case of abductive tasks in medicine. This is done by positioning these modeling approaches within the same framework (an abstract computational model) and interpreting in this context both computational complexity results concerning the nature of the task and empirical results studies of human problem-solving behavior.

Diagnostic reasoning of high- and low-domain-knowledge clinicians: a reanalysis

Thinking-aloud protocols provided by Joseph and Patel were reanalyzed to determine the extent to which their conclusions could be replicated by independently developed coding schemes. The data set consisted of protocols from four cardiologists (low domain knowledge = LDK) and four endocrinologists (high domain knowledge = HDK), individually working on a diagnostic problem in endocrinology. The two analyses agree that the HDK physicians related data to potential diagnoses more than did the LDK group and were more focused on the correct diagnostic components. However, the reanalysis found no meaningful difference between the groups in diagnostic accuracy, speed of diagnosis, or the breadth of the search space used to seek a solution. In the reanalysis, the HDK physicians employed more single-cue inference and less multiple-cue inference. The generalizability of results of protocol-analysis studies can be assessed by using several complementary coding schemes.

Heuristics in medical and non-medical decision-making

Pairs of hypothetical medical and non-medical problems were given to 44 pediatric residents at three levels of hospital training. Each problem was designed to detect a specific heuristic-based bias in making diagnoses. Discounting, disregarding base rate, and over-confidence in contextually embedded redundant information were more evident on medical than on non-medical problems. In particular, a greater number of third-year residents disregarded base-rate information than did first- and second-year residents on medical but not on non-medical problems. On medical problems, a greater number of first-year residents expressed greater confidence in redundant information that was contextually embedded than in information that was presented in a listed format. Over one-third of the residents confused prospective and retrospective probabilities; three-fourths showed evidence of augmentation; virtually all residents expressed greater confidence in a diagnosis based on redundant rather than on non-redundant listed information. These latter effects were consistent across training level and occurred on both medical and non-medical problems. The results are discussed in terms of prototype theory and the nature of medical training.

Disturbances of impulse formation: an expert system for ECG interpretation

This paper describes an expert system (ES) that aids in interpretation of some disturbances of impulse formation from electrocardiographic records. The system consists of a user interface, a knowledge base, an inference engine and an explanation facility. It is implemented using Turbo PROLOG and uses the built-in interpreter for goal proving or disproving. The user interface gets information about the case by interrogation through multiple choice or Yes/No questions. The response is processed and stored in a dynamic database. After the interview the processed data are stored in a permanent file for subsequent calls. The knowledge base contains domain rules of the If-Then variety. The inference engine supports the logic-based method of knowledge organization, which is controlled by backward-chaining. The explanation facility is able to give reasons for any fact in the dynamic database. The main diagnosis, the diagnostic criteria and the algorithm used are explained and illustrated with examples. Sample outputs of the system are also given.