A consumer-oriented model for evaluating computer-assisted instructional materials for medical education

Evaluating technology-enhanced learning: A comprehensive framework

BACKGROUND

The absence of a standard, comprehensive approach to evaluating technology-enhanced learning (TEL) limits the utility of individual evaluations, and impedes the integration and synthesis of results across studies.

PURPOSE

To outline a comprehensive framework for approaching TEL evaluation in medical education, and to develop instruments for measuring the perceptions of TEL learners and instructors.

METHODS AND RESULTS

Using both theoretical constructs of inquiry in education and a synthesis of existing models and instruments, we outlined a general model for evaluation that links utility, principles, and practices. From this we derived a framework for TEL evaluation that identifies seven data collection activities: needs analysis; documentation of processes, decisions, and final product; usability testing; observation of implementation; assessment of participant experience; assessment of learning outcomes; and evaluation of cost, reusability, and sustainability. We then used existing quality standards and approaches to develop instruments for assessing the experiences of learners and instructors using TEL.

CONCLUSIONS

No single evaluation is likely to collect all of this information, nor would any single audience likely find all information elements equally useful. However, consistent use of a common evaluation framework across different courses and institutions would avoid duplication of effort and allow cross-course comparisons.

Adoption of computer-assisted learning in medical education: the educators' perspective

CONTEXT

Computer-assisted learning (CAL) in medical education has been shown to be effective in the achievement of learning outcomes, but requires the input of significant resources and development time. This study examines the key elements and processes that led to the widespread adoption of a CAL program in undergraduate medical education, the Computer-assisted Learning in Paediatrics Program (CLIPP). It then considers the relative importance of elements drawn from existing theories and models for technology adoption and other studies on CAL in medical education to inform the future development, implementation and testing of CAL programs in medical education.

METHODS

The study used a mixed-methods explanatory design. All paediatric clerkship directors (CDs) using CLIPP were recruited to participate in a self-administered, online questionnaire. Semi-structured interviews were then conducted with a random sample of CDs to further explore the quantitative results.

RESULTS

Factors that facilitated adoption included CLIPP's ability to fill gaps in exposure to core clinical problems, the use of a national curriculum, development by CDs, and the meeting of CDs' desires to improve teaching and student learning. An additional facilitating factor was that little time and effort were needed to implement CLIPP within a clerkship. The quantitative findings were mostly corroborated by the qualitative findings.

CONCLUSIONS

This study indicates issues that are important in the consideration and future exploration of the development and implementation of CAL programs in medical education. The promise of CAL as a method of enhancing the process and outcomes of medical education, and its cost, increase the need for future CAL funders and developers to pay equal attention to the needs of potential adopters and the development process as they do to the content and tools in the CAL program. Important questions that remain on the optimal design, use and integration of CAL should be addressed in order to adequately inform future development. Support is needed for studies that address these critical areas.

Peer reviewing e-learning: opportunities, challenges, and solutions

Peer review is the foundation of academic publication and a necessary step in the scrutiny of any scholarly work. Simply defined, peer review is the attentive, unbiased assessment of any scholarly work that is submitted for formal scrutiny. Although medical school faculty increasingly use technology in clinical teaching, e-learning materials are often not subjected to a rigorous peer review process. The authors contrast peer review of e-learning materials with that of print materials, describe peer review issues regarding e-learning materials, propose approaches to address the challenges of peer review of e-learning materials, and outline directions for refinement of the e-learning peer review process. At its core, the peer review of e-learning materials should not differ substantially from that of traditional manuscripts. However, e-learning introduces new demands that impel reviewers to consider aspects that are unique to educational technology, including pedagogy, format, usability, navigation, interactivity, delivery, ease of updating, distribution, and access. Four approaches are offered to ease the burden and improve the quality of e-learning peer review: develop peer review training, embrace multidisciplinary peer review, develop guidelines, and provide incentives and compensation. The authors conclude with suggestions about peer review research.

COMPARE/Radiology, an interactive Web-based radiology teaching program evaluation of user response

RATIONALE AND OBJECTIVES

The aim of this study is to assess user benefits of COMPARE/Radiology, a highly interactive World Wide Web-based training program for radiology, as perceived by its users.

MATERIALS AND METHODS

COMPARE/Radiology (http://www.idr.med.uni-erlangen.de/compare.htm), an interactive training program based on 244 teaching cases, was created by the authors and made publicly available on the Internet. An anonymous survey was conducted among users to investigate the composition of the program's user base and assess the acceptance of the training program. In parallel, Web access data were collected and analyzed using descriptive statistics.

RESULTS

The group of responding users (n = 1370) consisted of 201 preclinical medical students (14.7%), 314 clinical medical students (22.9%), 359 residents in radiology (26.2%), and 205 users of other professions (14.9%). A majority of respondents (1230; 89%) rated the interactivity of COMPARE/Radiology as good or excellent. Many respondents use COMPARE/Radiology for self-study (971; 70%) and for teaching others (600; 43%). Web access statistics show an increase in number of site visits from 1248 in December 2002 to 4651 in April 2004.

CONCLUSION

Users appreciate the benefits of COMPARE/Radiology. The interactive instructional design was rated positively by responding users. The popularity of the site is growing, evidenced by the number of network accesses during the observation period.

COMPARE radiology: creating an interactive Web-based training program for radiology with multimedia authoring software

RATIONALE AND OBJECTIVES

Computer-based training has two primary benefits: Content can be presented interactively, and students can choose the time, place, and pace of learning. As a subject of medical education, radiology lends itself particularly well to computer-based training because of its highly visual content. To improve the efficiency of radiology training at their institution, the authors decided to create an interactive Web-based training site.

MATERIALS AND METHODS

Working with a group of medical students knowledgeable in multimedia authoring, the authors used authoring software to create "COMPARE Radiology," an interactive training program that follows the modality-based structure of the undergraduate curriculum for radiology at the University of Erlangen-Nuremberg, Erlangen, Germany, and at medical schools worldwide.

RESULTS

The Web-based program offers cases and exercises in radiographic anatomy at different selectable levels of difficulty, allowing users to test and build their knowledge of radiology. Pathologic images are initially presented without any further information. Additional information (patient history, laboratory results, reports from other imaging studies, and normal images for comparison) can be retrieved selectively and successively. Further information regarding the diagnosis and pathologic findings can be found by following links to external Web sites. The COMPARE Radiology program content is extended and updated regularly. The program is subject to internal peer review and can be evaluated by the user online.

CONCLUSION

The authors' experience shows that a highly interactive Web-based training program for radiology, tailored to the requirements of the target group, can be developed economically by a team of medical students using an advanced storing system, with the guidance of a radiologist and without the help of professionally trained computer experts.

Tutor versus computer: a prospective comparison of interactive tutorial and computer-assisted instruction in radiology education

RATIONALE AND OBJECTIVES

This study compared the educational effectiveness of an interactive tutorial with that of interactive computer-assisted instruction (CAI) and determined the effects of personal preference, learning style, and level of training.

MATERIALS AND METHODS

Fifty-four medical students and four radiology residents were prospectively, randomly assigned to receive instruction from different sections of an interactive tutorial and an interactive CAI module. Participants took tests of factual knowledge at the beginning and end of the instruction and a test of visual diagnosis at the end. They completed questionnaires to evaluate their preferred learning styles objectively and to elicit their subjective attitudes toward the two formats. Mean test scores of the tutorial and CAI groups were compared by means of analysis of covariance and two-tailed repeated-measures F test.

RESULTS

Both the tutorial and CAI groups demonstrated significant improvement in posttest scores (P < .01 and P < .01, respectively) with the tutorial group's mean posttest score marginally but significantly higher (32.84 vs 28.13, P < .001). There were no significant interaction effects with participants' year of training (P = .845), objectively evaluated preferred learning style (P = .312), subjectively elicited attitude toward learning with CAI (P = .703), or visual diagnosis score (tutorial, 7.61; CD-ROM, 7.75; P = .79).

CONCLUSION

Interactive tutorial and optimal CAI are both effective instructional formats. The tutorial was marginally but significantly more effective at teaching factual knowledge, an effect unrelated to students' year of training, learning style, or stated enjoyment of CAI. The superiority of the tutorial is expected to increase when it is compared with commercially expedient CAI modules.

A longitudinal evaluation of an educational software program: a case study of Urinalysis-Tutor

PURPOSE

To examine students' learning before and after revising an educational software program and to explore students' patterns of use of an interactive feature that compares images.

METHOD

Study participants were 466 University of Washington School of Medicine students. Two cohorts of students (one in 1996 and one in 1997) used the original version of the software. Following analysis of the students' learning, the software program was modified based on instructional design principles pertaining to visual learning and concept acquisition. A 1998 cohort of students used the revised program and their performance was compared with that of the 1996 cohort. Analyses were based on pre- and post-test scores, data collected from the observation of students, and navigational pathways tracked by the program.

RESULTS

There was very little difference in the overall performances of the students who used the original program and those who used the revised program. Error analysis focusing on 11 conceptual areas showed that reductions in errors occurred for six of 11 concepts, with statistically significant reductions of errors for two concepts. Additional navigational data collected in 1998 showed that students used an interactive feature for comparing images in different patterns. The data showed a positive association between performance and the anchored viewing mode of image display.

CONCLUSIONS

While this study cannot point to specific design components that facilitated or hindered learning, it demonstrated a potential benefit of linking usage-pattern data and performance. Future studies should evaluate design factors that affect usage patterns and performances based on navigational data collected while students interact with software programs.

Implementing and integrating computer-based activities into a problem-based gross anatomy curriculum

A problem-based learning curriculum in gross anatomy was begun for a limited number of students to address unsuccessful methodology inherent in a traditional instructional approach. To eliminate some concerns associated with the laboratory component, computer-based instruction and other computer- related activities were actively integrated into the total instructional process. Prosections, directions, quizzes, images, and grades were provided in lab at table-side computer workstations, in the library, and on the web. Results were assessed through questionnaires in which students rated their learning experience according to a Likert-type scale. Success was measured by quantitative improvements in student perception. In this three-year study, observations and measurements have suggested increasingly positive student attitudes toward educational technology, for networks as a faster and more effective method of student/faculty communication, and in the utilization of computer-based instruction for greater flexibility and efficiency in learning. This allowed a rethinking of the structure and content of the curriculum by the faculty, which permitted reduced laboratory time, more small-group activity, and less reliance on staff.

Effectiveness of a CD-ROM multimedia tutorial in transferring cognitive knowledge essential for laparoscopic skill training

BACKGROUND

Computer-assisted instruction (CAI) can benefit surgical education by improving efficiency, effectiveness, standardization, and access. This study compares knowledge gains for laparoscopic skill acquisition following a standardized tutorial delivered via CD-ROM versus live instructor.

METHODS

A standardized tutorial was written and subsequently converted to multimedia CD-ROM format by its author (JR). During a laparoscopic development course, experienced US-trained surgeons (n = 52) participated in the tutorial delivered live by the author. The CD-ROM tutorial replaced the instructor for the following groups: (1) experienced US-trained surgeons (n = 27); (2) US-trained surgical residents (n = 59); and (3) Greek surgeons (n = 63). A 51-item knowledge test was administered before and after tutorial instruction.

RESULTS

The mean increase in scores between pretest and posttest was significant (P <0.01) and of similar magnitude in each group, with nonsignificant posttest mean differences among US-trained groups.

CONCLUSIONS

The CD-ROM tutorial effectively transfers cognitive information necessary for skill development. Distance learning modes of this tutorial program may be feasible.

MedEthEx Online: a computer-based learning program in medical ethics and communication skills

BACKGROUND

Bioethics education often focuses on lectures and discussions to set a foundation for ethical decision making. Our goal was to bridge the gap between classroom learning and bedside competence through computer-assisted instruction. This article assesses the efficacy of MedEthEx Online, a computer-based learning program as part of a required Bioethics course.

DESCRIPTION

Of 173 American medical students, 89 in Section 1 attended 8 bioethics lectures and 8 small-group discussions. Eighty-four in Section 2 had a similar course, although two group discussions were replaced with computerized learning. We compared (a) final exam scores, (b) topic-specific question scores, (c) performance with standardized patients, (d) self-assessments, and (e) course evaluations.

EVALUATION

Exam scores were comparable, although computerized-learning students scored higher in specific exam areas, felt somewhat more clinically prepared, and rated the course slightly better. Standardized patient interactions differed, although they were comparable overall.

CONCLUSION

MedEthEx Online is a viable option for fostering effective communication and problem resolution skills.

Comparison of computer-assisted instruction and seminar instruction to acquire psychomotor and cognitive knowledge of epistaxis management

Epistaxis is a common problem faced by primary care physicians. Typically, first-hand experience with this problem is not obtained in medical school. A computer learning module was developed to address practical management of epistaxis. It was evaluated by use of third-year clerks and compared with an interactive seminar based on the same material. Fifty-eight students doing their required otolaryngology rotations were randomly separated into 3 groups: (1) doing a preinstruction test, (2) using the computer module, and (3) participating in a small-group seminar. All participants were tested with a short written test and a practical test that involved performing anterior nasal pack placement in a model patient. Percentage scores for the 2 groups were compared by use of t tests, and there was no significant difference between the written, practical, or combined scores at a level where P = 0.05. This study shows that basic patient management and a simple procedure can be taught as effectively with a computer module as with a small-group interactive seminar.

Online continuing dental education

BACKGROUND

An increasing number of continuing dental education, or CDE, courses are available on the Internet. The authors conducted this study to determine general characteristics of online CDE course offerings.

METHODS

The authors found online CDE courses through Internet search engines and dental indexes. They recorded each course's Web page address, title, topic, length, credit hours and cost. Then they classified course providers, categorized topics, compared course length with credit hours and calculated cost per credit hour.

RESULTS

The authors located 157 online CDE courses offered by 32 providers. The courses covered a wide range of topics, and most were five screens long or shorter. Credit hours per screen ranged from 0.05 to two, and cost per credit hour ranged from no charge to $25.

CONCLUSIONS

Online CDE courses are hard to locate, making this material accessible only to people who are well-versed in retrieving information on the Internet. The brevity of most courses may make them appropriate for incremental study at the dental practitioner's convenience. Guidelines to correlate credit hours with course length should be developed.

PRACTICE IMPLICATIONS

Online CDE courses may become an important tool to help practitioners keep current. Several issues, however, need to be addressed before online CDE can reach its full potential.

On-line interactive evaluation in course and clinical instruction

Student evaluations incorporated into feedback for instruction modification are an important aspect of any educational program. It is particularly important for healthcare programs in which teaching processes are continually dynamic. An interactive evaluation was designed and implemented on-line for students. Confidentiality was assured and privacy protected through a unique coding system. Our experience with the interactive, automated system proved particularly successful with respondents and faculty expectations have improved since its integration into course and clinical objectives.

Multimedia to teach urology to medical students

OBJECTIVES

To test the hypothesis that an interactive multimedia-based computer application may be used to teach urology to undergraduate medical students.

METHODS

Third-year medical students rotating on their urology clerkship were studied. Student knowledge was measured with a multiple choice test administered in a pretest-post-test experimental design. The educational intervention was a multimedia-based application that presented a clinical module on hematuria, using natural language-like entries. Student attitudes toward the multimedia application were assessed by a survey.

RESULTS

Twenty-three consecutive third-year medical students participated. Mean pretest and post-test scores +/- standard deviation were 35%+/-11% versus 74%+/-17%, respectively, P<0.0001. Student questionnaire responses indicated highly positive opinions that the multimedia-based module was easy to use, was fun, provided natural patient responses, had clear exercises, provided immediate feedback, was educational, and had a nonthreatening format, and that the multiple choice questions were clear and fair.

CONCLUSIONS

Multimedia-based education may be used to teach urology to undergraduate medical students.

The use of multimedia to teach new cardiac anesthesia procedures

Cardiac anesthesia is a complex field requiring extensive training and experience. Multimedia can be used effectively to teach new, complex, and rapidly developing anesthetic and surgical techniques. Minimally invasive Port-Access cardiac surgery is an area in which the computer can be used for educational purposes. As cardiac anesthesiologists become more adept at developing computerized instructional methods, and as new techniques in medicine emerge, greater use can be made of the computer as a valuable teaching tool.

AI in medical education--another grand challenge for medical informatics

The potential benefits of artificial intelligence in medicine (AIM) were never realized as anticipated. This paper addresses ways in which such potential can be achieved. Recent discussions of this topic have proposed a stronger integration between AIM applications and health information systems, and emphasize computer guidelines to support the new health care paradigms of evidence-based medicine and cost-effectiveness. These proposals, however, promote the initial definition of AIM applications as being AI systems that can perform or aid in diagnoses. We challenge this traditional philosophy of AIM and propose a new approach aiming at empowering health care workers to become independent self-sufficient problem solvers and decision makers. Our philosophy is based on findings from a review of empirical research that examines the relationship between the health care personnel's level of knowledge and skills, their job satisfaction, and the quality of the health care they provide. This review supports addressing the quality of health care by empowering health care workers to reach their full potential. As an aid in this empowerment process we argue for reviving a long forgotten AIM research area, namely, AI based applications for medical education and training. There is a growing body of research in artificial intelligence in education that demonstrates that the use of artificial intelligence can enhance learning in numerous domains. By examining the strengths of these educational applications and the results from previous AIM research we derive a framework for empowering medical personnel and consequently raising the quality of health care through the use of advanced AI based technology.

Systematic assessment of World Wide Web materials for medical education: online, cooperative peer review

OBJECTIVE

: To develop a generic methodology for the online assessment of medical education materials available on the World Wide Web and to implement it for pilot subject areas.

DESIGN

An online questionnaire was developed, based on an existing scheme for computer-based learning material. It was extended to involve five stages, covering general suitability, local suitability, the user interface, educational style, and a general review. It is available on the Web, so expert reviewers may be recruited from outside the home institution. The methodology was piloted in three subjects areas--clinical chemistry, radiology, and medical physics--concentrating initially on undergraduate teaching.

MEASUREMENTS

The contents of completed questionnaires were stored in an offline database. Selected fields, likely to be of use to students and educators searching for material, were input into an online database.

RESULTS

The online assessment was used successfully in clinical chemistry and medical physics but less well in radiology. Fewer resources were found to fit local needs than expected.

CONCLUSION

The methodology was found to work well for topics where teaching is highly structured and formal and is potentially applicable in other such disciplines. The approach produces more structured and applicable lists of resources than can be obtained from search engines.

Computer-based neuroanatomy laboratory for medical students

BACKGROUND

To present the laboratory portion of our first-semester Human Neuroanatomy course at Temple University Medical School more effectively and efficiently and to replace the glass slide/microscope-based laboratory component of the course, we developed a computer-based substitute.

METHODS

For this computer-based neuroanatomy laboratory program, we photographed the (a) gross brain sliced and dissected specimens and (b) all the glass slides, from the sacral cord through the head of the caudate nucleus. We digitized the photographed images and, using Multimedia ToolBook (Asymetrix), created a computerized atlas, laboratory guide, and a clinical problem-solving section. To assess the effectiveness of the computerized laboratory, we compared student performances between those classes that previously had the traditional laboratory with two succeeding classes that used the computer-based laboratory.

RESULTS

Test score results of the laboratory portion of the course suggested that performance on laboratory material was virtually unchanged by the substitution of the computer program. By a survey taken at the end of the course, the students were very satisfied with the computerized program as a teaching method.

CONCLUSIONS

The students and faculty enthusiastically agreed that the computer program was an effective substitute method for the traditional glass-slide laboratory and that it was a beneficial self-educational tool that fostered independent learning. The program encouraged student interaction and group learning and fostered independence. It was a more efficient method for faculty and students without sacrificing performance.