Development and Validation of the Conceptual Assessment of Natural Selection (CANS)

The EvMed Assessment: A test for measuring student understanding of core concepts in evolutionary medicine

Background and objectives

Universities throughout the USA increasingly offer undergraduate courses in evolutionary medicine (EvMed), which creates a need for pedagogical resources. Several resources offer course content (e.g. textbooks) and a previous study identified EvMed core principles to help instructors set learning goals. However, assessment tools are not yet available. In this study, we address this need by developing an assessment that measures students' ability to apply EvMed core principles to various health-related scenarios.

Methodology

The EvMed Assessment (EMA) consists of questions containing a short description of a health-related scenario followed by several likely/unlikely items. We evaluated the assessment's validity and reliability using a variety of qualitative (expert reviews and student interviews) and quantitative (Cronbach's α and classical test theory) methods. We iteratively revised the assessment through several rounds of validation. We then administered the assessment to undergraduates in EvMed and Evolution courses at multiple institutions.

Results

We used results from the pilot to create the EMA final draft. After conducting quantitative validation, we deleted items that failed to meet performance criteria and revised items that exhibited borderline performance. The final version of the EMA consists of six core questions containing 25 items, and five supplemental questions containing 20 items.

Conclusions and implications

The EMA is a pedagogical tool supported by a wide range of validation evidence. Instructors can use it as a pre/post measure of student learning in an EvMed course to inform curriculum revision, or as a test bank to draw upon when developing in-class assessments, quizzes or exams.

Assessing how well students understand the molecular basis of evolution by natural selection

Researchers have called for undergraduate courses to update teaching frameworks based on the Modern Synthesis with insights from molecular biology, by stressing the molecular underpinnings of variation and adaptation. To support this goal, we developed a modified version of the widely used Assessing Conceptual Reasoning of Natural Selection (ACORNS) instrument. The expanded tool, called the E-ACORNS, is explicitly designed to test student understanding of the connections among genotypes, phenotypes, and fitness. E-ACORNS comprises a slight modification to the ACORNS open-response prompts and a new scoring rubric. The rubric is based on five core concepts in evolution by natural selection, with each concept broken into elements at the novice, intermediate, and expert-level understanding. Initial tests of the E-ACORNS showed that (1) upper-level undergraduates can score responses reliably and quickly, and (2) students who were just starting an introductory biology series for majors do not yet grasp the molecular basis of phenotypic variation and its connection to fitness.

Der Simulierte Klassenraum Biologie – Erfassung deklarativen und prozeduralen Wissens bei Lehramtsstudierenden der Biologie

In der Lehrerprofessionalisierungsforschung werden zahlreiche Studien durchgeführt, die sich intensiv mit verschiedenen Wissensbereichen – fachliches, fachdidaktisches und pädagogisches Wissen – beschäftigen. Vernachlässigt werden häufig spezifische Wissenstypen, die nach psychologischen Ansätzen in deklaratives und prozedurales Wissen differenziert werden. Vor diesem Hintergrund untersucht die vorliegende Studie die verschiedenen Bereiche des Professionswissens angehender Biologiestudierenden (N = 51) zur Evolution und versucht gleichzeitig, erste Erkenntnisse über den Zusammenhang zwischen dem deklarativen und prozeduralen Wissen zu gewinnen. Zu diesem Zweck wird eine Kombination zweier Instrumente verwendet - ein Fragebogen zur Evolution, der das deklarative Wissen misst - und der Simulierte Klassenraum Biologie (SKRBio), der das prozedurale Wissen im Bereich der Diagnosefähigkeit erfasst. Im SKRBio können die Lehramtsstudierenden Fragen zur natürlichen Selektion an virtuelle Schüler*innen richten, woraufhin die gegebenen Antworten und die gezeigten Leistungen diagnostiziert werden müssen. Die Ergebnisse im SKRBio zeigen, dass die Lehramtsstudierenden in der Lage sind, die wissenschaftliche Korrektheit der virtuellen Schülerantworten zu beurteilen (91% Diagnoserate). Größere Schwierigkeiten werden bei der Diagnose spezifischer Fehlvorstellungskategorien innerhalb der virtuellen Schülerantworten identifiziert (59% Diagnoserate). Zwischen dem deklarativen und dem prozeduralen Wissens zeigten sich schwache bis moderate Zusammenhänge. Erwartungswidrige Ergebnisse lieferten die differenzierten Zusammenhangsanalysen zwischen deklarativem und prozeduralem Wissen innerhalb eines Wissensbereichs. Diese Befunde zeigen, dass deklaratives und prozedurales Wissen angehender Lehrer*innen zusammenhängen. Weitere Replikationsstudien könnten dazu beitragen, die beobachteten Unterschiede in den Zusammenhangsgrößen innerhalb der Wissensbereiche zu erklären. In Zukunft könnte der SKRBio eine simulierte Klassenraumumgebung darstellen, um Lehramtsstudierenden neben den schulpraktischen Phasen die Möglichkeit zu geben, prozedurales Wissen anzuwenden und trainieren zu können.

Illuminating the complexities of conflict with evolution: validation of the scales of evolutionary conflict measure (SECM)

BACKGROUND

Although personal, familial, and community conflict with evolution have been documented in the literature, these scales require conceptualization as a construct and operationalization as a measure. The Scales of Conflict with Evolution Measure (SECM) instrument was developed in response to these needs. Using a construct validity framework, the content, internal structure, convergent, and substantive validity of the SECM were evaluated using Rasch analysis, Structural Equation Modeling (SEM), and follow up questioning. The conceptual utility of the instrument was explored by examining whether it added explanatory insights into evolution acceptance above and beyond religiosity, evolution knowledge, and background variables.

RESULTS

A literature review and expert consultation indicated that construct of evolutionary conflict perception should (i) encompass the hierarchical nature of human social structures (personal, family, community) and (ii) probe conflict as it relates to human values, cultures, and beliefs. A three-dimensional construct was operationalized as a nine-item rating scale measure. Using Rasch analyses of SECM responses from a diverse sample of > 1000 students studying evolution, the instrument met criteria of robust measurement, including: fit to model expectations; three-dimensional structure; high reliability; good rating scale function; measurement invariance with time; and convergence with a similar construct. SEM showed that: (i) family and community conflict had unique causal contributions to personal conflict, with family showing a stronger and modest impact, and (ii) personal conflict had a significant and modest causal impact on evolution acceptance above and beyond the contributions of religiosity, evolution knowledge, and background variables.

CONCLUSION

The SECM is an easy-to-administer instrument to measure conflict with evolution and is supported by several forms of validity evidence. The SECM has potential for facilitating measurement of evolutionary conflict in educational settings, thereby raising instructor awareness of conflict levels in students, promoting rigorous evaluations of educational interventions designed to reduce conflict, and fostering conceptual advances in the field of evolution education. Future work is needed to gather additional forms of validity evidence and to test current validity claims in additional participant samples. SECM measures should also be incorporated into more complex SEM models that treat evolution knowledge and religiosity as part of the structural paths to evolution acceptance. Such models could provide insights into the most worthwhile targets for the development of educational interventions to mitigate conflict at multiple scales.

Acceptance and knowledge of evolutionary theory among third-year university students in Spain

The theory of evolution is one of the greatest scientific achievements in the intellectual history of humankind, yet it is still contentious within certain social groups. Despite being as robust and evidence-based as any other notable scientific theory, some people show a strong reluctance to accept it. In this study, we used the Measure of Acceptance of the Theory of Evolution (MATE) and Knowledge of Evolution Exam (KEE) questionnaires with university students from four academic degree programs (Chemistry, English, History, and Biology) of ten universities from Spain to measure, respectively, acceptance and knowledge of evolutionary theory among third-year undergraduate students (nMATE = 978; nKEE = 981). Results show that acceptance of evolution is relatively high (87.2%), whereas knowledge of the theory is moderate (5.4 out of 10) although there are differences across degrees (Biology>Chemistry>History>English), and even among various universities (ranging from 4.71 to 5.81). Statistical analysis reveals that knowledge of evolutionary theory among Biology students is partially explained by the relative weight of evolutionary themes within the curriculum, suggesting that an increase in the number of hours dedicated to this topic could have a direct influence on students' knowledge of it. We also found that religion may have a significant-although relatively small-negative influence on evolutionary theory acceptance. The moderate knowledge of evolution in our undergraduate students, together with the potential problem of acceptance in certain groups, suggests the need for a revision of the evolutionary concepts in the teaching curricula of our students since primary school.

A Graphical Method for Displaying the Model Fit of Item Response Theory Trace Lines

Item response theory (IRT) is a statistical paradigm for developing educational tests and assessing students. IRT, however, currently lacks an established graphical method for examining model fit for the three-parameter logistic model, the most flexible and popular IRT model in educational testing. A method is presented here to do this. The graph, which is referred to herein as a "bin plot," is the IRT equivalent of a scatterplot for linear regression. Bin plots display a conventional IRT trace line (with ability on the horizontal axis and probability correct on the vertical axis). Students are binned according to how well they performed on the entire test, and the proportion of students in each bin who answered the focal question correctly is displayed on the graph as points above or below the trace line. With this arrangement, the difference between each point and the trace line is the residual for the bin. Confidence intervals can be added to the observed proportions in order to display uncertainty. Computer simulations were used to test four alternative ways for binning students. These simulations showed that binning students according to number of questions they answered correctly on the entire test works best. Simulations also showed confidence intervals for bin plots had coverage probabilities close to nominal values for common testing scenarios, but that there are scenarios in which confidence intervals had inflated error rates.

A new assessment to monitor student performance in introductory neurophysiology: Electrochemical Gradients Assessment Device

The basis for understanding neurophysiology is understanding ion movement across cell membranes. Students in introductory courses recognize ion concentration gradients as a driving force for ion movement but struggle to simultaneously account for electrical charge gradients. We developed a 17-multiple-choice item assessment of students' understanding of electrochemical gradients and resistance in neurophysiology, the Electrochemical Gradients Assessment Device (EGAD). We investigated the internal evidence validity of the assessment by analyzing item characteristic curves of score probability and student ability for each question, and a Wright map of student scores and ability. We used linear mixed-effect regression to test student performance and ability. Our assessment discriminated students with average ability (weighted likelihood estimate: -2 to 1.5 Θ); however, it was not as effective at discriminating students at the highest ability (weighted likelihood estimate: >2 Θ). We determined the assessment could capture changes in both assessment scores (model r² = 0.51, P < 0.001, n = 444) and ability estimates (model r² = 0.47, P < 0.001, n = 444) after a simulation-based laboratory and course instruction for 222 students. Differential item function analysis determined that each item on the assessment performed equitably for all students, regardless of gender, race/ethnicity, or economic status. Overall, we found that men scored higher (r² = 0.51, P = 0.014, n = 444) and had higher ability scores (P = 0.003) on the EGAD assessment. Caucasian students of both genders were positively correlated with score (r² = 0.51, P < 0.001, n = 444) and ability (r² = 0.47, P < 0.001, n = 444). Based on the evidence gathered through our analyses, the scores obtained from the EGAD can distinguish between levels of content knowledge on neurophysiology principles for students in introductory physiology courses.

GenBio-MAPS: A Programmatic Assessment to Measure Student Understanding of Vision and Change Core Concepts across General Biology Programs

The Vision and Change report provides a nationally agreed upon framework of core concepts that undergraduate biology students should master by graduation. While identifying these concepts was an important first step, departments also need ways to measure the extent to which students understand these concepts. Here, we present the General Biology-Measuring Achievement and Progression in Science (GenBio-MAPS) assessment as a tool to measure student understanding of the core concepts at key time points in a biology degree program. Data from more than 5000 students at 20 institutions reveal that this instrument distinguishes students at different stages of the curriculum, with an upward trend of increased performance at later time points. Despite this trend, we identify several concepts that advanced students find challenging. Linear mixed-effects models reveal that gender, race/ethnicity, English-language status, and first-generation status predict overall performance and that different institutions show distinct performance profiles across time points. GenBio-MAPS represents the first programmatic assessment for general biology programs that spans the breadth of biology and aligns with the Vision and Change core concepts. This instrument provides a needed tool to help departments monitor student learning and guide curricular transformation centered on the teaching of core concepts.

EcoEvo-MAPS: An Ecology and Evolution Assessment for Introductory through Advanced Undergraduates

A new assessment tool, Ecology and Evolution-Measuring Achievement and Progression in Science or EcoEvo-MAPS, measures student thinking in ecology and evolution during an undergraduate course of study. EcoEvo-MAPS targets foundational concepts in ecology and evolution and uses a novel approach that asks students to evaluate a series of predictions, conclusions, or interpretations as likely or unlikely to be true given a specific scenario. We collected evidence of validity and reliability for EcoEvo-MAPS through an iterative process of faculty review, student interviews, and analyses of assessment data from more than 3000 students at 34 associate's-, bachelor's-, master's-, and doctoral-granting institutions. The 63 likely/unlikely statements range in difficulty and target student understanding of key concepts aligned with the Vision and Change report. This assessment provides departments with a tool to measure student thinking at different time points in the curriculum and provides data that can be used to inform curricular and instructional modifications.

Moving Evolution Education Forward: A Systematic Analysis of Literature to Identify Gaps in Collective Knowledge for Teaching

Evolution is a unifying theory in biology and is challenging for undergraduates to learn. An instructor's ability to help students learn is influenced by pedagogical content knowledge (PCK), which is topic-specific knowledge of teaching and learning. Instructors need PCK for every topic they teach, which is a tremendous body of knowledge to develop alone. However, investigations of undergraduate thinking and learning have produced collective PCK that is available in peer-reviewed literature. Currently, it is unclear whether the collective PCK available adequately addresses the topics in evolution that college instructors teach. We systematically examined existing literature to determine what collective PCK for teaching evolution is available and what is missing. We conducted an exhaustive literature search and analyzed 316 relevant papers to determine: the evolutionary topics addressed; whether the focus was student thinking, assessment, instructional strategies, or goals; and the type of work (e.g., empirical, literature review). We compared the collective PCK available in the literature with the topics taught in a sample of 32 undergraduate evolution courses around the country. On the basis of our findings, we propose priorities for the evolution education research community and propose that PCK is a useful lens for guiding future research on teaching and learning biology.

Development and Validation of the Homeostasis Concept Inventory

We present the Homeostasis Concept Inventory (HCI), a 20-item multiple-choice instrument that assesses how well undergraduates understand this critical physiological concept. We used an iterative process to develop a set of questions based on elements in the Homeostasis Concept Framework. This process involved faculty experts and undergraduate students from associate's colleges, primarily undergraduate institutions, regional and research-intensive universities, and professional schools. Statistical results provided strong evidence for the validity and reliability of the HCI. We found that graduate students performed better than undergraduates, biology majors performed better than nonmajors, and students performed better after receiving instruction about homeostasis. We used differential item analysis to assess whether students from different genders, races/ethnicities, and English language status performed differently on individual items of the HCI. We found no evidence of differential item functioning, suggesting that the items do not incorporate cultural or gender biases that would impact students' performance on the test. Instructors can use the HCI to guide their teaching and student learning of homeostasis, a core concept of physiology.

Making evolution stick: using sticky notes to teach the mechanisms of evolutionary change

Evolution and its mechanisms of action are concepts that unite all aspects of biology, but remain some of the most difficult for students to understand. To address this challenge, we designed a hands-on activity that introduces fundamental mechanisms of evolutionary change: natural selection, genetic drift, and gene flow. In small groups, students use a population of sticky notes to reveal the consequences of each mechanism on phenotype frequency. In a followup homework assignment, students then explore how changes in phenotype frequency reflect changes in allele frequency in the population. This activity is suitable for anyone learning the basics of evolution, from high-school through the undergraduate level. We have provided detailed instructions, in-class worksheets, follow-up homework, and extensions that allow the activity to be simplified or made more complex as needed. In our own classrooms, we have observed that the concrete and collaborative nature of this activity enables students to deepen their understanding of the mechanisms through which evolution occurs. We have designed this study such that, in completing this activity, we hope to offer students the opportunity to confront potential misconceptions about evolution and gain a solid foundation for future explorations in the discipline.