(Mis)use of scientific measurements in forensic science

A call for open science in forensics

The modern canon of open science consists of five "schools of thought" that justify unfettered access to the fruits of scientific research: i) public engagement, ii) democratic right of access, iii) efficiency of knowledge gain, iv) shared technology, and v) better assessment of impact. Here, we introduce a sixth school: due process. Due process under the law includes a right to "discovery" by a defendant of potentially exculpatory evidence held by the prosecution. When such evidence is scientific, due process becomes a Constitutional mandate for open science. To illustrate the significance of this new school, we present a case study from forensics, which centers on a federally funded investigation that reports summary statistics indicating that identification decisions made by forensic firearms examiners are highly accurate. Because of growing concern about validity of forensic methods, the larger scientific community called for public release of the complete analyzable dataset for independent audit and verification. Those in possession of the data opposed release for three years while summary statistics were used by prosecutors to gain admissibility of evidence in criminal trials. Those statistics paint an incomplete picture and hint at flaws in experimental design and analysis. Under the circumstances, withholding the underlying data in a criminal proceeding violates due process. Following the successful open-science model of drug validity testing through "clinical trials," which place strict requirements on experimental design and timing of data release, we argue for registered and open "forensic trials" to ensure transparency and accountability.

Unpacking workplace stress and forensic expert decision-making: From theory to practice

Workplace stress can affect forensic experts' job satisfaction and performance, which holds financial and other implications for forensic service providers. Therefore, it is important to understand and manage workplace stress, but that is not simple or straightforward. This paper explores stress as a human factor that influences forensic expert decision-making. First, we identify and highlight three factors that mitigate decisions under stress conditions: nature of decision, individual differences, and context of decision. Second, we situate workplace stress in forensic science within the Challenge-Hindrance Stressor Framework. We argue that stressors in forensic science workplaces can have a positive or a negative impact, depending on the type, level, and context of stress. Developing an understanding of the stressors, their sources, and their possible impact can help forensic service providers and researchers to implement context-specific interventions to manage stress at work and optimize expert performance.

Inconclusive decisions and error rates in forensic science

In recent years, there has been discussion and controversy relating to the treatment of inconclusive decisions in forensic feature comparison disciplines when considering the reliability of examination methods and results. In this article, we offer a brief review of the various viewpoints and suggestions that have been recently put forth, followed by a solution that we believe addresses the treatment of inconclusive decisions. We consider the issues in the context of method conformance and method performance as two distinct concepts, both of which are necessary for the determination of reliability. Method conformance relates to an assessment of whether the outcome of a method is the result of the analyst's adherence to the procedures that define the method. Method performance reflects the capacity of a method to discriminate between different propositions of interest (e.g., mated and non-mated comparisons). We then discuss implications of these issues for the forensic science community.

The false promise of firearms examination validation studies: Lay controls, simplistic comparisons, and the failure to soundly measure misidentification rates

Several studies have recently attempted to estimate practitioner accuracy when comparing fired ammunition. But whether this research has included sufficiently challenging comparisons dependent upon expertise for accurate conclusions regarding source remains largely unexplored in the literature. Control groups of lay people comprise one means of vetting this question, of assessing whether comparison samples were at least challenging enough to distinguish between experts and novices. This article therefore utilizes such a group, specifically 82 attorneys, as a post hoc control and juxtaposes their performance on a comparison set of cartridge case images from one commonly cited study (Duez et al. in J Forensic Sci. 2018;63:1069-1084) with that of the original participant pool of professionals. Despite lacking the kind of formalized training and experience common to the latter, our lay participants displayed an ability, generally, to distinguish between cartridge cases fired by the same versus different guns in the 327 comparisons they performed. And while their accuracy rates lagged substantially behind those of the original participant pool of professionals on same-source comparisons, their performance on different-source comparisons was essentially indistinguishable from that of trained examiners. This indicates that although the study we vetted may provide useful information about professional accuracy when performing same-source comparisons, it has little to offer in terms of measuring examiners' ability to distinguish between cartridge cases fired by different guns. If similar issues pervade other accuracy studies, then there is little reason to rely on the false-positive rates they have generated.

Two heads are better than one: Dual systems obtain better performance in facial comparison

Forensic facial image comparison based on recognition algorithms has been widely applied in forensic science. Previous researches have been concentrating on the cases of using single system during comparison, while how to use multiple systems has not yet been studied. In this paper, a dual-systems model (including SeetaFace and FaceNet) for facial comparison was constructed, and Bayesian networks were utilized as the basic frame. In order to prove its superiority, a large-scale experiment (on the dataset CelebA) has been carried on to evaluate the score-based likelihood ratio. We used three likelihood ratio evaluation tools (Empirical Cross-Entropy, Cost Likelihood Ratio, Limit Tippett Plots) to assess the performance of the model. The Wasserstein distance was also used to evaluate the detailed likelihood ratio performance. The experimental results show that the likelihood ratio performance of our dual-systems model is better than single system. Besides, our method of model building and evaluation can also be used in the condition of triple or more systems.

Measuring pelvises in 3D surface scans and in MDCT generated virtual environment: Considerations for applications in the forensic context

Virtual Anthropology (VA) transposes the traditional methods of physical anthropology to virtual environments using imaging techniques and exploits imaging technologies to devise new methodological protocols. In this research, we investigate whether the measurements used in the Diagnose Sexuelle Probabiliste (DSP) and Ischio-Pubic Index (IPI) differ significantly when 3D models of a bone are generated using 3D surface scans (3DSS) and Multidetector Computed Tomography (MDCT) scans. Thirty pelvises were selected from the SIMON identified skeletal collection. An equal ratio of females to males was sought, as well as a good preservation of the bones. The pelvises were scanned using an MDCT scanner and a 3D surface scanner. The measurements of the DSP and IPI methods on the dry bones (referred to as macroscopic measurements here), and then to the 3D models. The intra- and interobserver, using the Technical Error of Measurement (TEM) and relative Technical Error of Measurement (rTEM) error was assessed, and we aimed to observe if the measurements made on the MDCT and 3DSS generated models were significantly different from those taken on the dry bones. Additionally, the normality of the data was tested (Shapiro-Wilk test) and the differences in measurements was evaluated using parametric (Student t-tests) and non-parametric (Wilcoxon) tests. The TEM and rTEM calculations show high intra and interobserver consistency in general. However, some measurements present insufficient inter- and intraobserver agreement. Student t and Wilcoxon tests indicate potentially significant differences of some measurements between the different environments. The results show that especially in the virtual environment, it is not easy to find the right angle for some of the DSP measurements, However, when comparing the measurement differences between dry and virtual bones, the results show that most of the differences are less than or equal to 2.5 mm. Considering the IPI, the landmarks are already difficult to determine on the dry bone, but they are even more difficult to locate in the virtual environment. Nevertheless, this study shows that quantitative methods may be better suited for application in the virtual environment, but further research using different methods is needed.

The most consistent finding in forensic science is inconsistency

The most consistent finding in many forensic science domains is inconsistency (i.e., lack of reliability, reproducibility, repeatability, and replicability). The lack of consistency is a major problem, both from a scientific and a criminal justice point of view. Examining forensic conclusion data, from across many forensic domains, highlights the underlying cognitive issues and offers a better understanding of the issues and challenges. Such insights enable the development of ways to minimize these inconsistencies and move forward. The aim is to highlight the problem, so that it can be minimized and the reliability of forensic science evidence can be improved.

The secret life of crime labs

Houston TX experienced a widely known failure of its police forensic laboratory. This gave rise to the Houston Forensic Science Center (HFSC) as a separate entity to provide forensic services to the City of Houston. HFSC is a very large forensic laboratory and has made significant progress at remediating the past failures and improving public trust in forensic testing. HFSC has a large and robust blind testing program, which has provided many insights into the challenges forensic laboratories face. HFSC's journey from a notoriously failed lab to a model also gives perspective to the resource challenges faced by all labs in the country. Challenges for labs include the pervasive reality of poor-quality evidence. Also that forensic laboratories are necessarily part of a much wider system of interdependent functions in criminal justice making blind testing something in which all parts have a role. This interconnectedness also highlights the need for an array of oversight and regulatory frameworks to function properly. The major essential databases in forensics need to be a part of blind testing programs and work is needed to ensure that the results from these databases are indeed producing correct results and those results are being correctly used. Last, laboratory reports of "inconclusive" results are a significant challenge for laboratories and the system to better understand when these results are appropriate, necessary and most importantly correctly used by the rest of the system.

Scientific guidelines for evaluating the validity of forensic feature-comparison methods

When it comes to questions of fact in a legal context-particularly questions about measurement, association, and causality-courts should employ ordinary standards of applied science. Applied sciences generally develop along a path that proceeds from a basic scientific discovery about some natural process to the formation of a theory of how the process works and what causes it to fail, to the development of an invention intended to assess, repair, or improve the process, to the specification of predictions of the instrument's actions and, finally, empirical validation to determine that the instrument achieves the intended effect. These elements are salient and deeply embedded in the cultures of the applied sciences of medicine and engineering, both of which primarily grew from basic sciences. However, the inventions that underlie most forensic science disciplines have few roots in basic science, and they do not have sound theories to justify their predicted actions or results of empirical tests to prove that they work as advertised. Inspired by the "Bradford Hill Guidelines"-the dominant framework for causal inference in epidemiology-we set forth four guidelines that can be used to establish the validity of forensic comparison methods generally. This framework is not intended as a checklist establishing a threshold of minimum validity, as no magic formula determines when particular disciplines or hypotheses have passed a necessary threshold. We illustrate how these guidelines can be applied by considering the discipline of firearm and tool mark examination.

The scientific reinvention of forensic science

Forensic science is undergoing an evolution in which a long-standing "trust the examiner" focus is being replaced by a "trust the scientific method" focus. This shift, which is in progress and still partial, is critical to ensure that the legal system uses forensic information in an accurate and valid way. In this Perspective, we discuss the ways in which the move to a more empirically grounded scientific culture for the forensic sciences impacts testing, error rate analyses, procedural safeguards, and the reporting of forensic results. However, we caution that the ultimate success of this scientific reinvention likely depends on whether the courts begin to engage with forensic science claims in a more rigorous way.

Repeatability and reproducibility of comparison decisions by firearms examiners

In a comprehensive study to assess various aspects of the performance of qualified forensic firearms examiners, volunteer examiners compared both bullets and cartridge cases fired from three different types of firearms. They rendered opinions on each comparison according to the Association of Firearm & Tool Mark Examiners (AFTE) Range of Conclusions, as Identification, Inconclusive (A, B, or C), Elimination, or Unsuitable. In this part of the study, comparison sets used previously to characterize the overall accuracy of examiners were blindly resubmitted to examiners to assess the repeatability (105 examiners; 5700 comparisons of bullets and cartridge cases) and reproducibility (191 examiners of bullets, 193 of cartridge cases; 5790 comparisons) of firearms examinations. Data gathered using the prevailing AFTE Range were also recategorized into two hypothetical scoring systems. Consistently positive differences between observed agreement and expected agreement indicate that the repeatability and reproducibility of examiners exceed chance agreement. When averaged over bullets and cartridge cases, the repeatability of comparison decisions (involving all five levels of the AFTE Range) was 78.3% for known matches and 64.5% for known nonmatches. Similarly averaged reproducibility was 67.3%% for known matches and 36.5% for known nonmatches. For both repeatability and reproducibility, many of the observed disagreements were between a definitive and inconclusive category. Examiner decisions are reliable and trustworthy in the sense that identifications are unlikely when examiners are comparing non-matching items, and eliminations are unlikely when they are comparing matching items.

The inconclusive category, entropy, and forensic firearm identification

There has been extensive recent discussion of the difficulty in estimating meaningful error rates in forensic firearms examinations, and other areas of pattern evidence. The 2016 President's Council of Advisors on Science and Technology (PCAST) report was clear in criticizing many forensic disciplines as lacking the types of studies that would provide error rate measurements seen in other scientific fields. However, there is a substantial lack of consensus on the approach to measuring an "error rate" for fields such as forensic firearm examination that include in the conclusion scale the "inconclusive" category, as occurs in the Association of Firearm and Tool Mark Examiners (AFTE) Range of Conclusions and many other such fields. Many authors appear to assume the error rate calculated in the binary decision model is the only appropriate way to report errors, but there have been attempts made to adapt the error rate from the binary decision model to scientific fields in which the inconclusive category is viewed as a meaningful outcome of the examination process. In this study we present three neural networks of differing complexity and performance trained to classify the outlines of ejector marks on cartridge cases fired from different firearm models, as a model system for examining the performance of various metrics of error in systems using the inconclusive category. We also discuss an entropy, or information, based method to assess the similarity of classifications to ground truth that is applicable to range of conclusion scales, even when the inconclusive category is used.

A method for quantifying individual decision thresholds of latent print examiners

In recent years, 'black box' studies in forensic science have emerged as the preferred way to provide information about the overall validity of forensic disciplines in practice. These studies provide aggregated error rates over many examiners and comparisons, but errors are not equally likely on all comparisons. Furthermore, inconclusive responses are common and vary across examiners and comparisons, but do not fit neatly into the error rate framework. This work introduces Item Response Theory (IRT) and variants for the forensic setting to account for these two issues. In the IRT framework, participant proficiency and item difficulty are estimated directly from the responses, which accounts for the different subsets of items that participants often answer. By incorporating a decision-tree framework into the model, inconclusive responses are treated as a distinct cognitive process, which allows inter-examiner differences to be estimated directly. The IRT-based model achieves superior predictive performance over standard logistic regression techniques, produces item effects that are consistent with common sense and prior work, and demonstrates that most of the variability among fingerprint examiner decisions occurs at the latent print evaluation stage and as a result of differing tendencies to make inconclusive decisions.

Error rates and proficiency tests in the fingerprint domain: A matter of perspective and conceptualization

The purpose of this work is to critically analyse the aspects connected both with the measurement of error rates and with the design of proficiency tests and collaborative exercises in the fingerprint domain. All from the dual perspective of practitioners and organizers of PT's/CE's. A thorough analysis of the types of errors, of the methods to infer them through black-box studies and PT's/CE's is carried out, and the limits to the generalization of error rates are described, providing insightful indication on how to design PT's/CE's in the fingerprint domain, which are aimed to represent the complexity of casework.

Accuracy of comparison decisions by forensic firearms examiners

This black box study assessed the performance of forensic firearms examiners in the United States. It involved three different types of firearms and 173 volunteers who performed a total of 8640 comparisons of both bullets and cartridge cases. The overall false-positive error rate was estimated as 0.656% and 0.933% for bullets and cartridge cases, respectively, while the rate of false negatives was estimated as 2.87% and 1.87% for bullets and cartridge cases, respectively. The majority of errors were made by a limited number of examiners. Because chi-square tests of independence strongly suggest that error probabilities are not the same for each examiner, these are maximum-likelihood estimates based on the beta-binomial probability model and do not depend on an assumption of equal examiner-specific error rates. Corresponding 95% confidence intervals are (0.305%, 1.42%) and (0.548%, 1.57%) for false positives for bullets and cartridge cases, respectively, and (1.89%, 4.26%) and (1.16%, 2.99%) for false negatives for bullets and cartridge cases, respectively. The results of this study are consistent with prior studies, despite its comprehensive design and challenging specimens.

Registered reports in forensic science

Research assessing the validity and reliability of many forensic science disciplines has been published; however, the quality of this research varies depending on the methodologies employed. This was a major point of contention with the United States' President's Council of Advisors on Science and Technology, who recognized the existing literature but found the majority lacking because of methodological issues. Questionable scientific methodologies have undermined the forensic science community's ability to defend the scientific foundations and examination protocols used to examine evidence in criminal cases. Such scientific failures have significant legal implications. Registered reports, which strengthen the quality of scientific research and reliability of laboratory protocols, can provide transparency, validity and a stronger scientific foundation for forensic science.

How to make better forensic decisions

Much of forensic practice today involves human decisions about the origins of patterned sensory evidence, such as tool marks and fingerprints discovered at a crime scene. These decisions are made by trained observers who compare the evidential pattern to an exemplar pattern produced by the suspected source of the evidence. The decision consists of a determination as to whether the two patterns are similar enough to have come from the same source. Although forensic pattern comparison disciplines have for decades played a valued role in criminal investigation and prosecution, the extremely high personal and societal costs of failure-the conviction of innocent people-has elicited calls for caution and for the development of better practices. These calls have been heard by the scientific community involved in the study of human information processing, which has begun to offer much-needed perspectives on sensory measurement, discrimination, and classification in a forensic context. Here I draw from a well-established theoretical and empirical approach in sensory science to illustrate the vulnerabilities of contemporary pattern comparison disciplines and to suggest specific strategies for improvement.

Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners

This paper describes design and logistical aspects of a decision analysis study to assess the performance of qualified firearms examiners working in accredited laboratories in the United States in terms of accuracy (error rate), repeatability, and reproducibility of decisions involving comparisons of fired bullets and cartridge cases. The purpose of the study was to validate current practice of the forensic discipline of firearms/toolmarks (F/T) examination. It elicited error rate data by counting the number of false positive and false negative conclusions. Preceded by the experimental design, decisions, and logistics described herein, testing was ultimately administered 173 qualified, practicing F/T examiners in public and private crime laboratories. The first round of testing evaluated accuracy, while two subsequent rounds evaluated repeatability and reproducibility of examiner conclusions. This project expands on previous studies by involving many F/T examiners in challenging comparisons and by executing the study in the recommended double-blind format.

Forensic science and the principle of excluded middle: "Inconclusive" decisions and the structure of error rate studies

In a paper published recently in this journal, Dror and Scurich (2020) [20] critically discuss the notions of "inconclusive evidence" (i.e., test items for which it is difficult to render a categorical response) and "inconclusive decisions" (i.e., experts' conclusions or responses) in the context of forensic science error rate studies. They expose several ways in which the understanding and use of "inconclusives" in current forensic science research and practice can adversely affect the outcomes of error rate studies. A main cause of distortion, according to Dror and Scurich, is what they call "erroneous inconclusive" decisions, in particular the lack of acknowledgment of this type of erroneous conclusion in the computation of error rates. To overcome this complication, Dror and Scurich call for a more explicit monitoring of "inconclusives" using a modified error rate study design. Whilst we agree with several well-argued points raised by the authors, we disagree with their framing of "inconclusive decisions" as potential errors. In this paper, we argue that referring to an "inconclusive decision" as an error is a contradiction in terms, runs counter to an analysis based on decision logic and, hence, is questionable as a concept. We also reiterate that the very term "inconclusive decision" disregards the procedural architecture of the criminal justice system across modern jurisdictions, especially the fact that forensic experts have no decisional rights in the criminal process. These positions do not ignore the possibility that "inconclusives" - if used excessively - do raise problems in forensic expert reporting, in particular limited assertiveness (or, overcautiousness). However, these drawbacks derive from inherent limitations of experts rather than from the seemingly erroneous nature of "inconclusives" that needs to be fixed. More fundamentally, we argue that attempts to score "inconclusives" as errors amount to philosophical claims disguised as forensic methodology. Specifically, these attempts interfere with the metaphysical substrate underpinning empirical research. We point this out on the basis of the law of the excluded middle, i.e. the principle of "no third possibility being given" (tertium non datur).

Forensic science in Seychelles: An example of a micro-jurisdiction forensic delivery system

Forensic science has become an indispensable tool for even the smallest of jurisdictions. However, micro-jurisdictions often face significant challenges with respect to resource availability, administration and local governance. This paper examines the forensic service provision in Seychelles as an example of a micro-jurisdiction forensic delivery system. The impact of limited resources and remote access to consumables or services have prompted the prospective shift to localise commonly utilised forensic services. The potential for a solid foundation for a sustainable forensic service is examined in relation to jurisdictions with more advanced forensic service delivery. Reforms of the legal framework, administration, and governance structures are some of the key underpinnings for an effective forensic delivery system built on a culture of transparent science that promotes justice and creates public confidence.

•

The Seychelles as an example of a micro-jurisdiction forensic delivery system.

•

Geographically remote location brings challenges to sustainable service provision.

•

Current investment into capacity building of commonly utilised forensic services.

•

Innovative solutions required for effective and efficient forensic delivery system.

•

Transparent science culture needed to promote justice and create public confidence.

Translational Application of a Neuro-Scientific Multi-Modal Approach Into Forensic Psychiatric Evaluation: Why and How?

A prominent body of literature indicates that insanity evaluations, which are intended to provide influential expert reports for judges to reach a decision "beyond any reasonable doubt," suffer from a low inter-rater reliability. This paper reviews the limitations of the classical approach to insanity evaluation and the criticisms to the introduction of neuro-scientific approach in court. Here, we explain why in our opinion these criticisms, that seriously hamper the translational implementation of neuroscience into the forensic setting, do not survive scientific scrutiny. Moreover, we discuss how the neuro-scientific multimodal approach may improve the inter-rater reliability in insanity evaluation. Critically, neuroscience does not aim to introduce a brain-based concept of insanity. Indeed, criteria for responsibility and insanity are and should remain clinical. Rather, following the falsificationist approach and the convergence of evidence principle, the neuro-scientific multimodal approach is being proposed as a way to improve reliability of insanity evaluation and to mitigate the influence of cognitive biases on the formulation of insanity opinions, with the final aim to reduce errors and controversies.

Biasability and reliability of expert forensic document examiners

The performance of experts can be characterized in terms of biasability and reliability of their judgments. The current research is the first to explore the judgments of practicing forensic document experts, professionals who examine and compare disputed handwritten evidence to handwriting exemplars of individuals involved in criminal or civil litigation. Forensic handwriting experts determine if questioned and known handwritten items are of common authorship or written by different individuals, and present their findings in legal proceedings. The expert participants in our study (N=25) were not aware that they were part of a research study. Thirteen participants were led to believe that they were working on a case commissioned from the prosecution and the other twelve that it was for the defense. We did not find evidence in this study that this information biased their judgments, which may make sense since document examiners (in contrast to many other forensic domains) do not primarily work within an organizational forensic laboratory culture. The lack of bias in our findings may have been also due to the stimuli we used or/and the great variability in the judgments within each group, reflecting a lack of consistency in conclusions among examiners. A detailed discussion of our findings is presented along with the limitations that may have affected our results.