1
|
Leegwater AJ, Vergeer P, Alberink I, van der Ham LV, van de Wetering J, El Harchaoui R, Bosma W, Ypma RJF, Sjerps MJ. From data to a validated score-based LR system: A practitioner's guide. Forensic Sci Int 2024; 357:111994. [PMID: 38522325 DOI: 10.1016/j.forsciint.2024.111994] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2023] [Revised: 03/12/2024] [Accepted: 03/16/2024] [Indexed: 03/26/2024]
Abstract
Likelihood ratios (LRs) are a useful measure of evidential strength. In forensic casework consisting of a flow of cases with essentially the same question and the same analysis method, it is feasible to construct an 'LR system', that is, an automated procedure that has the observations as input and an LR as output. This paper is aimed at practitioners interested in building their own LR systems. It gives an overview of the different steps needed to get to a validated LR system from data. The paper is accompanied by a notebook that illustrates each step with an example using glass data. The notebook introduces open-source software in Python constructed by NFI (Netherlands Forensic Institute) data scientists and statisticians.
Collapse
Affiliation(s)
| | - Peter Vergeer
- Netherlands Forensic Institute, PO Box 24044, The Hague 2490 AA, the Netherlands
| | - Ivo Alberink
- Netherlands Forensic Institute, PO Box 24044, The Hague 2490 AA, the Netherlands
| | - Leen V van der Ham
- Netherlands Forensic Institute, PO Box 24044, The Hague 2490 AA, the Netherlands
| | | | - Rachid El Harchaoui
- Netherlands Forensic Institute, PO Box 24044, The Hague 2490 AA, the Netherlands
| | - Wauter Bosma
- Netherlands Forensic Institute, PO Box 24044, The Hague 2490 AA, the Netherlands
| | - Rolf J F Ypma
- Netherlands Forensic Institute, PO Box 24044, The Hague 2490 AA, the Netherlands
| | - Marjan J Sjerps
- Netherlands Forensic Institute, PO Box 24044, The Hague 2490 AA, the Netherlands; Korteweg-de Vries Institute for mathematics, University of Amsterdam, PO Box 94248, Amsterdam 1090 GE, the Netherlands.
| |
Collapse
|
2
|
Taylor D, Kokshoorn B, Champod C. A practical treatment of sensitivity analyses in activity level evaluations. Forensic Sci Int 2024; 355:111944. [PMID: 38277913 DOI: 10.1016/j.forsciint.2024.111944] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Revised: 01/09/2024] [Accepted: 01/15/2024] [Indexed: 01/28/2024]
Abstract
Evaluations of forensic observations considering activity level propositions are becoming more common place in forensic institutions. A measure that can be taken to interrogate the evaluation for robustness is called sensitivity analysis. A sensitivity analysis explores the sensitivity of the evaluation to the data used when assigning probabilities, or to the level of uncertainty surrounding a probability assignment, or to the choice of various assumptions within the model. There have been a number of publications that describe sensitivity analysis in technical terms, and demonstrate their use, but limited literature on how that theory can be applied in practice. In this work we provide some simplified examples of how sensitivity analyses can be carried out, when they are likely to show that the evaluation is sensitive to underlying data, knowledge or assumptions, how to interpret the results of sensitivity analysis, and how the outcome can be reported. We also provide access to an application to conduct sensitivity analysis.
Collapse
Affiliation(s)
- Duncan Taylor
- Forensic Science SA, GPO Box 2790, Adelaide, SA 5001, Australia; School of Biological Sciences, Flinders University, GPO Box 2100, Adelaide, SA 5001, Australia.
| | - Bas Kokshoorn
- Netherlands Forensic Institute, P.O.Box 24044, 2490 AA The Hague, the Netherlands; Forensic Trace Dynamics, Faculty of Technology, Amsterdam University of Applied Sciences, Amsterdam, the Netherlands
| | - Christophe Champod
- Faculty of Law, Criminal Justice and Public Administration, School of Criminal Justice, University of Lausanne, Lausanne-Dorigny, Switzerland
| |
Collapse
|
3
|
Baiker-Sørensen M, Alberink I, Granell LB, van der Ham L, Mattijssen EJAT, Smith ED, Soons J, Vergeer P, Zheng XA. Automated interpretation of comparison scores for firearm toolmarks on cartridge case primers. Forensic Sci Int 2023; 353:111858. [PMID: 37863005 DOI: 10.1016/j.forsciint.2023.111858] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Revised: 09/04/2023] [Accepted: 10/10/2023] [Indexed: 10/22/2023]
Abstract
An automated approach for evaluating the strength of the evidence of firearm toolmark comparison results is presented for a common source scenario. First, comparison scores are derived describing the similarity of marks typically encountered on the primer of fired cartridge cases: aperture shear striations as well as breechface and firing pin impressions. Subsequently, these scores are interpreted using reference distributions of comparison scores obtained for representative known matching (KM) and known non-matching (KNM) ballistic samples in a common source, score-based likelihood ratio (LR) system. We study various alternatives to set up such an LR system and compare them using qualitative and quantitative criteria known from the literature. As an example, results are applied to establish a system suitable for a firearm-ammunition combination often encountered in casework: Glock firearms with Fiocchi nickel primer ammunition. The system outputs an LR and a measure of LR uncertainty. The range of possible LR-values is limited to a minimum and maximum value in areas of the score domain with little reference data. Finally, the feasibility of combining LRs of different mark types into one LR for the entire primer is assessed. For the distribution models considered in this paper, different modeling approaches are optimal for different types of similarity scores. For the chosen firearm-ammunition combination, non-parametric Kernel Density Estimation (KDE) models perform best for similarity scores based on the correlation coefficient, whereas parametric models perform best for the Congruent Matching Cells (CMC) scores, assuming binomial and beta-binomial models for KM and KNM score distributions respectively. Finally, it is demonstrated that individual LRs of different mark types can be combined into one LR, to interpret a set of different marks on the primer as a whole.
Collapse
Affiliation(s)
| | - Ivo Alberink
- Netherlands Forensic Institute, Laan van Ypenburg 6, 2497GB Den Haag, the Netherlands
| | - Laura B Granell
- Federal Bureau of Investigation, 2500 Investigation Parkway, Quantico, VA 22134, USA
| | - Leen van der Ham
- Netherlands Forensic Institute, Laan van Ypenburg 6, 2497GB Den Haag, the Netherlands
| | | | - Erich D Smith
- Federal Bureau of Investigation, 2500 Investigation Parkway, Quantico, VA 22134, USA
| | - Johannes Soons
- National Institute of Standards and Technology, 100 Bureau Drive, Gaithersburg, MD 20899, USA
| | - Peter Vergeer
- Netherlands Forensic Institute, Laan van Ypenburg 6, 2497GB Den Haag, the Netherlands
| | - Xiaoyu A Zheng
- National Institute of Standards and Technology, 100 Bureau Drive, Gaithersburg, MD 20899, USA
| |
Collapse
|
4
|
Sjerps M, Alberink I, Visser R, Stoel RD. The evidential strength of a combination of corresponding class features in tire examination. Forensic Sci Int 2022; 337:111351. [DOI: 10.1016/j.forsciint.2022.111351] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2022] [Revised: 05/20/2022] [Accepted: 05/23/2022] [Indexed: 11/04/2022]
|
5
|
Gill P, Benschop C, Buckleton J, Bleka Ø, Taylor D. A Review of Probabilistic Genotyping Systems: EuroForMix, DNAStatistX and STRmix™. Genes (Basel) 2021; 12:1559. [PMID: 34680954 PMCID: PMC8535381 DOI: 10.3390/genes12101559] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2021] [Revised: 09/24/2021] [Accepted: 09/28/2021] [Indexed: 11/24/2022] Open
Abstract
Probabilistic genotyping has become widespread. EuroForMix and DNAStatistX are both based upon maximum likelihood estimation using a γ model, whereas STRmix™ is a Bayesian approach that specifies prior distributions on the unknown model parameters. A general overview is provided of the historical development of probabilistic genotyping. Some general principles of interpretation are described, including: the application to investigative vs. evaluative reporting; detection of contamination events; inter and intra laboratory studies; numbers of contributors; proposition setting and validation of software and its performance. This is followed by details of the evolution, utility, practice and adoption of the software discussed.
Collapse
Affiliation(s)
- Peter Gill
- Forensic Genetics Research Group, Department of Forensic Sciences, Oslo University Hospital, 0372 Oslo, Norway;
- Department of Forensic Medicine, Institute of Clinical Medicine, University of Oslo, 0315 Oslo, Norway
| | - Corina Benschop
- Division of Biological Traces, Netherlands Forensic Institute, P.O. Box 24044, 2490 AA The Hague, The Netherlands;
| | - John Buckleton
- Department of Statistics, University of Auckland, Private Bag 92019, Auckland 1142, New Zealand;
- Institute of Environmental Science and Research Limited, Private Bag 92021, Auckland 1142, New Zealand
| | - Øyvind Bleka
- Forensic Genetics Research Group, Department of Forensic Sciences, Oslo University Hospital, 0372 Oslo, Norway;
| | - Duncan Taylor
- Forensic Science SA, GPO Box 2790, Adelaide, SA 5001, Australia;
- School of Biological Sciences, Flinders University, GPO Box 2100, Adelaide, SA 5001, Australia
| |
Collapse
|
6
|
Affiliation(s)
- Danica M. Ommen
- Danica M. Ommen is Assistant Professor, Department of Statistics, Iowa State University, Ames, Iowa 50011, USA
| | - Christopher P. Saunders
- Christopher P. Saunders is Associate Professor, Department of Mathematics & Statistics, South Dakota State University, Brookings, South Dakota 57007, USA
| |
Collapse
|
7
|
Taylor D, Balding D. How can courts take into account the uncertainty in a likelihood ratio? Forensic Sci Int Genet 2020; 48:102361. [PMID: 32769057 DOI: 10.1016/j.fsigen.2020.102361] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2020] [Revised: 05/17/2020] [Accepted: 07/22/2020] [Indexed: 11/19/2022]
Abstract
As legal practitioners and courts become more aware of scientific methods and evidence evaluation, they are demanding measures of the reliability of expert opinion. In particular, there are calls for error rates to accompany opinion evidence in comparative forensic sciences. While error rates or confidence intervals can be useful for those disciplines that claim to identify the source of a trace, the call for these statistical tools has extended to sciences that present opinions in the form of a likelihood ratio. In this article we argue against presenting both a likelihood ratio and numerical measures of its uncertainty. We explain how the LR already encapsulates uncertainty. Instead we consider how sensitivity analyses can be used to guide the presentation of LRs that are informative to the court and not unfair to defendants.
Collapse
Affiliation(s)
- Duncan Taylor
- School of Biological Sciences, Flinders University, GPO Box 2100 Adelaide, SA, 5001, Australia; Forensic Science SA, PO Box 2790, Adelaide, SA, 5000, Australia.
| | - David Balding
- Melbourne Integrative Genomics, School of BioSciences and School of Mathematics & Statistics, University of Melbourne, Australia
| |
Collapse
|
8
|
Abstract
We comment on the contributions of Dahlman and of Fenton et al., who both suggested a Bayesian approach to analyze the Simonshaven case. We argue that analyzing a full case with a Bayesian approach is not feasible, and that there are serious problems with assigning actual numbers to probabilities and priors. We also discuss the nature of Bayesian thinking in court, and the nature and interpretation of the likelihood ratio. In particular, we discuss what it could mean that a likelihood ratio is in some sense uncertain.
Collapse
|
9
|
Rosas C, Sommerhoff J, Morrison GS. A method for calculating the strength of evidence associated with an earwitness's claimed recognition of a familiar speaker. Sci Justice 2019; 59:585-96. [PMID: 31606096 DOI: 10.1016/j.scijus.2019.07.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/24/2019] [Revised: 07/01/2019] [Accepted: 07/06/2019] [Indexed: 11/22/2022]
Abstract
The present paper proposes and demonstrates a method for assessing strength of evidence when an earwitness claims to recognize the voice of a speaker who is familiar to them. The method calculates a Bayes factor that answers the question: What is the probability that the earwitness would claim to recognize the offender as the suspect if the offender was the suspect versus what is the probability that the earwitness would claim to recognize the offender as the suspect if the offender was not the suspect but some other speaker from the relevant population? By "claim" we mean a claim made by a cooperative earwitness not a claim made by an earwitness who is intentionally deceptive. Relevant data are derived from naïve listeners' responses to recordings of familiar speakers presented in a speaker lineup. The method is demonstrated under recording conditions that broadly reflect those of a real case.
Collapse
|
10
|
Taylor D, Kokshoorn B, Biedermann A. Evaluation of forensic genetics findings given activity level propositions: A review. Forensic Sci Int Genet 2018; 36:34-49. [DOI: 10.1016/j.fsigen.2018.06.001] [Citation(s) in RCA: 67] [Impact Index Per Article: 11.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2018] [Revised: 05/31/2018] [Accepted: 06/01/2018] [Indexed: 12/31/2022]
|
11
|
Affiliation(s)
- Danica M. Ommen
- Department of Mathematics and Statistics, South Dakota State University, Brookings, SD, USA
| | | | - Cedric Neumann
- Department of Mathematics and Statistics, South Dakota State University, Brookings, SD, USA
| |
Collapse
|
12
|
Abstract
This article is a response to the position papers published in the Science & Justice virtual special issue on measuring and reporting the precision of forensic likelihood ratios. I point out a number of serious statistical errors in some of these papers. These issues need to be properly addressed before the philosophical debate can be conducted in earnest.
Collapse
Affiliation(s)
- A Philip Dawid
- Statistical Laboratory, University of Cambridge, Wilberforce Road, Cambridge CB3 0WB, UK.
| |
Collapse
|