From data to a validated score-based LR system: A practitioner's guide

Likelihood ratios (LRs) are a useful measure of evidential strength. In forensic casework consisting of a flow of cases with essentially the same question and the same analysis method, it is feasible to construct an 'LR system', that is, an automated procedure that has the observations as input and an LR as output. This paper is aimed at practitioners interested in building their own LR systems. It gives an overview of the different steps needed to get to a validated LR system from data. The paper is accompanied by a notebook that illustrates each step with an example using glass data. The notebook introduces open-source software in Python constructed by NFI (Netherlands Forensic Institute) data scientists and statisticians.

Automated interpretation of comparison scores for firearm toolmarks on cartridge case primers

An automated approach for evaluating the strength of the evidence of firearm toolmark comparison results is presented for a common source scenario. First, comparison scores are derived describing the similarity of marks typically encountered on the primer of fired cartridge cases: aperture shear striations as well as breechface and firing pin impressions. Subsequently, these scores are interpreted using reference distributions of comparison scores obtained for representative known matching (KM) and known non-matching (KNM) ballistic samples in a common source, score-based likelihood ratio (LR) system. We study various alternatives to set up such an LR system and compare them using qualitative and quantitative criteria known from the literature. As an example, results are applied to establish a system suitable for a firearm-ammunition combination often encountered in casework: Glock firearms with Fiocchi nickel primer ammunition. The system outputs an LR and a measure of LR uncertainty. The range of possible LR-values is limited to a minimum and maximum value in areas of the score domain with little reference data. Finally, the feasibility of combining LRs of different mark types into one LR for the entire primer is assessed. For the distribution models considered in this paper, different modeling approaches are optimal for different types of similarity scores. For the chosen firearm-ammunition combination, non-parametric Kernel Density Estimation (KDE) models perform best for similarity scores based on the correlation coefficient, whereas parametric models perform best for the Congruent Matching Cells (CMC) scores, assuming binomial and beta-binomial models for KM and KNM score distributions respectively. Finally, it is demonstrated that individual LRs of different mark types can be combined into one LR, to interpret a set of different marks on the primer as a whole.

From facial images of different quality to score based LR

A simple method is proposed to assess the quality of a trace facial image in the context of the facial recognition system used using the similarity scores with low quality different-source facial images, defined as the Confusion Score (CS). Methods are proposed to calculate the probability of finding the correct facial image in a database using low quality images for investigational purposes using the CS, as well as calculation of the Likelihood Ratio (LR) for comparison of low quality trace facial images with good quality reference facial images, based on the assessed CS of the trace image. Improvement of performance of an LR-system using training datasets stratified on CS over the use of pooled data is demonstrated. Examples of using the proposed approach in simulated case examples are presented.

Measuring calibration of likelihood-ratio systems: A comparison of four metrics, including a new metric devPAV

Numerical likelihood-ratio (LR) systems aim to calculate evidential strength for forensic evidence evaluation. Calibration of such LR-systems is essential: one does not want to over- or understate the strength of the evidence. Metrics that measure calibration differ in sensitivity to errors in calibration of such systems. In this paper we compare four calibration metrics by a simulation study based on Gaussian Log LR-distributions. Three calibration metrics are taken from the literature (Good, 1985; Royall, 1997; Ramos and Gonzalez-Rodriguez, 2013) [1-3], and a fourth metric is proposed by us. We evaluated these metrics by two performance criteria: differentiation (between well- and ill-calibrated LR-systems) and stability (of the value of the metric for a variety of well-calibrated LR-systems). Two metrics from the literature (the expected values of LR and of 1/LR, and the rate of misleading evidence stronger than 2) do not behave as desired in many simulated conditions. The third one (C_llr^cal) performs better, but our newly proposed method (which we coin devPAV) is shown to behave equally well to clearly better under almost all simulated conditions. On the basis of this work, we recommend to use both devPAV and C_llr^cal to measure calibration of LR-systems, where the current results indicate that devPAV is the preferred metric. In the future external validity of this comparison study can be extended by simulating non-Gaussian LR-distributions.

Artificial intelligence and thermodynamics help solving arson cases

In arson cases, evidence such as DNA or fingerprints is often destroyed. One of the most important evidence modalities left is relating fire accelerants to a suspect. When gasoline is used as accelerant, the aim is to find a strong indication that a gasoline sample from a fire scene is related to a sample of a suspect. Gasoline samples from a fire scene are weathered, which prohibits a straightforward comparison. We combine machine learning, thermodynamic modeling, and quantum mechanics to predict the composition of unweathered gasoline samples starting from weathered ones. Our approach predicts the initial (unweathered) composition of the sixty main components in a weathered gasoline sample, with error bars of ca. 4% when weathered up to 80% w/w. This shows that machine learning is a valuable tool for predicting the initial composition of a weathered gasoline, and thereby relating samples to suspects.

Evaluation of glass evidence at activity level: A new distribution for the background population

For evidence evaluation of the physicochemical properties of glass at activity level a well-known formula introduced by Evett & Buckleton [1,2] is commonly used. Parameters in this formula are, amongst others, the probability in a background population to find on somebody's clothing the observed number of glass sources and the probability in a background population to find on somebody's clothing a group of fragments with the same size as the observed matching group. Recently, for efficiency reasons, the Netherlands Forensic Institute changed its methodology to measure not all the glass fragments but a subset of glass fragments found on clothing. Due to the measurement of subsets, it is difficult to get accurate estimates for these parameters in this formula. We offer a solution to this problem. The heart of the solution consists of relaxing the assumption of conditional independence of group sizes of background fragments, and modelling the probability of an allocation of background fragments into groups given a total number of background fragments by a two-parameter Chinese restaurant process (CRP) [3]. Under the assumption of random sampling of fragments to be measured from recovered fragments in the laboratory, parameter values for the Chinese restaurant process may be estimated from a relatively small dataset of glass in other relevant cases. We demonstrate this for a dataset of glass fragments collected from upper garments in casework, show model fit and provide a prototypical calculation of an LR at activity level accompanied with a parameter sensitivity analysis for reasonable ranges of the CRP parameter values. Considering that other laboratories may want to measure subsets as well, we believe this is an important alternative approach to the evaluation of numerical LRs for glass analyses at activity level.

Combining evidence in complex cases - a practical approach to interdisciplinary casework

Activity level evaluations, although still a major challenge for many disciplines, bring a wealth of possibilities for a more formal approach to the evaluation of interdisciplinary forensic evidence. This paper proposes a practical methodology for combining evidence from different disciplines within the likelihood ratio framework. Evidence schemes introduced in this paper make the process of combining evidence more insightful and intuitive thereby assisting experts in their interdisciplinairy evaluation and in explaining this process to the courts. When confronted with two opposing scenarios and multiple types of evidence, the likelihood ratio approach allows experts to combine this evidence in a probabilistic manner. Parts of the prosecution and defence scenarios for which forensic science is expected to be informative are identified. For these so called core elements, activity level propositions are formulated. Afterwards evidence schemes are introduced to assist the expert in combining the evidence in a logical manner. Two types of evidence relations are identified: serial and parallel evidence. Practical guidelines are given on how to deal with both types of evidence relations when combining the evidence.

Decision support for using mobile Rapid DNA analysis at the crime scene

Mobile Rapid DNA technology is close to being incorporated into crime scene investigations, with the potential to identify a perpetrator within hours. However, the use of these techniques entails the risk of losing the sample and potential evidence, because the device not only consumes the inserted sample, it is also is less sensitive than traditional technologies used in forensic laboratories. Scene of Crime Officers (SoCOs) therefore will face a 'time/success rate trade-off' issue when making a decision to apply this technology. In this study we designed and experimentally tested a Decision Support System (DSS) for the use of Rapid DNA technologies based on Rational Decision Theory (RDT). In a vignette study, where SoCOs had to decide on the use of a Rapid DNA analysis device, participating SoCOs were assigned to either the control group (making decisions under standard conditions), the Success Rate (SR) group (making decisions with additional information on DNA success rates of traces), or the DSS group (making decisions supported by introduction to RDT, including information on DNA success rates of traces). This study provides positive evidence that a systematic approach for decision-making on using Rapid DNA analysis assists SoCOs in the decision to use the rapid device. The results demonstrated that participants using a DSS made different and more transparent decisions on the use of Rapid DNA analysis when different case characteristics were explicitly considered. In the DSS group the decision to apply Rapid DNA analysis was influenced by the factors "time pressure" and "trace characteristics" like DNA success rates. In the SR group, the decisions depended solely on the trace characteristics and in the control group the decisions did not show any systematic differences on crime type or trace characteristic. Guiding complex decisions on the use of Rapid DNA analyses with a DSS could be an important step towards the use of these devices at the crime scene.

Performance Study of a Score-based Likelihood Ratio System for Forensic Fingermark Comparison

In this article, the performance of a score-based likelihood ratio (LR) system for comparisons of fingerprints with fingermarks is studied. The system is based on an automated fingerprint identification system (AFIS) comparison algorithm and focuses on fingerprint comparisons where the fingermarks contain 6-11 minutiae. The hypotheses under consideration are evaluated at the level of the person, not the finger. The LRs are presented with bootstrap intervals indicating the sampling uncertainty involved. Several aspects of the performance are measured: leave-one-out cross-validation is applied, and rates of misleading evidence are studied in two ways. A simulation study is performed to study the coverage of the bootstrap intervals. The results indicate that the evidential strength for same source comparisons that do not meet the Dutch twelve-point standard may be substantial. The methods used can be generalized to measure the performance of score-based LR systems in other fields of forensic science.

Quantifying Uncertainty in Estimations of the Total Weight of Drugs in Groups of Complex Matrices: Using the Welch-Satterthwaite Equation

In this paper, a method is described to quantify estimations of the total amount of drugs in groups of seized items, including quantification of the precision. Previous work on this topic was based on the assumptions of normally distributed measurements and grouping of items with a common relative standard deviation. In practice, these assumptions are often violated, for example, for data with point masses at 0, or if certain items in a group have a very high standard deviation. The method described in this paper is based on work by Welch and Satterthwaite and does not assume constant relative standard deviations. Case examples are described for which the method is applied, and simulation studies are carried out for which both methods are applied. In the cases, both methods perform reasonably well. If the assumption of common relative standard deviations clearly does not apply, it is advised to use the method described.

A more straightforward derivation of the LR for a database search

Matching DNA profiles of an accused person and a crime scene trace are one of the most common forms of forensic evidence. A number of years ago the so-called 'DNA controversy' was concerned with how to quantify the value of such evidence. Given its importance, the lack of understanding of such a basic issue was quite surprising and concerning. Deriving the equation for the likelihood ratio of a DNA database match in a much more direct and simple way is the topic of this paper. As it is much easier to follow it is hoped that this derivation will contribute to the understanding.

Development and validation of the 48-item Symptom Questionnaire (SQ-48) in patients with depressive, anxiety and somatoform disorders

Self-report measures of psychological distress or psychopathology are widely used and can be easily implemented as psychiatric screening tools. Positive psychological constructs such as vitality/optimism and work functioning have scarcely been incorporated. We aimed to develop and validate a psychological distress instrument, including measures of vitality and work functioning. A patient sample with suspected depressive, anxiety, and somatoform disorders (N=242) and a reference sample of the general population (N=516) filled in the 48-item Symptom Questionnaire (SQ-48) plus a battery of observer-rated and self-report scales (MINI Plus, MADR, BAS, INH, BSI), using a web-based ROM programme. The resulting SQ-48 is multidimensional and includes the following nine subscales: Depression (MOOD, six items), Anxiety (ANXI, six items), Somatization (SOMA, seven items), Agoraphobia (AGOR, four items), Aggression (AGGR, four items), Cognitive problems (COGN, five items), Social Phobia (SOPH, five items), Work functioning (WORK, five items), and Vitality (VITA, six items). The results showed good internal consistency as well as good convergent and divergent validity. The SQ-48 is meant to be available in the public domain for Routine Outcome Monitoring (ROM) and can be used as a screening/ monitoring tool in clinical settings (psychiatric and non-psychiatric), as a benchmark tool, or for research purposes.

Single-family versus multi-family introductions

The early success of plant introduction as a function of source material and habitat quality was tested in Arnica montana to determine whether different introduction strategies could affect plant viability of the introduced population. Plants originating from related (single-family introductions) and unrelated (multi-family introductions) individuals were introduced into undisturbed sites and into sites which were limed, turf cut, or both turf cut and limed. For four consecutive years, we analysed plant performance by measuring survival time, growth, and reproductive capacity. Introduction success was found to be strongly affected by habitat quality. Turf cutting in combination with additional liming significantly increased reproductive capacity and stimulated early flowering. To restore eutrophic and acidified soil conditions, turf cutting with additional liming prior to introduction is recommended. Furthermore, a significant effect of multi-family introductions was observed. Multi-family introductions showed higher introduction success compared to single-family introductions. Although the long-term effects of multi-family introductions will emerge after several generations, the preliminary results suggest multi-family introductions as the most successful introduction strategy.