Inconclusive decisions and error rates in forensic science

In recent years, there has been discussion and controversy relating to the treatment of inconclusive decisions in forensic feature comparison disciplines when considering the reliability of examination methods and results. In this article, we offer a brief review of the various viewpoints and suggestions that have been recently put forth, followed by a solution that we believe addresses the treatment of inconclusive decisions. We consider the issues in the context of method conformance and method performance as two distinct concepts, both of which are necessary for the determination of reliability. Method conformance relates to an assessment of whether the outcome of a method is the result of the analyst's adherence to the procedures that define the method. Method performance reflects the capacity of a method to discriminate between different propositions of interest (e.g., mated and non-mated comparisons). We then discuss implications of these issues for the forensic science community.

Automatic imitation in school-aged children

Children imitate others for different reasons: To learn from others and to reach social goals such as affiliation or prosociality. So far, imitative acts have been measured using diverging methods in children and adults. Here, we investigated whether school-aged children's imitation can be measured via their automatic imitation with a classical imitation-inhibition task (Brass et al., 2000) as has been used in adults. To this end, we measured automatic imitation in N=94 7-8-year-olds and N=10 adults. The results were similar in children and adults: Observing actions that are incongruent with participants' actions interferes with their responses resulting in increased reaction times and error rates. This shows that assessing automatic imitation via the imitation-inhibition task is feasible in children, and creates the basis for future studies to compare the behaviour of different age groups with the same imitation task.

Novices cannot fill the examiners' shoes: Evidence of footwear examiners' expertise in shoe comparisons

The value of a footwear examiner's opinion centres on their ability to determine whether a particular shoe made an impression with greater accuracy than a novice. However, there has been limited research on the expertise of footwear examiners and the accuracy and reproducibility of their decisions. In the current study, we measured the accuracy and consensus of 31 footwear examiners versus a comparison group of 29 novices. Participants completed 20 ground truth known mock shoe comparisons. Results demonstrated that footwear examiners were more accurate than novices, regardless of comparison difficulty. Overall, on trials where probative decisions were given, examiners made false identifications and false exclusions on a total of 3% and 2% of trials, while novices made false identifications and false exclusions on a total of 19% and 17% of trials. Examiners also demonstrated better consensus in their opinions than novices, although both groups demonstrated low levels of agreement in their responses and variability in their interpretation of the conclusion scale. In summary, these findings support the proposition that footwear examiners show expert-level performance in matching known and unknown footwear impressions. These performance estimates may help the criminal justice system to appropriately value footwear examination evidence.

Empathic pain observation does not influence automatic imitation in an online setting

Previous research has shown that empathic pain observation can lead to motor facilitation in the form of faster reaction times. However, it is unclear whether participants are focusing on the others' pain or simply focusing on their own discomfort/distress (from watching the videos) during the task. This is an important issue as self- vs other-oriented focusing plays a key role in empathic processing. To address this issue, we combined empathic pain observation with the automatic imitation task (AIT). Previous work has shown that AIT effects are smaller after experiencing pain, which has been interpreted as the result of the experience of pain leading to a self-oriented focus. If empathic pain observation similarly leads to a self-oriented focus, then we should expect similar AIT results after pain observation (smaller AIT effects); however, if it instead leads to an other-oriented focus, then we should see the opposite (larger AIT effects). Although we found initial evidence for the latter hypothesis (Experiment 1), subsequent failed replications suggests that we do not have sufficient evidence to claim that pain observation influences automatic imitation one way or the other (Experiment 2 and 3). We discuss some possible reasons for finding null results in these experiments and suggest future avenues of research to better elucidate this topic.

Statistical consideration when adding new arms to ongoing clinical trials: the potentials and the caveats

BACKGROUND

Platform trials improve the efficiency of the drug development process through flexible features such as adding and dropping arms as evidence emerges. The benefits and practical challenges of implementing novel trial designs have been discussed widely in the literature, yet less consideration has been given to the statistical implications of adding arms. MAIN: We explain different statistical considerations that arise from allowing new research interventions to be added in for ongoing studies. We present recent methodology development on addressing these issues and illustrate design and analysis approaches that might be enhanced to provide robust inference from platform trials. We also discuss the implication of changing the control arm, how patient eligibility for different arms may complicate the trial design and analysis, and how operational bias may arise when revealing some results of the trials. Lastly, we comment on the appropriateness and the application of platform trials in phase II and phase III settings, as well as publicly versus industry-funded trials.

CONCLUSION

Platform trials provide great opportunities for improving the efficiency of evaluating interventions. Although several statistical issues are present, there are a range of methods available that allow robust and efficient design and analysis of these trials.

How can courts take into account the uncertainty in a likelihood ratio?

As legal practitioners and courts become more aware of scientific methods and evidence evaluation, they are demanding measures of the reliability of expert opinion. In particular, there are calls for error rates to accompany opinion evidence in comparative forensic sciences. While error rates or confidence intervals can be useful for those disciplines that claim to identify the source of a trace, the call for these statistical tools has extended to sciences that present opinions in the form of a likelihood ratio. In this article we argue against presenting both a likelihood ratio and numerical measures of its uncertainty. We explain how the LR already encapsulates uncertainty. Instead we consider how sensitivity analyses can be used to guide the presentation of LRs that are informative to the court and not unfair to defendants.

Testing the accuracy and reliability of palmar friction ridge comparisons - A black box study

Critics and commentators have been calling for some time for black box studies in the forensic science disciplines to establish the foundational validity of those fields-that is, to establish a discipline-wide, base-rate estimate of the error rates that may be expected in each field. While the well-known FBI/Noblis black box study has answered that call for fingerprints, no research to establish similar error rates for palmar impressions has been previously undertaken. We report the results of the first large-scale black box study to establish a discipline-wide error rate estimate for palmar comparisons. The 226 latent print examiner participants returned 12,279 decisions over a dataset of 526 known ground-truth pairings. There were 12 false identification decisions made yielding a false positive error rate of 0.7%. There were also 552 false exclusion decisions made yielding a false negative error rate of 9.5%. Given their larger number, false negative error rates were further stratified by size, comparison difficulty, and area of the palm from which the mark originated. The notion of "questionable conclusions," in which the ground truth response may not be the most appropriate, is introduced and discussed in light of the data obtained in the study. Measures of examiner consistency in analysis and comparison decisions are presented along with statistical analysis of the ability of many variables, such as demographics or image quality, to predict outcomes. Two online apps are introduced that will allow the reader to fully explore the results on their own, or to explore the notions of frequentist confidence intervals and Bayesian credible intervals.

Characterization of background noise in MiSeq MPS data when sequencing human mitochondrial DNA from various sample sources and library preparation methods

Improved resolution of massively parallel sequencing (MPS) allows for the characterization of mitochondrial (mt) DNA heteroplasmy to levels previously unattainable with traditional sequencing approaches. An essential criterion for the reporting of heteroplasmy is the ability of the MPS method to distinguish minor sequence variants (MSVs) from system noise, or error. Therefore, an assessment of the background noise in the MPS method is desirable to identify the point at which reliable data can be reported. Substitution and sequence specific error (SSE) was evaluated for a variety of sample types and two library preparations. Substitution error rates ranged from 0.18 to 0.49 per 100 nucleotides with C positions generally having the highest rate of misincorporation. Comparison of error rates across sample types indicated a significant increase for samples with damaged DNA. The positions of error were varied across datasets (pairwise concordance 0-68%), but had greater consistency within the damaged samples (80-96%). The most commonly observed motif preceding error in forward reads was CCG, while GGT was most common in reverse reads, both consistent with previous findings. The findings illustrate that for datasets containing samples with damaged DNA, reporting thresholds for heteroplasmy may have to be modified and individual sites with error levels exceeding thresholds should be scrutinized. Collectively, the shifting error profiles observed across the various sample types and library preparation methods demonstrates the need for an assessment of error under these varying circumstances. Characterization of the applicable background noise will help to ensure that thresholds are reliably set for detection of true MSVs.

(Mis)use of scientific measurements in forensic science

Forensic science error rate studies have not given sufficient attention or weight to inconclusive evidence and inconclusive decisions. Inconclusive decisions can be correct decisions, but they can also be incorrect decisions. Errors can occur when inconclusive evidence is determined as an identification or exclusion, or conversely, when same- or different-source evidence is incorrectly determined as inconclusive. We present four common flaws in error rate studies: 1. Not including test items which are more prone to error; 2. Excluding inconclusive decisions from error rate calculations; 3. Counting inconclusive decisions as correct in error rate calculations; and 4. Examiners resorting to more inconclusive decisions during error rate studies than they do in casework. These flaws seriously undermine the credibility and accuracy of error rates reported in studies. To remedy these shortcomings, we present the problems and show the way forward by providing a corrected experimental design that quantifies error rates more accurately.

Development and validation of a systematic approach for the quantitative assessment of the quality of duct tape physical fits

Duct tape is a common type material found at crime scenes such as sexual assaults, murders, kidnappings, and bombings. During the examination of a known and questioned item, a 3D realignment along their edges is known as a physical fit and is often regarded as conclusive evidence that the items were once part of a single object. The conclusion of a fit between edges relies on the examiner's judgment to identify distinctive features across the tape ends. However, there are currently no consensus-based methodologies or standards to inform their opinions. This study developed a practical method to qualify and quantify tape end match features using edge similarity scores (ESS) and provided an empirically demonstrable basis to assess the significance of duct tape fracture fits. ESS were calculated as the proportion of observed matching sections per scrim bins across the fractured edge, providing a quantifiable criterion and means for a systematic peer review process. A set of 2280 duct tape end comparisons were analyzed for the validation study. The probative value of physical fits was evaluated through similarity metrics, error rates, and score-based likelihood ratios. The effects of separation method, stretching, and tape grade on the distribution of ESS and the overall accuracy are reported. The accuracy ranged from 84.9 % (higher quality hand-torn set) to over 99 % (low and mid-quality sets). No false positives were reported for any of the sets examined. On average, ESS higher than 80 % provided a score likelihood ratio (SLR) that supported the conclusion of a match, and ESS lower than 25 % provided an SLR supporting the conclusion of non-match.

A Bayesian approach for the analysis of error rate studies in forensic science

Over the past decade, the field of forensic science has received recommendations from the National Research Council of the U.S. National Academy of Sciences, the U.S. National Institute of Standards and Technology, and the U.S. President's Council of Advisors on Science and Technology to study the validity and reliability of forensic analyses. More specifically, these committees recommend estimation of the rates of occurrence of erroneous conclusions drawn from forensic analyses. "Black box" studies for the various subjective feature-based comparison methods are intended for this purpose. In general, "black box" studies often have unbalanced designs, comparisons that are not independent, and missing data. These aspects pose difficulty in the analysis of the results and are often ignored. Instead, interpretation of the data relies on methods that assume independence between observations and a balanced experiment. Furthermore, all of these projects are interpreted within the frequentist framework and result in point estimates associated with confidence intervals that are confusing to communicate and understand. We propose to use an existing likelihood-free Bayesian inference method, called Approximate Bayesian Computation (ABC), that is capable of handling unbalanced designs, dependencies among the observations, and missing data. ABC allows for studying the parameters of interest without recourse to incoherent and misleading measures of uncertainty such as confidence intervals. By taking into account information from all decision categories for a given examiner and information from the population of examiners, our method also allows for quantifying the risk of error for the given examiner, even when no error has been recorded for that examiner. This opens the door to the detection of behavioural patterns in the decision-making of examiners through their ABC rate estimates. These patterns could be used to detect error prone examiners, enabling additional training efforts to be more tailored to each examiner, limiting the risk of errors before they occur. We illustrate our proposed method by reanalysing the results of the "Noblis Black Box" study by Ulery et al. [18]. We did not choose this study because we disagree with their results, but because it is a good example of a study with dependent observations and missing data, and the data is publicly available. The ABC estimates for the population generally agreed with Ulery et al.'s plug-in estimates. However, credible intervals obtained from ABC are much wider than the confidence intervals for the corresponding parameter estimates that did not account for the dependencies among observations.

An analysis of deficiencies in the data of interventional drug trials registered with Clinical Trials Registry - India

Background

Clinical Trials Registry - India (CTRI) was established in July 2007 and today hosts thousands of trials, a significant fraction of them registered in the last couple of years. We wished to undertake an up-to-date analysis of specific fields of the registered trials. In doing so we discovered problems with the quality of the data, which we describe in this paper.

Methods

We downloaded CTRI records and reformatted the data into an SQLite database, which we then queried. We also accessed ClinicalTrials.gov records as needed.

Results

We discovered various categories of problems with the data in the CTRI database, including (1) a lack of clarity in the classification of Types of Study, (2) internal inconsistencies, (3) incomplete or non-standard information, (4) missing data, (5) variations in names or classification, and (6) incomplete or incorrect details of ethics committees. For most of these problems, error rates have been calculated, over time. Most were found to be in single digits, although others were significantly higher. We suggest how data quality in future editions of CTRI could be improved, including (1) a more elaborate and structured way of classifying the Type of Study, (2) the use of logic rules to prevent internal inconsistencies, (3) less use of free text fields and greater use of drop-down menus, (4) more fields to be made compulsory, (5) the pre-registration of individuals’ and organizations’ names and their subsequent selection from drop-down menus while registering a trial, and (6) more information about each ethics committee, including (a) its address and (b) linking the name of the trial site to the relevant ethics committee. As we discuss problems with the data of specific fields, we also examine — where possible — the quality of the data in the corresponding fields in ClinicalTrials.gov, the largest clinical trial registry in the world.

Conclusions

It is a scientific and ethical obligation to correctly record all information pertaining to each trial run in India. CTRI is a valuable database that has proved its worth in terms of improving the record of trials in the country. The suggestions made herein would improve it further.

Electronic supplementary material

The online version of this article (10.1186/s13063-019-3592-0) contains supplementary material, which is available to authorized users.

Perceptions and estimates of error rates in forensic science: A survey of forensic analysts

Every scientific technique features some error, and legal standards for the admissibility of scientific evidence (e.g., Daubert v. Merrill Dow Pharmaceuticals, Inc., 1993; Kumho Tire Co v. Carmichael, 1999) guide trial courts to consider known error rates. However, recent reviews of forensic science conclude that error rates for some common techniques are not well-documented or even established (e.g., NAS, 2009; PCAST, 2016). Furthermore, many forensic analysts have historically denied the presence of error in their field. Therefore, it is important to establish what forensic scientists actually know or believe about errors rates in their disciplines. We surveyed 183 practicing forensic analysts to examine what they think and estimate about error rates in their various disciplines. Results revealed that analysts perceive all types of errors to be rare, with false positive errors even more rare than false negatives. Likewise, analysts typically reported that they prefer to minimize the risk of false positives over false negatives. Most analysts could not specify where error rates for their discipline were documented or published. Their estimates of error in their fields were widely divergent - with some estimates unrealistically low.

Curation and analysis of clinical pathology parameters and histopathologic findings from eTOXsys, a large database project (eTOX) for toxicologic studies

Large data sharing projects amongst the pharmaceutical industry have the potential to generate new insights using data on a scale that has not been previously available. A retrospective analysis of the preclinical toxicology data collected as part of the eTOX project was conducted with the aim to provide background rates and treatment-related value analysis on both clinical pathology and histopathology datasets. Incorporated into this analysis was an extensive data consolidation task to standardise all data. Reference intervals for common clinical pathology parameters in rat and dog were generated, alongside background histopathology incidence rates in the liver, heart and kidney. Systematically applied decision thresholds allowed consistent relabelling of data points considered anomalous, and maximum fold change estimates. Relabelling of anomalous data points was conducted for the histopathology data using a Bayesian model to identify dose-dependent increases in pathologies. The results of this study allow: newly generated data to be analysed using the same methodology, rates and distributions to be used when building predictive dose-response models, and the possibility to correlate clinical pathology findings with concurrent histopathology findings. In the first half of this paper we discuss data curation, in the second half we report on the analytical methods and results.

Error management, reliability and cognitive evolution

The paper offers a partial vindication of Sterelny's view on the role of error rates and reliability in his theory of decoupled representation based on modelling techniques borrowed from the biological literature on evolution in stochastic environments. In the case of a tight link between tracking states and behaviour, I argue that in its full generality Sterelny's account instantiates the base-rate fallacy. With regard to non-tightly linked behaviour, I show that Sterelny's account can be vindicated subject to an adequate evolutionary model and a suitable notion of reliability.

Asymptotic performance of the quadratic discriminant function to skewed training samples

This study investigates the asymptotic performance of the quadratic discriminant function (QDF) under skewed training samples. The main objective of this study is to evaluate the performance of the QDF under skewed distribution considering different sample size ratios, varying the group centroid separators and the number of variables. Three populations [Formula: see text] with increasing group centroid separator function were considered. A multivariate normal distributed data was simulated with MatLab R2009a. There was an increase in the average error rates of the sample size ratios 1:2:2 and 1:2:3 as the total sample size increased asymptotically in the skewed distribution when the centroid separator increased from 1 to 3. The QDF under the skewed distribution performed better for the sample size ratio 1:1:1 as compared to the other sampling ratios and under centroid separator [Formula: see text].

Analysis of medication prescribing errors in critically ill children

Medication prescribing errors (MPE) can result in serious consequences for patients. In order to reduce errors, we need to know more about the frequency, the type and the severity of such errors. We therefore performed a prospective observational study to determine the number and type of medication prescribing errors in critically ill children in a paediatric intensive care unit (PICU). Prescribing errors were prospectively identified by a clinical pharmacist. A total of 1129 medication orders were analysed. There were 151 prescribing errors, giving an overall error rate of 14 % (95 % CI 11 to 16). The medication groups with the highest proportion of MPEs were antihypertensives, antimycotics and drugs for nasal preparation with error rates of each 50 %, followed by antiasthmatic drugs (25 %), antibiotics (15 %) and analgesics (14 %). One hundred four errors (70 %) were classified as MPEs which required interventions and/or resulted in patient harm equivalent to 9 % of all medication orders (95 % CI 6.5 to 14.4). Forty-five MPEs (30 %) did not result in patient harm.

CONCLUSION

With a view to reduce MPEs and to improve patient safety, our data may help to prevent errors before they occur.

WHAT IS KNOWN

• Prescribing errors may be the most frequent medication errors. • In paediatric populations, the incidence of prescribing errors is higher than in adults. What is New: • Several risk factors for medication prescribing errors, such as medication groups, long PICU stay, and mechanical ventilation could be presented. • Analysing the combination of the most frequent prescribing errors and the severity of these errors.

Differences in recognition of similar medication names between pharmacists and nurses: a retrospective study

Background

Differences in error rates between pharmacists and nurses in terms of drug confirmation have not been studied. The purpose of this study was to analyze differences in error rates between pharmacists and nurses from the viewpoint of error categories, and to clarify differences in recognition regarding drug name similarity.

Methods

In this study, preparation errors and incidents were classified into three categories (drug strength errors, drug name errors, and drug count errors) to investigate the influence of error categories on pharmacists and nurses. In addition, errors in two categories (drug strength errors and drug name errors) were reclassified into another two error groups, to investigate the influence of drug name similarity on pharmacists and nurses: a “drug name similarity (−) group” and a “drug name similarity (+) group”. Then, differences in error rates of pharmacists and those of nurses were analyzed respectively within three categories and two groups. Furthermore, differences in error rates between pharmacists and nurses were analyzed in each of the three categories and two groups.

Results

Error rates of pharmacists for both drug strength errors and drug name errors were significantly higher than that for drug count errors, and similar results were obtained for nurses (P < 0.05). However, there were no significant differences in error rates between pharmacists and nurses in each of the three categories. Furthermore, error rate of nurses was significantly higher than that of pharmacists in the drug name similarity (+) group (P < 0.05), while there was no significant difference in error rates between pharmacists and nurses in the drug name similarity (−) group.

Conclusions

These results suggest that in contrast to pharmacists, nurses are easily affected by similarities in drug names. Therefore, pharmacists should offer information on medications having plural strengths or similar names to nurses, in order to minimize damage to patients resulting from errors.

Electronic supplementary material

The online version of this article (doi:10.1186/s40780-015-0017-4) contains supplementary material, which is available to authorized users.

A comment on experimental results of fingerprint comparison validity and reliability: A review and critical analysis

We respond to the article "Experimental results of fingerprint comparison validity and reliability: A review and critical analysis" by Ralph and Lyn Haber which offers (a) a one-sided criticism of the state of affairs in latent print examination, (b) lack of original data supporting that their suggested approach, and (c) a host of incorrect statements, inaccuracies, or obscure interpretations of the existing data.

Error analysis of idealized nanopore sequencing

This numerical study provides an error analysis of an idealized nanopore sequencing method in which ionic current measurements are used to sequence intact single-stranded DNA in the pore, while an enzyme controls DNA motion. Examples of systematic channel errors when more than one nucleotide affects the current amplitude are detailed, which if present will persist regardless of coverage. Absent such errors, random errors associated with tracking through homopolymer regions are shown to necessitate reading known sequences (Escherichia coli K-12) at least 140 times to achieve 99.99% accuracy (Q40). By exploiting the ability to reread each strand at each pore in an array, arbitrary positioning on an error rate versus throughput tradeoff curve is possible if systematic errors are absent, with throughput governed by the number of pores in the array and the enzyme turnover rate.