Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

Convergence of HbA1c values towards target in 272 primary care patients following nine years of target-driven care

BACKGROUND

We wished to determine the effect of a target-driven incentivised programme on haemoglobin A1c (HbA1c ) values in a UK diabetic population.

METHODS

An audit was carried out in 1999-2000, which included an estimation of glycaemic control in a randomly selected diabetic cohort from ten primary care practices in Sutton Coldfield, serving a population of 90 000 patients. Each practice was given a randomised list of patients and asked to complete detailed questionnaires on patients with confirmed diabetes. We collected data on 516 patients, 425 of whom had their HbA1c measured in 1999-2000 (Audit 2000). A re-audit of HbA1c was carried out in 2007-08 (Audit 2008) determining the changes in HbA1c since the original audit. Of the original cohort, 272 patients had an audit of HbA1c carried out in Audit 2008.

RESULTS

Overall, a small increase in median and mean HbA1c values was observed. We estimated that the proportion of patients with HbA1c achieving the lower Quality and Outcomes Framework HbA1c target of < 7.5%; 173 of the 272 patients met this target in Audit 2000, whereas the number was 162 in Audit 2008. To understand the changes observed, patients were stratified as quintiles based on the HbA1c in Audit 2000 and changes in HbA1c after 8 years for each quintile were estimated. The mean changes for the different quintiles are: quintile 1 (HbA1c < 6.1%), +1.49%; quintile 2 (HbA1c 6.1- 6.6%), +0.8%; quintile 3 (HbA1c 6.7-7.3%), +0.3%; quintile 4 (HbA1c 7.4-8.5%), -0.18%; and quintile 5 (HbA1c > 8.5%), -1.55%.

CONCLUSION

Our results suggest that, eight years on, patients with poor glycaemic control in 2000 saw an overall decrease in HbA1c by 2008, with the reverse seen in patients with good control.

Measurement properties of the WOMAC LK 3.1 pain scale

OBJECTIVE

The Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) is applied extensively to patients with osteoarthritis of the hip or knee. Previous work has challenged the validity of its physical function scale however an extensive evaluation of its pain scale has not been reported. Our purpose was to estimate internal consistency, factorial validity, test-retest reliability, and the standard error of measurement (SEM) of the WOMAC LK 3.1 pain scale.

METHOD

Four hundred and seventy-four patients with osteoarthritis of the hip or knee awaiting arthroplasty were administered the WOMAC. Estimates of internal consistency (coefficient alpha), factorial validity (confirmatory factor analysis), and the SEM based on internal consistency (SEM(IC)) were obtained. Test-retest reliability [Type 2,1 intraclass correlation coefficients (ICC)] and a corresponding SEM(TRT) were estimated on a subsample of 36 patients.

RESULTS

Our estimates were: internal consistency alpha=0.84; SEM(IC)=1.48; Type 2,1 ICC=0.77; SEM(TRT)=1.69. Confirmatory factor analysis failed to support a single factor structure of the pain scale with uncorrelated error terms. Two comparable models provided excellent fit: (1) a model with correlated error terms between the walking and stairs items, and between night and sit items (chi2=0.18, P=0.98); (2) a two factor model with walking and stairs items loading on one factor, night and sit items loading on a second factor, and the standing item loading on both factors (chi2=0.18, P=0.98).

CONCLUSION

Our examination of the factorial structure of the WOMAC pain scale failed to support a single factor and internal consistency analysis yielded a coefficient less than optimal for individual patient use. An alternate strategy to summing the five-item responses when considering individual patient application would be to interpret item responses separately or to sum only those items which display homogeneity.

Maternal serum screening for trisomy 18: assessing different statistical models to optimize detection rates

We have evaluated three alternative models for trisomy 18 screening using the maternal serum markers alpha-fetoprotein (AFP) and intact human chorionic gonadotrophin (hCG). Using data from 46 affected pregnancies and 48 150 unaffected pregnancies, we calculated distribution parameters for AFP and hCG multiples of the median (MoMs) and the factor comprising AFP MoMxhCG MoM. The trisomy 18 risk at mid-trimester was then calculated using either bivariate analysis of AFP and hCG MoMs or univariate analysis of AFP MoMxhCG MoM. The observed detection rates and positive rates obtained using either published distribution parameters or those derived from the West Midlands population were compared for each model. Using fixed cut-offs for AFP and hCG of 0.66 and 0.40 MoMs resulted in a detection rate of 28.3% for a 0.5% false positive rate (FPR). Using published parameters, the univariate analysis model had a slightly higher detection rate of 32.6% for a 0.5% FPR (cut-off 1:248) compared to the bivariate model which was 28.3% (cut-off 1:239). Locally derived distribution parameters significantly improved the detection rate for the bivariate model for FPRs between 0.4-1.3% but worsened it below 0.4%. For the univariate model there was little difference in detection whether local or published parameters were used. Thus, we have confirmed that trisomy 18 screening using two markers can be a worthwhile addition to Down screening.

Measurement of urinary free cortisol using the Acs:180 serum cortisol chemiluminescent immunoassay

Measurement of urinary free cortisol using the Bayer Automated Chemiluminescent System (ACS:180 PLUS) was evaluated and compared with an in-house extraction radioimmunoassay (RIA). Inter-assay coefficients of variation were acceptable, being respectively 5.3%, 4.8% and 3.8% at 141, 406 and 942 nmol/L (n=20) for the ACS direct assay and 19.4% and 12.1% at 27 and 116 nmol/L with dichloromethane extraction (n=10). Using tritiated cortisol, mean extraction efficiency for the ACS extraction protocol was 89% compared with 104% for the RIA method (n=6, P<0.001) and using urine spiked with ACS serum calibrator, extraction efficiency was only 76% (standard deviation 8.8%, n=49) on the ACS. Urine spiked with cortisol dissolved in ethanol also gave significantly lower recoveries on the ACS for both direct and extraction methods compared with the RIA. Regression analysis of results from a mixture of control serum samples and samples from patients (n=93) showed good correlations between the direct and extraction ACS methods and the RIA extraction assay. The median concentration in 23 normal subjects and 95% reference intervals (nmol/24 h) were: ACS direct, 237 (135-505); ACS extraction, 91 (33-239); and RIA extraction, 146 (80-334). 5Beta-dihydrocortisol was the only major interferent in all assays. The ACS extraction assay showed acceptable performance and correlated well with the extraction RIA, although there was evidence of matrix-dependent effects causing low recovery.

Impact of preinjury warfarin use in elderly trauma patients

BACKGROUND

This study examined the hypothesis that elderly trauma patients on warfarin before injury will have increased morbidity and mortality compared with elderly trauma patients not on warfarin.

METHODS

From 1993 to 1995, trauma patients were grouped by age and presence or absence of warfarin use before injury. Groups were analyzed with respect to Injury Severity Score, Trauma Registry and Injury Severity Score, Glasgow Coma Scale score, Intensive Care Unit days, hospital days, units of blood transfused, and mortality rates. Statistical analysis was completed by using the Student's t test.

RESULTS

Records of 61 patients administered warfarin and 800 patients not administered warfarin were available for analysis. There were no statistically significant differences between patients on prehospital warfarin and those not on prehospital warfarin.

CONCLUSION

This study indicates that elderly trauma patients on warfarin before injury do not have increased morbidity and mortality compared with elderly trauma patients not on warfarin.

Evaluation of different weight correction methods for antenatal serum screening using data from two multi-centre programmes

Weight correction of serum markers is widely used when screening for Down's syndrome and open neural tube defects (NTD) because marker concentrations decrease with increasing maternal weight. Log-linear regression is frequently used for determining weight correction factors, but recently reciprocal-linear regression has been suggested to have advantages. We compared both methods of weight correction using data from two screening programmes carried out by this laboratory, one using alpha-fetoprotein (AFP) and total human chorionic gonadotrophin (HCG) (n = 129,143) and the other, AFP and free beta-HCG (n = 39,982). The reciprocal-linear method fitted the data more closely but did not significantly alter the detection rate or screen positive rate (SPR) for Down's syndrome or NTD with either dataset. Without correction, women heavier or lighter than average weight had significantly different SPRs for Down's syndrome and NTD compared with those weighing close to the median weight. Both correction methods smoothed out the variability in the SPR for Down's syndrome to a similar degree, but reciprocal-linear regression was much better at reducing the variability in SPR for NTD and its use is therefore worthwhile.

A comparison of automated fructosamine and HbA1c methods for monitoring diabetes in pregnancy

Two automated methods for measuring fructosamine (Test Plus and the original fructosamine assay) and glycated haemoglobin (Tina-quant immunoassay) were compared to determine which is the best index of blood glucose control during pregnancy. Thirteen women with type 1 diabetes were studied, with four-weekly measurements of HbA1c and fructosamine Test Plus using a Hitachi 911 analyser and fructosamine measured using an Olympus AU800 analyser. HbA1c correlated better (r = 0.573) with mean blood glucose (MBG) concentration than did fructosamine Test Plus (r = 0.347), even after correction for total protein concentration (r = 0.463), while there was no significant correlation with the original fructosamine method (r = 0.201). HbA1c correlated better with fasting/pre-prandial MBG concentrations, whereas fructosamine Test Plus correlated better with post-prandial MBG concentrations. Fructosamine Test Plus decreased with gestational age, and correlated with albumin and total protein concentrations, whereas HbA1c did not change with gestational age. Thus, HbA1c and fructosamine Test Plus were found to be useful in verifying home blood glucose measurements in diabetic pregnancy, with HbA1c being the best predictor of MBG concentration.

Adherence to balance tolerance limits at the Upper Mississippi Science Center, La Crosse, Wisconsin

Verification of balance accuracy entails applying a series of standard masses to a balance prior to use and recording the measured values. The recorded values for each standard should have lower and upper weight limits or tolerances that are accepted as verification of balance accuracy under normal operating conditions. Balance logbooks for seven analytical balances at the Upper Mississippi Science Center were checked over a 3.5-year period to determine if the recorded weights were within the established tolerance limits. A total of 9435 measurements were checked. There were 14 instances in which the balance malfunctioned and operators recorded a rationale in the balance logbook. Sixty-three recording errors were found. Twenty-eight operators were responsible for two types of recording errors: Measurements of weights were recorded outside of the tolerance limit but not acknowledged as an error by the operator (n = 40); and measurements were recorded with the wrong number of decimal places (n = 23). The adherence rate for following tolerance limits was 99.3%. To ensure the continued adherence to tolerance limits, the quality-assurance unit revised standard operating procedures to require more frequent review of balance logbooks.

Calibration or verification? A balanced approach for science

The calibration of balances is routinely performed both in the laboratory and the field. This process is required to accurately determine the weight of an object or chemical. The frequency of calibration and verification of balances is mandated by their use and location. Tolerance limits for balances could not be located in any standard procedure manuals. A survey was conducted to address the issues of calibration and verification frequency and to discuss the significance of defining tolerance limits for balances. Finally, for the benefit of laboratories unfamiliar with such procedures, we provide a working model based on our laboratory, the Upper Mississippi Science Center (UMSC), in La Crosse, Wisconsin.

Glycation of monoclonal antibodies impairs their ability to bind antigen

As elevated levels of glycated IgG have been detected in the plasma of patients with diabetes mellitus, a disease associated with increased susceptibility to infection, we have investigated whether glycation of MoAbs affects the kinetics and/or affinity of antigen binding. Three mouse MoAbs were incubated with 0.5 M glucose at pH 7.4 for 14-21 days at 37 degrees C. Control MoAbs were incubated using identical conditions but with no added glucose. Using a surface plasmon resonance technique we found that glycation significantly increased the rate of dissociation (kdiss) of the antigen-antibody complex for all three MoAbs (P < 0.05, n = 4), but had no significant effect on the rate of association (kass). For one of the MoAbs, against human IgG (Fab), we also measured kdiss by an alternative method utilizing radiolabelled antigen, which confirmed that glycation of the antibody significantly increases kdiss (P < 0.001, n = 8). We also found using an ELISA-based method that glycation of the same MoAb significantly increased the equilibrium dissociation constant (Kd) (P < 0.05, n = 6). A significant increase in kd was observed after glycation using glucose concentrations consistent with those found in poorly controlled diabetics (P < 0.02, n = 5). We conclude that in vitro glycation can significantly lower the affinity of an antibody for its antigen, and significantly increases the rate of dissociation of the antigen-antibody complex.

Glycation increases the vascular clearance rate of IgG in mice

As elevated levels of glycated IgG have been detected in the plasma of diabetics we have investigated whether glycation of IgG affects its vascular clearance rate, using a mouse model system. Polyclonal mouse IgG was aseptically incubated for 14-19 days with 0.5 M glucose in 0.1 M phosphate buffer (pH 7.4) at 37 degrees C. As control, IgG was incubated under identical conditions but with no added glucose. After incubation, both forms were labelled with 125I and injected intravenously into BALB/c mice. The rate of vascular clearance of the glycated IgG was found to be significantly higher than the control IgG in the periods 5-24 h (P < 0.001, n = 6) and 24-48 h (P < 0.01, n = 6) after injection. After 2-3 days the mice were killed and the major organs were harvested. With glycated IgG there was a significant increase in the 125I accumulated in the kidney (P < 0.02). In later experiments, dual labelling with 131I and 125I allowed mixtures of glycated and unglycated IgG to be injected into the same mouse so that the vascular clearance of both forms of IgG could be followed simultaneously. These experiments confirmed that glycation of the IgG significantly increases its vascular clearance rate.

Colorimetric assay of glycoprotein glycation free of interference from glycosylation residues

We have developed a colorimetric assay for determining the degree of glycation of serum proteins that is unaffected by glycosylation residues. This was accomplished by reducing the proteins with sodium borohydride prior to periodate oxidation. Previous periodate-based methods, which offer several advantages over other glycation assays, cannot determine glycoprotein glycation because interference from sialic residues in the glycan chain can lead to overestimation of the amount of glycation products. Without reduction, glycation of fetuin was double that of asialofetuin glycated under identical conditions. We found that borohydride reduction before periodate oxidation increases the amount of formaldehyde released in proportion to the extent of glycation, irrespective of the degree of glycosylation. Using two glycoproteins and an unglycosylated protein, we showed how measurement of the formaldehyde increase enables the extent of glycoprotein glycation to be determined without removal of interfering sugars.

Colorimetric assay of glycoprotein glycation free of interference from glycosylation residues

We have developed a colorimetric assay for determining the degree of glycation of serum proteins that is unaffected by glycosylation residues. This was accomplished by reducing the proteins with sodium borohydride prior to periodate oxidation. Previous periodate-based methods, which offer several advantages over other glycation assays, cannot determine glycoprotein glycation because interference from sialic residues in the glycan chain can lead to overestimation of the amount of glycation products. Without reduction, glycation of fetuin was double that of asialofetuin glycated under identical conditions. We found that borohydride reduction before periodate oxidation increases the amount of formaldehyde released in proportion to the extent of glycation, irrespective of the degree of glycosylation. Using two glycoproteins and an unglycosylated protein, we showed how measurement of the formaldehyde increase enables the extent of glycoprotein glycation to be determined without removal of interfering sugars.

Mechanism of self-tolerance of gamma/delta T cells in epithelial tissue

The present study examined mechanisms of tolerance for T cell receptor gamma/delta (TCR-gamma/delta) cells. Using a transgenic (Tg) model, we demonstrate that although alloantigen (Ag)-specific TCR-gamma/delta cells are deleted in the thymus and spleen of Ag-bearing mice, intraepithelial lymphocytes (IELs) expressing normal levels of the Tg TCR were present. However, Tg+ IELs from Ag-bearing mice were unresponsive to activation. Furthermore, self-reactive Tg+ IELs decreased in number over time. Thus, in epithelial tissue, Tg TCR-gamma/delta cells are eliminated subsequent to and most likely as a result of the induction of clonal anergy.

Control of self-reactivity in the intestine

In the intestine maintenance of self-tolerance may involve tissue-specific self-Ags, APCs, 'second signals', and extrathymic pathways of T cell maturation. These factors combine to create a unique environment where autoimmune tissue destruction is prevented despite local inflammatory influences. In this review we summarize our findings using a TCR-gamma delta transgenic model where self-tolerance was maintained by clonal deletion for cells localizing to peripheral lymphoid tissue and by clonal anergy for cells localizing to the intraepithelial compartments. Several possible explanations exist for these results but in general, these findings have implications for the maintenance of self-tolerance of normal TCR-alpha beta and TCR-gamma delta IELs in epithelial tissues such as the intestine.

Calcium mobilization in human platelets using indo-1 and flow cytometry

Regulation of cytoplasmic free calcium concentration is believed to be important in the response of platelets to external stimuli. A relatively new fluorescent calcium indicator, indo-1, has properties by which alterations of cytoplasmic calcium can be evaluated in single platelets by flow cytometry. Activation of platelets at a temperature lower than 37 degrees C allows examination of the heterogeneity of intracellular free calcium levels and can distinguish variations among platelets in the initiation, duration, and magnitude of calcium fluxes. The clear advantage of flow cytometric analysis of platelet cytosolic calcium is that stimulus-response coupling can now be studied on a single cell basis. Platelets were activated by addition of human alpha-thrombin or ADP at 37 degrees C or at room temperature (22 degrees C). Activation at 37 degrees C approaches more closely an in vivo response and, as expected, increases in cytosolic calcium occurred within seconds of agonist addition. Transient increases in cytoplasmic calcium levels occurred when platelets were challenged with a low concentration of agonist. Heterogeneity in cytoplasmic calcium levels was also observed at 10(-5) mol/L ADP and 0.1 U/mL alpha-thrombin. Some of this heterogeneity was no longer observed at higher concentrations of agonist (10(-4) mol/L ADP and 0.5 U/mL thrombin), suggesting that a sufficient magnitude of signal is required to induce changes in platelet cytosolic calcium. Light-scatter properties of the activated platelets were also monitored simultaneously and showed changes in response to both agonists. The ability to measure changes in cytoplasmic free calcium by ratio flow cytofluorimetry provides a new approach to study of the role of alterations in intracellular calcium in response to agonists acting through different membrane receptors as well as providing a sensitive technique to detect functional subpopulations of platelets.