Statistical methodology: II. Reliability and variability assessment in study design, Part A

Validity and Reliability of Handheld Dynamometry to Assess Isometric Hamstrings and Quadriceps Strength at Varying Muscle Lengths

CONTEXT

The hamstrings are the most commonly injured muscle in sports and are especially injury prone in lengthened positions. Measuring knee muscle strength in such positions could be relevant to establish injury risk. Handheld dynamometry has been shown to be a valid, reliable, and practical tool to measure isometric muscle strength clinically. The aim of this study was to assess the validity and reliability of the assessment of isometric knee muscle strength with a handheld dynamometer (HHD) at various muscle lengths, by modifying the hip and knee angles during testing.

DESIGN

Concurrent validity and test-retest reliability.

METHODS

Thirty young healthy participants were recruited. Hamstring and quadriceps isometric strength was measured with a HHD and with an isokinetic dynamometer, over 2 testing sessions, in a randomized order. Isometric strength was measured on the right lower limb in 6 different positions, with the hip at either 0° or 80° of flexion and the knee at either 30°, 60°, or 90° of flexion. Pearson and Spearman correlations were used to assess the validity, and intraclass correlation coefficients were calculated to establish the test-retest reliability of the HHD.

RESULTS

Good to excellent reliability and moderate to high validity were found in all the tested muscle length positions, except for the hamstrings in a seated position with the knee extended at 30°.

CONCLUSIONS

The use of a HHD is supported in the clinical setting to measure knee muscle strength at varying muscle lengths in healthy adults, but not for the hamstrings in a lengthened position (hip flexed and knee extended). These results will have to be confirmed in sport-specific populations.

Perceptions of COVID-19 Vaccine, Racism, and Social Vulnerability: An Examination among East Asian Americans, Southeast Asian Americans, South Asian Americans, and Others

As COVID-19 vaccines are readily available and most U.S. adults who are enthusiastic about the vaccine have received it, motivating those who have not been vaccinated to accept it has become a challenge. The purpose of this study was to understand the mechanisms behind COVID-19 vaccine acceptance in Asian American ethnic groups, including how sociodemographic characteristics and racism predict COVID-19 and vaccine perceptions. The study also examined associations between social vulnerability and COVID-19 and vaccine perceptions. Social vulnerability is defined as the degree to which a community is able to prepare and respond to a natural or man-made disaster. This cross-sectional study used community-based survey data collected from April to September 2021. Study measures included demographics, perceptions of COVID-19 and COVID-19 vaccines, and racism-related experiences. The results showed that, compared to Non-Asians, East Asians reported that they had significantly more challenges accessing COVID-19 vaccines, and South Asians reported significantly higher safety concerns about COVID-19 vaccines. Our study also found that racism experience mediates the association between race/ethnicity and safety concerns about COVID-19 vaccines. Three Asian subgroups (East Asians, South Asians, and Southeast Asians) experienced more racism (compared to Non-Asians), and more experience of racism was related to greater safety concerns. Geographical Information System (GIS) maps revealed that residents of lower social vulnerability index (SVI) areas reported fewer unfairness perceptions and that higher SVI areas had lower vaccine accessibility and trust in public health agencies. Our study advances the understanding of racism, social vulnerability, and COVID-19 vaccine-related perceptions among Asian Americans. The findings have implications for policymakers and community leaders with respect to tailoring COVID-19 program efforts for socially vulnerable populations and Asian American groups that experience greater challenges regarding vaccine safety concerns and accessibility.

A systematic review of how adherence to caregiver facilitated therapeutic activities for children are assessed and reported in published research

This systematic review assessed four measurement properties of measures (instruments) used by researchers to assess adherence to caregiver facilitated therapeutic activities (CFTA) for children in published research. The measurement properties considered in this systematic review were dyadic considerations, interpretability, reliability, and validity of the instruments used to assess adherence or compliance. Two databases (http://www.pubmed.gov and EMBASE) were searched for studies that reported adherence or compliance to CFTA prescribed by physiotherapists or occupational therapists or speech and language therapists. Papers included in this review were those that studied children less than 18 years old and or their primary caregivers. Data were extracted by the authors using a data extraction table adapted from the work of Bollen and colleagues (2014) and by rating the measurement properties of the adherence measures identified based on predefined rating criteria (see supplementary file and Table 1 respectively). The authors relied only on the published research articles and any associated published supplementary files/data. None of the 40 adherence instruments identified wholly fulfilled all the criteria of the four measurement properties assessed. The results of this systematic review show that the measures used by researchers for assessing adherence to CFTA generally lack validity, reliability, and dyadic considerations.

Interrater Reliability among Novice Raters in the Assessment of Pelvic Floor Muscle Tone Using the Reissing Tone Scale

Purpose: The authors sought to determine the interrater reliability among novice raters of intra-vaginal manual assessment of pubococcygeus muscle tone in women using the Reissing tone scale (RTS). Method: Three graduating physiotherapy students (novice raters) and one experienced pelvic floor physiotherapist assessed 31 female participants (aged 20-66 y). Assessors gave RTS scores for pubococcygeus tone at three intra-vaginal locations (6:00, 9:00, and 3:00). Interrater reliability was determined for the novice raters using a two-way random single-measures absolute agreement intra-class correlation coefficient (ICC). Spearman rank correlation (SRC) analysis determined the correlation between the novice and expert scores. Results: The ICC values for the novice raters were 0.523, 0.274, and 0.336 at 6:00, 9:00, and 3:00, and the SRC values between the novice and expert raters were 0.580, 0.320, and 0.340. Conclusions: The novice raters demonstrated low to moderate interrater reliability for intra-vaginal manual assessment of pubococcygeus tone. This result indicates that manual assessment of pelvic floor muscle tone is not reliable enough to use as a stand-alone test to guide treatment, at least for physiotherapists with limited clinical experience.

Reliability, Validity, and Responsiveness of the Chinese Learning Accomplishment Profile (C-LAP)

Objectives: To evaluate the reliability, validity, and responsiveness of the Chinese Learning Accomplishment Profile in China. Methods: 12,098 participants aged from 0 to 36 months from 30 provinces (mostly from Shanghai) in China were enrolled between 2013 and 2020. The reliability was reflected by Pearson correlation coefficients, Cronbach’s alpha coefficients and standard errors; the validity was shown by the coefficients between the dimensions, and we also evaluated the responsiveness as a supplement to the validity. Results: Reliability: in six domains among each subgroup, Pearson correlation coefficients between developmental age and chronological age ranged from 0.89 to 0.98, Cronbach’s alpha coefficients from 0.71 to 0.99, and standard errors from 0.15 to 2.76. Validity: after controlling for chronological age, the correlation coefficients between the dimensions were between 0.18 and 0.78, and most of them were below 0.70. Responsiveness: developmental age of all domains obtained via the Chinese Learning Accomplishment Profile system changed significantly (p < 0.001) with time (gap of 1–3 months), and the standardized response mean ranged from 0.66 to 2.45. Conclusions: The Chinese Learning Accomplishment Profile is suitable for assessing children’s development in Shanghai, but still needs confirmation when used in other provinces in China due to the great differences between regions in China.

Virtual functional mobility test: A potential novel tool for assessing mobility of individuals with Parkinson's disease in a multitask condition

There are few instruments available for evaluating functional mobility during multitasking in people with Parkinson's Disease (PD). Virtual Reality is a potentially tool capable of aiding in the evaluation of functional mobility. The purpose of this study is to verify the potential of the Virtual Functional Mobility Test (VFMT) as a clinical tool to assess functional mobility of people with PD during multitasking condition. 25 people with PD and 25 people without PD, matched for age and sex, were recruited. Participants were evaluated through the Trail Making Test, Timed "UP and GO" test, Timed "UP and GO" test in dual task condition and through the VFMT, composed of 1) a simple task, and 2) a complex task. The VFMT and clinical tests were sensitive to differentiate the groups, except the trail making test part B (p = 0.332) and complex task (p = 0.052). Strong correlations were observed between parts A and B of the trail making test (r = 0.75) and complex task (r = 0.72); Moderate correlations between Timed Up and Go test and Timed Up and Go test in dual task condition with simple task (r = 0.47) and complex task (r = 0.55), respectively, were found. The complex task and simple task showed excellent and moderate reliability intra-rater, respectively. It was concluded that the novel VFMT is feasible, sensible, reliable and has potential as an instrument for the evaluation of functional mobility during multitasking in people with PD.

Construct validity and reliability of tests for sacroiliac dysfunction: standing flexion test (STFT) and sitting flexion test (SIFT)

CONTEXT

Sacroiliac dysfunction is characterized by a hypomobility of the range of motion of the joint, followed by a positional change regarding the relationship between the sacrum and the iliac. In general, the clinical tests that evaluate the sacroiliac joint (SIJ) and its dysfunctions lack validity and reliability values.

OBJECTIVES

This article aims to evaluate the construct validity and intra- and inter-rater reliability of the standing flexion test (STFT) and sitting flexion test (SIFT).

METHODS

In this prospective study, the sample consisted of 30 individuals of both sexes, and the evaluation team was composed of five researchers. The evaluations took place on two different days: first day, inter-rater reliability and construct validity; and second day, intra-rater reliability. The reference standard for the construct validity was 3-dimensional measurements obtained utilizing the BTS SMART-DX system. For statistical analysis, the percentage (%) agreement and the kappa statistic (K) were utilized.

RESULTS

The construct validity was determined for STFT (70% agreement; K=0.49; p<0.01) and SIFT (56.7% agreement; K=0.29; p<0.05). The intra-rater reliability was determined for STFT (66.3% agreement; K=0.43; p<0.01) and SIFT (56.7% agreement; K=0.38; p<0.01). The inter-rater reliability was determined for STFT (10% agreement; K=-0.02; p=0.825) and SIFT (13.3% agreement; K=0.01; p=0.836).

CONCLUSIONS

The STFT confirmed the construct validity and was reliable when applied by the same rater to healthy people, even if the rater had no experience. It was not possible to achieve minimum scores using the SIFT either for construct validity or reliability. We suggest that further studies be conducted to investigate the measurement properties of palpatory clinical tests for SIJ mobility, especially in symptomatic patients.

Simulated Patient Role-Plays with Consumers with Lived Experience of Mental Illness Post-Mental Health First Aid Training: Interrater and Test Re-Test Reliability of an Observed Behavioral Assessment Rubric

Mental Health First Aid (MHFA) training teaches participants how to assist people experiencing mental health problems and crises. Observed behavioral assessments, post-training, are lacking, and the literature largely focuses on self-reported measurement of behaviors and confidence. This study explores the reliability of an observed behavioral assessment rubric used to assess pharmacy students during simulated patient (SP) role-play assessments with mental health consumers. Post-MHFA training, pharmacy students (n = 528) participated in SP role-play assessments (n = 96) of six mental health cases enacted by consumers with lived experience of mental illness. Each assessment was marked by the tutor, participating student, and consumer (three raters). Non-parametric tests were used to compare raters' means scores and pass/fail categories. Interrater reliability analyses were conducted for overall scores, as well as pass/fail categories using intra-class correlation coefficient (ICC) and Fleiss' Kappa, respectively. Test re-test reliability analyses were conducted using Pearson's correlation. For interrater reliability analyses, the intra-class correlation coefficient varied from poor-to-good to moderate-to-excellent for individual cases but was moderate-to-excellent for combined cases (0.70; CI 0.58-0.80). Fleiss' Kappa varied across cases but was fair-to-good for combined cases (0.57, p < 0.001). For test re-test reliability analyses, Pearson's correlation was strong for individual and combined cases (0.87; p < 0.001). Recommended modifications to the rubric, including the addition of barrier items, scoring guides, and specific examples, as well as the creation of new case-specific rubric versions, may improve reliability. The rubric can be used to facilitate the measurement of actual, observed behaviors post-MHFA training in pharmacy and other health care curricula.

Assessment of Post-COVID-19 Quality of Life Using the Quality of Life Index

BACKGROUND

COVID-19 has impacted many aspects of peoples' quality of life (QOL).

AIM

To assess the QOL of post-COVID-19 cases and determine their associated sociodemographic attributes.

METHODS

A cross-sectional community-based study targeted recovered COVID-19 cases using an online questionnaire. An Arabic version of the Quality of Life Index (QLI) was evaluated for reliability and validity using Cronbach's alpha and Spearman's product moment tests, respectively. Descriptive analysis, Mann-Whitney U, Kruskal-Wallis H, and Spearman correlation tests were used to investigate the relationships between QOL and study independent variables, including sociodemographic characteristics, time since infection, intention for vaccination, perception of the influence of COVID-19 on general health, history of hospitalization, and history of chronic illness. Subgroups were compared using a Student's t-test for normally distributed data and the Wilcoxon rank-sum test or the Kruskal-Wallis test for non-normally distributed data.

RESULTS

In a sample of 449 respondents, the total QLI score was 22.64 ± 4.36; for subscales, it ranged from 20.86 ± 5.04 to 24.99 ± 4.74. Males had significantly higher health and functioning subscale (HFSUB) scores compared to females (p = 0.033). Employed cases had significantly higher social and economic subscale (SOCSUB) scores than unemployed cases (p = 0.000). Married people reported significantly higher family subscale (FAMSUB) scores (p = 0.000) than unmarried people. People who believe that their general health was affected by COVID-19 had significantly lower HFSUB and psychological and spiritual subscale (PSPSUB) scores (p = 0.000 and p = 0.021, respectively). Individuals with a history of hospital admissions scored lower on PSPSUB (p = 0.000). Age was significantly associated with both SOCSUB (p = 0.006) and FAMSUB (p = 0.040).

CONCLUSION

Sociodemographic attributes of people with a history of COVID-19 revealed significant differences in some QOL domains, as measured by the Arabic version of the generic QLI. History of hospitalization and beliefs about COVID-19's impact on general health negatively influenced individual's HFSUB and PSPSUB.

Telemedicine via Smart Glasses in Critical Care of the Neurosurgical Patient-COVID-19 Pandemic Preparedness and Response in Neurosurgery

OBJECTIVE

The coronavirus disease 2019 pandemic poses major risks to health care workers in neurocritical care. Recommendations are in place to limit medical personnel attending to the neurosurgical patient as a protective measure and to conserve personal protective equipment. However, the complexity of the neurosurgical patient proves to be a challenge and an opportunity for innovation. The goal of our study was to determine if telemedicine delivered through smart glasses was feasible and effective in an alternative method of conducting ward round on neurocritical care patients during the pandemic.

METHODS

A random pair of neurosurgery resident and specialist conducted consecutive virtual and physical ward rounds on neurocritical patients. A virtual ward round was first conducted remotely by a specialist who received real-time audiovisual information from a resident wearing smart glasses integrated with telemedicine. Subsequently, a physical ward round was performed together by the resident and specialist on the same patient. The management plans of both ward rounds were compared, and the intrarater reliability was measured. On study completion a qualitative survey was performed.

RESULTS

Ten paired ward rounds were performed on 103 neurocritical care patients with excellent overall intrarater reliability. Nine out of 10 showed good to excellent internal consistency, and 1 showed acceptable internal consistency. Qualitative analysis indicated wide user acceptance and high satisfaction rate with the alternative method.

CONCLUSIONS

Virtual ward rounds using telemedicine via smart glasses on neurosurgical patients in critical care were feasible, effective, and widely accepted as an alternative to physical ward rounds during the coronavirus disease 2019 pandemic.

Development and Validation of Food and Nutrition Literacy Assessment Tool for Iranian High-school Graduates and Youth

Background:

Food and nutrition literacy (FNL) is an emerging concept that emphasizes not only on personal knowledge, but food and nutrition skills about. This study aimed to develop and validate a food and nutrition literacy assessment tool (FNLAT) for youth and high-school graduates in Iran.

Methods:

The study protocol included the following steps: First, FNL components for Iranian high-school graduates and youth were identified through literature review and interviews with experts. Delphi method was used in order to achieve consensus about FNL components. Then, the questionnaire items were generated, and its content and face validity were assessed. Construct validity of the questionnaire was evaluated through applying principal component analysis (PCA) and confirmatory factor analyses (CFA) in the next step. Finally, reliability of the FNLAT was assessed by calculating Cronbach's Alpha and evaluating test-retest reliability.

Results:

A 104-item questionnaire was developed. S-CVI was ≥90 which confirmed content validity of the questionnaire. PCA suggested that it was constructed of 6 factors, one in knowledge domain (food and nutrition knowledge) and five in skill domain (functional skills, interactive skills, advocacy, critical analysis of information, and food label reading skills). On the basis of CFA, the fit indices of the model had acceptable fit and confirmed construct validity of the FNLAT (X²/df = 1.58, RMSEA = 0.041; P = 1.00, RMR = 0.034, GFI = 0.79). The values of Cronbach's Alpha and intraclass correlation coefficient (ICC) confirmed internal consistency and time stability of the FNLAT and its subscales.

Conclusions:

The developed FNLAT is a valid and reliable tool to assess FNL in Iranian late adolescents and youth.

Interrater and Intrarater Reliability of the Colloid Cyst Risk Score

BACKGROUND

The Colloid Cyst Risk Score (CCRS) was developed to identify symptomatic patients and stratify risk of hydrocephalus among patients with colloid cysts. Its components consider patient age, cyst diameter, presence/absence of headache, fluid-attenuated inversion recovery (FLAIR) hyperintensity, and location within the third ventricle.

OBJECTIVE

To independently evaluate the inter- and intrarater reliability of the CCRS.

METHODS

Patients with a colloid cyst were identified from billing records and radiology archives. Three independent raters reviewed electronic medical records to determine age, presence/absence of headache, cyst diameter (mm), FLAIR hyperintensity, and risk zone location. Raters made 53 observations, including 5 repeat observations.Fleiss' generalized kappa (κ) was calculated for all of the nominal criteria, whereas Kendall's coefficient of concordance (W) and the intraclass correlation coefficient (ICC) were calculated for the overall score.

RESULTS

Total CCRS score demonstrated extremely strong agreement (W = 0.83) using Kendall's W coefficient and good agreement (ICC = 0.74) using the ICC (P < .001). For interrater reliability of individual criteria, age (κ = 1.00) and FLAIR hyperintensity (κ = 0.89) demonstrated near perfect agreement. Axial diameter (κ = 0.63) demonstrated substantial agreement, whereas agreement was moderate for risk zone (κ = 0.51) and fair for headache (κ = 0.26). Intrarater reliability for total CCRS score was extremely strong using Kendall's W, good to excellent using ICC, and fair to substantial using weighted kappa.

CONCLUSION

The CCRS has good inter- and intrarater reliability when tested in an independent sample of patients, though strength of agreement varies among individual criteria. The validity of the CCRS requires independent evaluation.

Concurrent Validity of a Novel Wireless Inertial Measurement System for Assessing Trunk Impairment in People with Stroke

Background: The Trunk Impairment Scale (TIS) is recommended for clinical research use to assess trunk impairment post-stroke. However, it is observer-dependent and neglects the quality of trunk movements. This study proposes an instrumented TIS (iTIS) using the Valedo system, comprising portable inertial sensors, as an objective measure of trunk impairment post-stroke. Objective: This study investigates the concurrent and discriminant ability of the iTIS in chronic stroke participants. Method: Forty participants (20 with chronic stroke, 20 healthy, age-matched) were assessed using the TIS and iTIS simultaneously. A Spearman rank correlation coefficient was used to examine concurrent validity. A ROC curve was used to determine whether the iTIS could distinguish between stroke participants with and without trunk impairment. Results: A moderate relationship was found between the observed iTIS parameters and the clinical scores, supporting the concurrent validity of the iTIS. The small sample size meant definitive conclusions could not be drawn about the parameter differences between stroke groups (participants scoring zero and one on the clinical TIS) and the parameter cut-off points. Conclusion: The iTIS can detect small changes in trunk ROM that cannot be observed clinically. The iTIS has important implications for objective assessments of trunk impairment in clinical practice.

Symptom-Disease Pair Analysis of Diagnostic Error (SPADE): a conceptual framework and methodological approach for unearthing misdiagnosis-related harms using big data

BACKGROUND

The public health burden associated with diagnostic errors is likely enormous, with some estimates suggesting millions of individuals are harmed each year in the USA, and presumably many more worldwide. According to the US National Academy of Medicine, improving diagnosis in healthcare is now considered 'a moral, professional, and public health imperative.' Unfortunately, well-established, valid and readily available operational measures of diagnostic performance and misdiagnosis-related harms are lacking, hampering progress. Existing methods often rely on judging errors through labour-intensive human reviews of medical records that are constrained by poor clinical documentation, low reliability and hindsight bias.

METHODS

Key gaps in operational measurement might be filled via thoughtful statistical analysis of existing large clinical, billing, administrative claims or similar data sets. In this manuscript, we describe a method to quantify and monitor diagnostic errors using an approach we call 'Symptom-Disease Pair Analysis of Diagnostic Error' (SPADE).

RESULTS

We first offer a conceptual framework for establishing valid symptom-disease pairs illustrated using the well-known diagnostic error dyad of dizziness-stroke. We then describe analytical methods for both look-back (case-control) and look-forward (cohort) measures of diagnostic error and misdiagnosis-related harms using 'big data'. After discussing the strengths and limitations of the SPADE approach by comparing it to other strategies for detecting diagnostic errors, we identify the sources of validity and reliability that undergird our approach.

CONCLUSION

SPADE-derived metrics could eventually be used for operational diagnostic performance dashboards and national benchmarking. This approach has the potential to transform diagnostic quality and safety across a broad range of clinical problems and settings.

Test-retest stability of patient experience items derived from the national GP patient survey

Purpose

The validity and reliability of various items on the GP Patient Survey (GPPS) survey have been reported, however stability of patient responses over time has not been tested. The purpose of this study was to determine the test–retest reliability of the core items from the GPPS.

Methods

Patients who had recently consulted participating GPs in five general practices across the South West England were sent a postal questionnaire comprising of 54 items concerning their experience of their consultation and the care they received from the GP practice. Patients returning the questionnaire within 3 weeks of mail-out were sent a second identical (retest) questionnaire. Stability of responses was assessed by raw agreement rates and Cohen’s kappa (for categorical response items) and intraclass correlation coefficients and means (for ordinal response items).

Results

348 of 597 Patients returned a retest questionnaire (58.3 % response rate). In comparison to the test phase, patients responding to the retest phase were older and more likely to have white British ethnicity. Raw agreement rates for the 33 categorical items ranged from 66 to 100 % (mean 88 %) while the kappa coefficients ranged from 0.00 to 1.00 (mean 0.53). Intraclass correlation coefficients for the 21 ordinal items averaged 0.67 (range 0.44–0.77).

Conclusions

Formal testing of items from the national GP patient survey examining patient experience in primary care highlighted their acceptable temporal stability several weeks following a GP consultation.

Physical activity and healthy eating environmental audit tools in youth care settings: A systematic review

BACKGROUND

There is a growing interest in evaluating the physical activity (PA) and healthy eating (HE) policy and practice environment characteristics in settings frequented by youth (≤18years).

OBJECTIVE

This review evaluates the measurement properties of audit tools designed to assess PA and HE policy and practice environment characteristics in settings that care for youth (e.g., childcare, school, afterschool, summer camp).

METHOD

Three electronic databases, reference lists, educational department and national health organizations' web pages were searched between January 1980 and February 2014 to identify tools assessing PA and/or HE policy and practice environments in settings that care for youth (≤18years).

RESULTS

Sixty-five audit tools were identified of which 53 individual tools met the inclusion criteria. Thirty-three tools assessed both the PA and HE domains, 6 assessed the PA domain and 14 assessed the HE domain solely. The majority of the tools were self-assessment tools (n=40), and were developed to assess the PA and/or HE environment in school settings (n=33), childcare (n=12), and after school programs (n=4). Four tools assessed the community at-large and had sections for assessing preschool, school and/or afterschool settings within the tool. The majority of audit tools lacked validity and/or reliability data (n=42). Inter-rater reliability and construct validity were the most frequently reported reliability (n=7) and validity types (n=5).

CONCLUSIONS

Limited attention has been given to establishing the reliability and validity of audit tools for settings that care for youth. Future efforts should be directed towards establishing a strong measurement foundation for these important environmental audit tools.

Measuring Food Brand Awareness in Australian Children: Development and Validation of a New Instrument

BACKGROUND

Children's exposure to food marketing is one environmental determinant of childhood obesity. Measuring the extent to which children are aware of food brands may be one way to estimate relative prior exposures to food marketing. This study aimed to develop and validate an Australian Brand Awareness Instrument (ABAI) to estimate children's food brand awareness.

METHODS

The ABAI incorporated 30 flashcards depicting food/drink logos and their corresponding products. An abbreviated version was also created using 12 flashcards (ABAI-a). The ABAI was presented to 60 primary school aged children (7-11 yrs) attending two Australian after-school centres. A week later, the full-version was repeated on approximately half the sample (n=27) and the abbreviated-version was presented to the remaining half (n=30). The test-retest reliability of the ABAI was analysed using Intra-class correlation coefficients. The concordance of the ABAI-a and full-version was assessed using Bland-Altman plots. The 'nomological' validity of the full tool was investigated by comparing children's brand awareness with food marketing-related variables (e.g. television habits, intake of heavily promoted foods).

RESULTS

Brand awareness increased with age (p<0.01) but was not significantly correlated with other variables. Bland-Altman analyses showed good agreement between the ABAI and ABAI-a. Reliability analyses revealed excellent agreement between the two administrations of the full-ABAI.

CONCLUSIONS

The ABAI was able to differentiate children's varying levels of brand awareness. It was shown to be a valid and reliable tool and may allow quantification of brand awareness as a proxy measure for children's prior food marketing exposure.

Inter-rater reliability for noninvasive measurement of cardiac function in children

INTRODUCTION

A transcutaneous ultrasound monitor has recently been developed which noninvasively and quickly measures cardiac output. Validity and reliability testing has been reported in adults. No reliability testing has been undertaken in the pediatric population.

OBJECTIVE

Our objective was to evaluate the inter-rater reliability of a transcutaneous Doppler ultrasound technique to measure cardiac index (CI) and stroke volume index (SVI) in pediatric emergency department patients.

METHODS

An 8-month prospective observational study was conducted on a convenience sample of emergency department patients younger than 18 years old. Five raters were trained to use an ultrasound cardiac output monitoring device. Two raters, blinded to each other's results, obtained independent measurements from the same patient within 15 minutes of each other. Inter-rater agreement was measured with the Pearson product correlation coefficient. Bland-Altman analysis demonstrated the extent of deviation from a line of agreement between raters.

RESULTS

Ninety-seven patients were enrolled. Major diagnostic categories included infection, trauma, and gastrointestinal disorders. There was significant inter-rater correlation for CI (r = 0.76; 95% confidence interval, 0.66Y0.83; P G 0.0001) and SVI (r = 0.79; 95% confidence interval, 0.70Y0.86; P G 0.0001). Bland-Altman analysis of CI measurements between 2 raters showed bias of 0.06, SD of bias 1.00, and 95% limits of agreement j1.91 to 2.02 L/min/m2. Stroke volume index showed bias of j0.5, SD of bias 11.01, and 95% limits of agreement j22.08 to 21.08 mL/m2.

CONCLUSIONS

Transcutaneous Doppler ultrasound technique demonstrates acceptable inter-rater agreement for measuring CI and SVI in children.

Data collection in the emergency setting

This is the ninth article in the research series and focuses on the collection of data.

Course of low back pain among nurses: a longitudinal study across eight years

AIMS

To describe the course of low back pain (LBP) among nurses across eight years.

METHODS

A longitudinal study was performed with a follow up at 1 and 8 years among nurses employed by a large university hospital in Switzerland. A modified version of the Nordic Questionnaire was distributed to obtain information about demographic data, occupational activities, and various aspects of LBP. A clinical examination and several functional tests were used to overcome the problems associated with subjective pain reporting. Nurses having answered the questionnaire on all three occasions (n = 269) were classified into subgroups according to their pain intensity. For each subgroup the course of LBP was recorded.

RESULTS

LBP was highly prevalent with an annual prevalence varying from 73% to 76%. A large percentage (38%) indicated the same intensity of LBP on all three occasions. The proportion of nurses reporting repeated increase of LBP (19%) was approximately as large as the proportion who complained about repeated decrease of LBP (17%).

CONCLUSION

It became evident that LBP poses a persistent problem among nurses. Over an eight year period almost half of the nurses indicated the same intensity of LBP, thus supporting a recurrent rather than a progressive nature of LBP.

Measures of low back function: a review of reproducibility studies

OBJECTIVE

The objective of the present study was to make a systematic literature review with preset quality criteria concerning reproducibility of the tests of the low back regarding strength, endurance and range of motion.

DESIGN

Literature in Medline and local databases was reviewed for articles concerning the reproducibility of strength, endurance, and range of motion measurements.

BACKGROUND

Measures of low back function are widely used, and are important for both clinical and research purposes in relation to low back problems. A review of the reproducibility of these tests has not previously been made.

METHODS

After extensive discussion among all the authors, general evaluation parameters were defined for the quality assessment. Every study was graded from 0 to 2 for each parameter. Parameters evaluated were: number of subjects, subject description, method description, test/retest interval, description of results, and statistics.

RESULTS

The literature search revealed a total of 79 studies. Most studies suffered from methodological weaknesses and only eleven studies received ten or more quality points (maximum 14). The results from the highest graded studies are highlighted.

CONCLUSIONS

It may be concluded that there is a considerable lack of information about the reproducibility of functional measures for the low back, and therefore a recommendation for consensus is difficult. However, most tests performed in the sagittal plane are reliable for use on groups.

RELEVANCE

Measures of low back function are thought to be of great importance for clinicians, and low back researchers in general. A review of reproducibility will be helpful both as a survey of tests, and to provide information on the usefulness in relation to the level of reproducibility.

Assessment of cardiac stress from massive pulmonary embolism with 12-lead ECG

BACKGROUND

Massive pulmonary embolism (PE) that causes severe pulmonary hypertension can produce specific ECG abnormalities. We hypothesized that an ECG scoring system would vary in proportion to the severity of pulmonary hypertension and would help to distinguish patients with massive PE from patients with smaller PE and those without PE.

METHODS

A 21-point ECG scoring system was derived (relative weights in parentheses): sinus tachycardia (2), incomplete right bundle branch block (2), complete right bundle branch block (3), T-wave inversion in leads V(1) through V(4) (0 to 12), S wave in lead I (0), Q wave in lead III (1), inverted T in lead III (1), and entire S(1)Q(3)T(3) complex (2). ECGs obtained within 48 h prior to pulmonary arteriography were located for 60 patients (26 positive for PE, 34 negative for PE) and for 25 patients with fatal PE.

RESULTS

Interobserver agreement (11 readers) for ECG score was good (Spearman r = 0.74). The ECG score showed significant positive relationship to systolic pulmonary arterial pressure (sPAP) in patients with PE (r = 0.387, p < 0.001), whereas no significant relationship was seen in patients without PE (r = - 0.08, p = 0.122). When patients were grouped by severity of pulmonary hypertension (low, moderate, severe), only patients with severe pulmonary hypertension from PE had a significantly higher ECG score (mean, 5.8 +/- 4.9). At a cutoff of 10 points, the ECG score was 23.5% (95% confidence interval [CI], 16 to 31%) sensitive and 97.7% (95% CI, 96 to 99%) specific for the recognition of severe pulmonary hypertension (sPAP > 50 mm Hg) secondary to PE. In 25 patients with fatal PE, the ECG score was 9.5 +/- 5.2.

CONCLUSIONS

The derived ECG score increases with severity of pulmonary hypertension from PE, and a score > or = 10 is highly suggestive of severe pulmonary hypertension from PE.

Pulseless electrical activity with witnessed arrest as a predictor of sudden death from massive pulmonary embolism in outpatients

BACKGROUND

the objective was to determine clinical characteristics that can quickly distinguish sudden death from massive pulmonary embolism (MPE) from other causes of sudden death.

METHODS AND RESULTS

all medical examiner reports from Charlotte, NC from 1992 to 1999 (n=4926) were hand-searched for cases of sudden death which met the inclusion criteria: non-traumatic death, age 18-65 years, transported to an emergency department (ED), and autopsy performed. Supplemental data from ED and prehospital records were retrieved to complete documentation. Data were analyzed by univariate odds ratios (OR) followed by chi-square (chi(2)) recursive partitioning for decision tree construction. Three hundred eighty four cases met inclusion criteria; MPE was the second most frequent cause of cardiac arrest in this cohort (37/384, 9.6%). The mean age of subjects with MPE (40.2+/-11.1 years) was significantly lower compared with non-PE subjects (46.5+/-9.9 years). Pulseless electrical activity was observed as the initial arrest rhythm (primary PEA) in 52/384 (13.5%) subjects. Out of 52 subjects with primary PEA, 28 (53%) died from MPE. Odds ratio data indicated significant association of MPE with female gender, arrest witnessed by medical providers, presence of primary PEA, and return of spontaneous circulation. The most accurate decision rule to recognize MPE consisted of witnessed arrest+primary PEA. This rule generated sensitivity=67.6% and specificity=94.5% and yielded a posttest probability of MPE of 57%.

CONCLUSIONS

outpatients with witnessed cardiac arrest and primary PEA carry a high probability of MPE.

Ability of patients to accurately recall the severity of acute painful events

OBJECTIVE

Pain studies require prospective patient enrollment to ensure accurate pain assessment. The authors correlated pain assessments of an acute painful episode over a one-week period and determined the accuracy of patient pain severity recall over time.

METHODS

This was a prospective, descriptive, longitudinal study. Participants were a convenience sample of 50 emergency department patients with acute pain resulting from injuries or painful invasive procedures. A trained research assistant administered a structured pain survey containing demographic and historical features to all patients. Patients sequentially assessed their pain severity using a vertical 100-mm visual analog scale marked "most pain" at the top, a verbal numeric rating scale ranging from 0 to 100 from none to worst (NRS100), and a verbal numeric rating scale ranging from 0 to 10 from none to worst (NRS10). Patients were contacted by phone and asked to reassess their initial pain severity one and seven days later using the two verbal numeric rating scales. Analysis of pain assessments using the various scales at the three time intervals was performed with Pearson's and Spearman's coefficients and repeated-measures analysis of variance (ANOVA).

RESULTS

There were 50 patients with a mean age of 41 years. Correlation between initial pain assessments on the three scales ranged from 0.83 to 0.92. Correlations between the initial and 24-hour assessments were NRS100-0.98 and NRS10-0.98. Correlations between the initial and one-week assessments were NRS100-0.96 and NRS10-0.97. Repeated-measures ANOVA showed no significant change in pain assessments over time for both verbal numeric scales.

CONCLUSIONS

Pain severity assessments of acute painful events one and seven days later were similar and highly correlated with initial assessments using both verbal numeric scales. Patients accurately recall the severity of an acute painful episode for at least one week after its occurrence, which may allow retrospective pain assessments.

Role of the peripheral intravenous catheter in false-positive D-dimer testing

OBJECTIVE

To determine whether inserting a peripheral intravenous catheter (IV) can significantly increase the circulating D-dimer concentration.

METHODS

Twenty healthy young adult volunteers underwent cannulation of an antecubital vein with a 20-gauge Teflon IV. Time 0 venous blood was drawn during IV insertion. The IV was salinelocked and left in place for 90 minutes, at which time a second venipuncture was performed in a contralateral antecubital vein (+90 min). A qualitative D-dimer assay [erythrocyte-agglutination assay, SimpliRED (SRDD)] and a quantitative spectrophotometric assay [enzyme-linked immunosorbent assay (EIA), Dimertest Gold] were performed on all samples. Time 0 means (+/-SD) were compared with +90 min means by paired t-test, and SRDD pairs were compared with McNemar's test.

RESULTS

Time 0 initial venipuncture blood samples yielded a mean D-dimer concentration of 15 +/- 24 ng/mL, with 2/20 SRDD tests read as positive (95% CI = 1% to 32%). At +90 min, the D-dimer concentration was 33 +/- 21 ng/mL (p = 0.04 vs time 0), with 5/20 SRDD tests read as positive (95% CI = 9% to 49%, p = 0.248).

CONCLUSIONS

Insertion of an IV increased the circulating D-dimer concentration (determined by EIA), but did not lead to a significant increase in false-positive conversion of the SRDD. An effort should be made to perform D-dimer testing on "first-stick" blood to optimize specificity. However, a strongly positive D-dimer reaction cannot be ascribed to the presence of an IV.

Research fundamentals: selection and development of clinical outcome measures

Clinical outcomes measure patient health or well-being. The choice of an outcome measure for use in a clinical trial or study is complex. It is even more difficult when no appropriate outcome measure exists, necessitating the development of a novel one. The ideal clinical outcome should be credible, comprehensive, sensitive to change, accurate, biologically sensible, and feasible. This paper describes the attributes of clinical outcomes and illustrates how to develop novel outcomes, using as an example the authors' experience in developing a wound cosmesis outcome measure.

Neural network analysis of the volumetric capnogram to detect pulmonary embolism

BACKGROUND

Pulmonary embolism (PE) produces ventilation/perfusion mismatch that may be manifested in various variables of the volume-based capnogram (VBC). We hypothesized that a neural network (NN) system could detect changes in VBC variables that reflect the presence of a PE.

METHODS

A commercial VBC system was used to record multiple respiratory variables from consecutive expiratory breaths. Data from 12 subjects (n = 6 PE+ and n = 6 PE-) were used as input to a fully connected back-propagating NN for model development. The derived model was tested in a prospective, observational study at an urban teaching hospital. Volumetric capnograms were then collected on 53 test subjects: 30 subjects with PE confirmed by pulmonary angiography or diagnostic scintillation lung scan, and 23 subjects without PE based on pulmonary angiography. The derived NN model was applied to VBC data from the test population.

RESULTS

Seventeen VBC variables were used by the derived NN model to generate a numeric probability of PE. When the derived NN model was applied to VBC data from the 53 test subjects, PE was detected with a sensitivity of 100% (95% CI = 89% to 100%) and a specificity of 48% (95% CI = 27% to 69%). The likelihood ratio positive [LR(+)] for the VBC-NN test was 1.82 and the LR (-) was 0.1.

CONCLUSION

This study demonstrates the feasibility of developing a rapid, noninvasive breath test for diagnosing PE using volumetric capnography and NN analysis.

A hierarchical outcomes approach to test assessment

This report describes a hierarchical classification system for clinical endpoints in diagnostic technology assessment. The model requires classification of investigative outcomes into 1 of 6 categories, including technical efficacy, diagnostic accuracy efficacy, diagnostic thinking efficacy, therapeutic efficacy, clinical outcome efficacy, and societal efficacy. Evaluations at successive levels indicate a broader understanding of the test's value. The purpose of this classification system is to help readers of the medical literature consider various aspects of test performance, and to identify aspects of a test that require additional investigation. It is also designed to be used as a template for systematic reviews of diagnostic tests. In this review, examples from the medical literature are discussed to highlight important concepts from the hierarchy model and their relevance to emergency medicine.

Effect of skin pigmentation on pulse oximetry accuracy in the emergency department

OBJECTIVE

To determine whether pulse oximeter (PO) accuracy and signal quality are affected by level of skin pigmentation.

METHODS

Observational study in a community hospital ED. Consecutive adult patients undergoing arterial blood gas determination were enrolled into the study. Skin pigmentation was determined by comparison with standardized color swatches under controlled lighting; assigned values were used to stratify patients into 3 groups (light, intermediate, and dark) using predetermined criteria. Simultaneous with arterial blood sampling, staff recorded PO reading of O2 saturation using the Nellcor D-25 oximeter. PO values were compared with criterion standard values measured using a 4-wavelength spectrophotometer or co-oximeter. PO signal quality also was recorded. Bias (the mean difference between PO and co-oximeter-measured values of hemoglobin saturation) and precision (the standard deviation of the bias) were calculated. Groups were compared using one-way ANOVA, Bartlett's test for variances, and chi2 test.

RESULTS

O2 saturation data were obtained for 284 patients. Bias values did not differ between the 3 skin pigment groups (p = 0.79). Precision was of borderline significance (p = 0.05), but there was no dose-response relation between skin pigmentation and precision. Study personnel reported suboptimal PO function most often among patients in the dark group (p = 0.003), but this finding was of no clinical significance. PO signal failure was rare (<1% of all patients).

CONCLUSIONS

Although several prior studies suggest the contrary, this study found that skin pigmentation does not affect the bias or precision of pulse oximetry. Furthermore, skin pigmentation has no clinically significant effect on PO signal quality.

Statistical methodology: II. Reliability and validity assessment in study design, Part B

Validity measures the correspondence between a test and other purported measures of the same or similar qualities. When a reference standard exists, a criterion-based validity coefficient can be calculated. If no such standard is available, the concepts of content and construct validity may be used, but quantitative analysis may not be possible. The Pearson and Spearman tests of correlation are often used to assess the correspondence between tests, but do not account for measurement biases and may yield misleading results. Techniques that measure interest differences may be more meaningful in validity assessment, and the kappa statistic is useful for analyzing categorical variables. Questionnaires often can be designed to allow quantitative assessment of reliability and validity, although this may be difficult. Inclusion of homogeneous questions is necessary to assess reliability. Analysis is enhanced by using Likert scales or similar techniques that yield ordinal data. Validity assessment of questionnaires requires careful definition of the scope of the test and comparison with previously validated tools.