Revisiting the Sample Size and Statistical Power of Randomized Controlled Trials in Orthopaedics After 2 Decades

Factors influencing study outcomes in recent literature on distal radial fracture treatment

The main tools used by an orthopaedic surgeon for managing distal radial fracture treatment are recent literature and treatment guidelines. The aim of the present study was to find which factors within the study design influence study outcomes the most. Trials in three major databases (PubMed, Scopus, Embase) comparing surgical and non-surgical treatment options for adolescent and adult distal radial fractures with their original data, between 2013 and 2021, were included. The selected 47 studies were classified according to their outcomes. The relationship between study characteristics and outcomes was statistically analysed. It was more likely to find no difference in outcomes between volar locking plate and less invasive treatments when the sample size was above 100, follow-up was more than 1 year and functional assessments were used. A small sample size and short follow-up time affect study outcomes in favour of a volar locking plate. Readers should focus on the design criteria and read the full text of the studies before making any conclusions.

LEVEL OF EVIDENCE

III.

MCID and PASS in Knee Surgeries. Theoretical Aspects and Clinical Relevance References

The application and interpretation of patient-reported outcome measures (PROM), following knee injuries, pathologies, and interventions, can be challenging. In recent years, the literature has been enriched with metrics to facilitate our understanding and interpretation of these outcome measures. Two commonly utilized tools include the minimal clinically important difference (MCID) and the patient acceptable symptoms state (PASS). These measures have demonstrated clinical value, however, they have often been under- or mis-reported. It is paramount to use them to understand the clinical significance of any statistically significant results. Still, it remains important to know their caveats and limitations. In this focused report on MCID and PASS, their definitions, methods of calculations, clinical relevance, interpretations, and limitations are reviewed and presented in a simple approach.

Choosing the appropriate measure of central tendency: mean, median, or mode?

Mean, median, and mode are among the most basic and consistently used measures of central tendency in statistical analysis and are crucial for simplifying data sets to a single value. However, there is a lack of understanding of when to use each metric and how various factors can impact these values. The aim of this article is to clarify some of the confusion related to each measure and explain how to select the appropriate metric for a given data set. The authors present this work as an educational resource, ensuring that these common statistical concepts are better understood throughout the Orthopedic research community.

A Systematic Review of Outcome Measures in Orthopaedic Trauma Trials: What Are We Measuring?

PURPOSE

The objectives of this study are to describe the outcome measures used in orthopaedic fracture care trials, with a particular focus on patient-reported outcome measures (PROMs), and to determine which study characteristics are associated with number of citations.

METHODS

We retrieved randomized clinical trials on fracture care between 2012 and 2017 from Embase, Medline and CENTRAL databases. Data collected included study characteristics (e.g., region, design, setting, sample size) and outcome measures (e.g., primary variable, measurement perspective, use of PROMs, study results and number of citations).

RESULTS

We identified a total of 8,580 articles in the initial search. After title screening, abstract screening and full-text review, we included 416 articles for analysis. 58.4% (243) of the studies clearly defined a primary outcome measure and 56.3% (234) reported sample size justifications for outcome selection. The most common primary outcome reported was a visual analogue scale for pain; used in 21 of the 243 (8.6%) studies that defined a primary outcome. At least one PROM was used in 68.5% (285) of the papers included.

CONCLUSIONS

A large proportion of studies reporting on PROMs for orthopaedic trauma patients do not provide key information on the outcome selection process; a step of utmost importance in and the designing and reporting of RCTs. There is substantial heterogeneity in the selection of PROMs for fracture care trials, which limits the ability to compare and summarize across studies. Future research in fracture care should strive towards improving the reporting of informative PROMs, with rationale that demonstrates understating of the injury, intervention and patient values.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s43465-022-00667-8.

Sample size justifications in Gait & Posture

BACKGROUND

Context regarding how researchers determine the sample size of their experiments is important for interpreting the results and determining their value and meaning. Between 2018 and 2019, the journal Gait & Posture introduced a requirement for sample size justification in their author guidelines.

RESEARCH QUESTION

How frequently and in what ways are sample sizes justified in Gait & Posture research articles and was the inclusion of a guideline requiring sample size justification associated with a change in practice?

METHODS

The guideline was not in place prior to May 2018 and was in place from 25th July 2019. All articles in the three most recent volumes of the journal (84-86) and the three most recent, pre-guideline volumes (60-62) at time of preregistration were included in this analysis. This provided an initial sample of 324 articles (176 pre-guideline and 148 post-guideline). Articles were screened by two authors to extract author data, article metadata and sample size justification data. Specifically, screeners identified if (yes or no) and how sample sizes were justified. Six potential justification types (Measure Entire Population, Resource Constraints, Accuracy, A priori Power Analysis, Heuristics, No Justification) and an additional option of Other/Unsure/Unclear were used.

RESULTS

In most cases, authors of Gait & Posture articles did not provide a justification for their study's sample size. The inclusion of the guideline was associated with a modest increase in the percentage of articles providing a justification (16.6-28.1%). A priori power calculations were the dominant type of justification, but many were not reported in enough detail to allow replication.

SIGNIFICANCE

Gait & Posture researchers should be more transparent in how they determine their sample sizes and carefully consider if they are suitable. Editors and journals may consider adding a similar guideline as a low-resource way to improve sample size justification reporting.

Inconsistencies in the Methodological Framework Throughout Published Studies in High-Impact Orthopaedic Journals: A Systematic Review

BACKGROUND

Both the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) and Consolidated Standards of Reporting Trials (CONSORT) guidelines recommend that clinical trials follow a study framework that aligns with their objective to test the relative efficacy or safety (equality) or effectiveness (superiority, noninferiority, or equivalence) between interventions. We conducted a systematic review to assess the proportion of studies that demonstrated inconsistency between the framing of their research question, sample size calculation, and conclusion and those that should have framed their research question differently based on the compared interventions.

METHODS

We included studies from 5 high-impact-factor orthopaedic journals published in 2017 and 2019 that compared at least 2 interventions using patient-reported outcome measures.

RESULTS

We included 228 studies. The sample size calculation was reported in 60.5% (n = 138) of studies. Of these, 52.2% (n = 72) were inconsistent between the framing of their research question, sample size calculation, and conclusion. The majority (n = 137) of sample size calculations were for equality, but 43.8% of these studies concluded superiority, noninferiority, or equivalence. Studies that framed their research question as equality (n = 186) should have been framed as superiority (n = 129), equivalence (n = 52), or noninferiority (n = 3). Only 2 studies correctly framed their research question as equality.

CONCLUSIONS

Studies published in high-impact journals were inconsistent between the framing of their research question, sample size calculation, and conclusion. Authors may be misinterpreting research findings and making clinical recommendations solely based on p values. Researchers are encouraged to state and justify their methodological framework and choice of margin(s) in a publicly published protocol as they have implications for sample size and the applicability of conclusions.

CLINICAL RELEVANCE

The results of clinical research must be interpreted using confidence intervals, with careful consideration as to how the confidence intervals relate to clinically meaningful differences in outcomes between treatments. The more typical practice of relying on p values leaves the clinician at high risk of erroneous interpretation, recommendation, and/or action.

Common errors in the design of orthopaedic trials: Has anything changed?

INTRODUCTION

The adoption of evidence-based orthopaedics has shifted the focus from expert base opinions and anecdotal evidence to a focus on integrating the best available clinical research. This shift has led to an increased focus on randomized controlled trials (RCTs) within the field. Although RCTs are considered the highest level of evidence, methodologic errors can introduce bias and limit the validity of the results. Early trials were hampered by lack of blinding, inadequate sample sizes and other design flaws. The objective of this review was to examine the current literature to determine if the design and execution of RCTs has improved.

DESIGN ERRORS

The awareness of the importance of sample size increased over time with substantially more trials reporting sample size calculations. However, many contemporary RCTs are still underpowered and fail to reach their calculated sample size. Given the challenges of surgically based RCTs, the majority of historical trials lacked blinding, increasing the risk of bias. There is evidence that there has been a concerted effort to increase the blinding in RCTs, particularly in outcome assessors. A more recent development in the design of surgical trials is the introduction of expertise-based trial designs in which patients are randomized to a surgeon with expertise in a particular intervention. These trials minimize the bias that can arise from differential expertise bias and have the potential to improve the validity and feasibility of RCTs. Finally, there has been an increased focus on the reporting of patient reported outcomes (PROs) in orthopaedic RCTs. Alongside this movement has been the development of minimal important differences (MIDs) to define the changes that are relevant and meaningful to patients. Both PROs and MIDs should be taken into consideration when calculating the sample size and study power in clinical trials.

CONCLUSIONS

Although marked improvements have been made in the design and implementation of trials, there is still considerable room for improvement. Adequately blinded and powered studies evaluating clinically important outcomes and differences should be key considerations in trial design moving forward.

Ultrasound-guided transversus abdominis plane block vs trigger point injections for chronic abdominal wall pain: a randomized clinical trial

ABSTRACT

The primary aim of this randomized clinical trial is to investigate the effects of ultrasound-guided transversus abdominis plane (TAP) vs ultrasound-guided trigger point injections (TPIs) on numerical rating scale pain scores at month 3 follow-up in patients with a chronic abdominal wall pain. The primary outcome measure was the difference in mean numeric rating scale pain scores between the TAP and TPI groups at month 3 in an intent-to-treat (ITT) analysis. A total of 60 patients were randomized 1:1 to receive an ultrasound-guided TAP block (n = 30) or an ultrasound-guided TPI (n = 30). No significant group differences in baseline demographic or clinical characteristics were observed. The mean baseline pain score for the TAP and TPI groups was 5.5 and 4.7, respectively. In the ITT analysis at month 3, the between-group difference in pain scores was 1.7 (95% confidence interval, 0.3-3.0) favoring the TPI group. In a secondary per-protocol analysis, the between-group difference in pain scores was 1.8 (95% confidence interval, 0.4-3.2) favoring the TPI group. For the ITT and per-protocol analyses, the group differences in pain scores were consistent with a medium effect size. The main finding of this randomized clinical trial is that adults with chronic abdominal wall pain who received a TPI reported significantly lower pain scores at month 3 follow-up compared with patients who received a TAP block.

Effect of Autologous Expanded Bone Marrow Mesenchymal Stem Cells or Leukocyte-Poor Platelet-Rich Plasma in Chronic Patellar Tendinopathy (With Gap >3 mm): Preliminary Outcomes After 6 Months of a Double-Blind, Randomized, Prospective Study

BACKGROUND

Patellar tendinopathy is common. The success of traditional management, including isometric or eccentric exercises combined with shockwave therapy and even surgery, is limited. Therefore, it is important to determine whether biological treatments such as ultrasound-guided intratendinous and peritendinous injections of autologous expanded bone marrow mesenchymal stem cells (BM-MSCs) or leukocyte-poor platelet-rich plasma (Lp-PRP) improve clinical outcomes in athletic patients with patellar tendinopathy.

STUDY DESIGN

Randomized controlled trial; Level of evidence, 2.

METHODS

A prospective, double-blinded, randomized, 2-arm parallel group, active controlled, phase 1/2 single-center clinical study was performed in patients who had proximal patellar tendinopathy with a lesion >3 mm. A total of 20 participants (age 18-48 years) with pain for >4 months (mean, 23.6 months) and unresponsive to nonoperative treatments were randomized into 2 groups. Of these, 10 participants were treated with BM-MSC (20 × 10⁶ cells) and 10 with Lp-PRP. Both groups performed the same postintervention rehabilitation protocol. Outcomes included the Victorian Institute of Sport Assessment for pain (VISA-P), self-reported tendon pain during activity (visual analog scale [VAS]), muscle function by dynamometry, tendon thickness and intratendinous vascularity by ultrasonographic imaging and Doppler signal, ultrasound tissue characterization (UTC) echo type changes, and magnetic resonance imaging (MRI) T2-weighted mapping changes. Participants were followed longitudinally for 6 months.

RESULTS

The average VAS scores improved in both groups at all time points, and there was a significant reduction in pain during sporting activities (P < .05). In both groups, the average mean VISA-P scores at 6 months were significantly increased compared with baseline (66 BM-MSC group and 72.90 Lp-PRP group), with no significant differences in VAS or VISA-P scores between the groups. There were statistically significant greater improvements in tendon structure on 2-dimensional ultrasound and UTC in the BM-MSC group compared with the Lp-PRP group at 6 months. Similarly, the BM-MSC group demonstrated significant evidence of restoration of tendon structure on MRI compared with the Lp-PRP group at 6 months. Only the participants in the BM-MSC group showed evidence of normalization of tendon structure, with statistically significant differences between the groups on T2-weighted, fat-saturated sagittal and coronal scans and hypersignal in T2-weighted on spin-echo T2-weighted coronal MRI scan. Both treatments were safe, and no significant adverse events were reported in either group.

CONCLUSION

Treatment with BM-MSC or Lp-PRP in combination with rehabilitation in chronic patellar tendinopathy is effective in reducing pain and improving activity levels in active participants. Participants who received BM-MSC treatment demonstrated greater improvement in tendon structure compared with those who received Lp-PRP.

REGISTRATION

2016-001262-28 (EudraCT identifier); NCT03454737 (ClinicalTrials.gov identifier).

The Blight of the Type II Error: When No Difference Does Not Mean No Difference

Much focus in research has been given to minimizing type I errors, where we incorrectly conclude that there is a difference between 2 treatments or populations. In contrast, our standard scientific method and power analysis allows for a much greater rate of type II errors, in which we fail to show a difference when, in fact, one exists (≥20% rate of type II errors vs ≤5% rate of type I errors). Additional factors that can cause type II errors may propel their prevalence to well in excess of 20%. Failure to reject the null hypothesis may be a tolerable outcome in a certain proportion of studies. However, type II errors may become dangerous when the conclusions of a study overreach, incorrectly stating that there is no difference, when, in fact, a difference exists. Type II errors resulting in overreaching conclusions may impede incremental advances in our field, as the advantages of small improvements may go undetected. To avert this danger in studies that fail to meet statistical significance, we as researchers (20% or more, vs 5% for type I errors) be precise in our conclusions stating simply that the null hypothesis could not be rejected.

Estimates of the mean difference in orthopaedic randomized trials: obligatory yet obscure

Background

Randomized controlled trials in orthopaedics are powered to mainly find large effect sizes. A possible discrepancy between the estimated and the real mean difference is a challenge for statistical inference based on p-values. We explored the justifications of the mean difference estimates used in power calculations. The assessment of distribution of observations in the primary outcome and the possibility of ceiling effects were also assessed.

Methods

Systematic review of the randomized controlled trials with power calculations in eight clinical orthopaedic journals published between 2016 and 2019. Trials with one continuous primary outcome and 1:1 allocation were eligible. Rationales and references for the mean difference estimate were recorded from the Methods sections. The possibility of ceiling effect was addressed by the assessment of the weighted mean and standard deviation of the primary outcome and its elaboration in the Discussion section of each RCT where available.

Results

264 trials were included in this study. Of these, 108 (41 %) trials provided some rationale or reference for the mean difference estimate. The most common rationales or references for the estimate of mean difference were minimal clinical important difference (16 %), observational studies on the same subject (8 %) and the ‘clinical relevance’ of the authors (6 %). In a third of the trials, the weighted mean plus 1 standard deviation of the primary outcome reached over the best value in the patient-reported outcome measure scale, indicating the possibility of ceiling effect in the outcome.

Conclusions

The chosen mean difference estimates in power calculations are rarely properly justified in orthopaedic trials. In general, trials with a patient-reported outcome measure as the primary outcome do not assess or report the possibility of the ceiling effect in the primary outcome or elaborate further in the Discussion section.

Assessing variability and uncertainty in orthopedic randomized controlled trials

Background and purpose - Low statistical power remains endemic in clinical medicine including orthopedics and manifests as high uncertainty and wide confidence intervals (CI). We evaluated the reporting and correspondence between power calculation and observed data on key parameters of variability and uncertainty in orthopedic randomized controlled trials (RCTs).Material and methods - RCTs with 1:1 allocation published in 8 major orthopedic journals between 2016 and 2017 with one continuous primary outcome were included in the review. The components of power calculation and observed standard deviation (SD), mean difference (MD), and confidence interval (CI) of MD between groups were assessed for primary outcome.Results - 160 RCTs were included, of which 93 (58%) and 138 (86%) studies reported the estimated SD and MD in the power calculation, respectively. The median ratio of the estimated SD and SDs observed in the data was 1.0 (IQR -0.76 to 1.32) for 69 (43%) studies. Only 31 of 138 studies reported the CI of MD in primary outcome. In 42% of the negative studies, the estimated MD was included in the CI of the observed MD.Interpretation - The key parameters of data variability, both in power analyses and in final study results, were poorly reported. Low power in orthopedics may result from too high an estimated effect size due to an overoptimistic estimate of MD between study groups. In almost half of the studies, overlap of the CI of the observed MD and estimated MD suggested that the reported results of these studies were inconclusive.