Assessment of abstracts submitted to the annual scientific meeting of the Royal Australian and New Zealand College of Radiologists

A simplified, 2-question grading system for evaluating abstracts in orthopedic scientific meetings: a serial randomization study

BACKGROUND AND PURPOSE

Efficient abstract scoring for congress presentation is important. Given the emergence of new study methodologies, a scoring system that accommodates all study designs is warranted. We aimed to assess the equivalence of a simplified, 2-question abstract grading system with a more complex currently used system in assessing abstracts submitted for orthopedic scientific meetings in a serial randomized study.

METHODS

Dutch Orthopedic Association Scientific Committee (DOASC) members were randomized to grade abstracts using either the current grading system, which includes up to 7 scoring categories, or the new grading system, which consists of only 2 questions. Pearson correlation coefficient and mean abstract score with 95% confidence intervals (CI) were calculated.

RESULTS

Analysis included the scoring of 195 abstracts by 12-14 DOASC members. The average score for an abstract using the current system was 60 points (CI 58-62), compared with 63 points (CI 62-64) using the new system. By using the new system, abstracts were scored higher by 3.3 points (CI 1.7-5.0). Pearson correlation was poor with coefficient 0.38 (P < 0.001).

CONCLUSION

The simplified abstract grading system exhibited a poor correlation with the current scoring system, while the new system offers a more inclusive evaluation of varying study designs and is preferred by almost all DOASC members.

Publication rate of abstracts presented at the Australian Orthopaedic Association Annual Scientific Meeting

BACKGROUND

Presentations at scientific conferences are an important method of research dissemination, with abstracts often used to inform clinical practice. Abstract to publication ratio is a commonly used tool for determining meeting quality. The aim of this study was to determine the publication rate for abstracts presented at the Australian Orthopaedic Association Annual Scientific Meeting (AOA ASM) between 2012 and 2015 inclusive and identify reasons for non-publication.

METHODS

MEDLINE, PubMed and Google Scholar were searched to determine whether each abstract presented at AOA ASMs between 2012 and 2015 was associated with a full text publication in a peer-reviewed journal. Where a publication could not be located, the presenter was contacted to confirm the reason for non-publication.

RESULTS

A total of 1130 abstracts were submitted (951 oral and 179 posters), and 573 resulted in full-text peer-reviewed publications (51%). The majority of publications (73%) were published within 2 years of presentation. There was no difference in likelihood of publication for oral presentations compared to posters, nor in the rate of publication across the 4 years of meetings. Common reasons for non-publication were lack of time (32%), publication considered low priority (27%) and journal rejections (22%).

CONCLUSION

The overall publication rate for abstracts presented at the AOA ASM is 51%, which is an increase from the 1998 ASM (31%). This publication rate is higher than many similar Australian meetings and on par with other international orthopaedic and subspecialty meetings. Future research should investigate potential publication bias and methods to minimise barriers to publication.

Full publication of results initially presented in abstracts

BACKGROUND

Abstracts of presentations at scientific meetings are usually available only in conference proceedings. If subsequent full publication of results reported in these abstracts is based on the magnitude or direction of the results, publication bias may result. Publication bias creates problems for those conducting systematic reviews or relying on the published literature for evidence about health and social care.

OBJECTIVES

To systematically review reports of studies that have examined the proportion of meeting abstracts and other summaries that are subsequently published in full, the time between meeting presentation and full publication, and factors associated with full publication.

SEARCH METHODS

We searched MEDLINE, Embase, the Cochrane Library, Science Citation Index, reference lists, and author files. The most recent search was done in February 2016 for this substantial update to our earlier Cochrane Methodology Review (published in 2007).

SELECTION CRITERIA

We included reports of methodology research that examined the proportion of biomedical results initially presented as abstracts or in summary form that were subsequently published. Searches for full publications had to be at least two years after meeting presentation.

DATA COLLECTION AND ANALYSIS

Two review authors extracted data and assessed risk of bias. We calculated the proportion of abstracts published in full using a random-effects model. Dichotomous variables were analyzed using risk ratio (RR), with multivariable models taking into account various characteristics of the reports. We assessed time to publication using Kaplan-Meier survival analyses.

MAIN RESULTS

Combining data from 425 reports (307,028 abstracts) resulted in an overall full publication proportion of 37.3% (95% confidence interval (CI), 35.3% to 39.3%) with varying lengths of follow-up. This is significantly lower than that found in our 2007 review (44.5%. 95% CI, 43.9% to 45.1%). Using a survival analyses to estimate the proportion of abstracts that would be published in full by 10 years produced proportions of 46.4% for all studies; 68.7% for randomized and controlled trials and 44.9% for other studies. Three hundred and fifty-three reports were at high risk of bias on one or more items, but only 32 reports were considered at high risk of bias overall.Forty-five reports (15,783 abstracts) with 'positive' results (defined as any 'significant' result) showed an association with full publication (RR = 1.31; 95% CI 1.23 to 1.40), as did 'positive' results defined as a result favoring the experimental treatment (RR =1.17; 95% CI 1.07 to 1.28) in 34 reports (8794 abstracts). Results emanating from randomized or controlled trials showed the same pattern for both definitions (RR = 1.21; 95% CI 1.10 to 1.32 (15 reports and 2616 abstracts) and RR = 1.17; 95% CI, 1.04 to 1.32 (13 reports and 2307 abstracts), respectively.Other factors associated with full publication include oral presentation (RR = 1.46; 95% CI 1.40 to 1.52; studied in 143 reports with 115,910 abstracts); acceptance for meeting presentation (RR = 1.65; 95% CI 1.48 to 1.85; 22 reports with 22,319 abstracts); randomized trial design (RR = 1.51; 95% CI 1.36 to 1.67; 47 reports with 28,928 abstracts); and basic research (RR = 0.78; 95% CI 0.74 to 0.82; 92 reports with 97,372 abstracts). Abstracts originating at an academic setting were associated with full publication (RR = 1.60; 95% CI 1.34 to 1.92; 34 reports with 16,913 abstracts), as were those considered to be of higher quality (RR = 1.46; 95% CI 1.23 to 1.73; 12 reports with 3364 abstracts), or having high impact (RR = 1.60; 95% CI 1.41 to 1.82; 11 reports with 6982 abstracts). Sensitivity analyses excluding reports that were abstracts themselves or classified as having a high risk of bias did not change these findings in any important way.In considering the reports of the methodology research that we included in this review, we found that reports published in English or from a native English-speaking country found significantly higher proportions of studies published in full, but that there was no association with year of report publication. The findings correspond to a proportion of abstracts published in full of 31.9% for all reports, 40.5% for reports in English, 42.9% for reports from native English-speaking countries, and 52.2% for both these covariates combined.

AUTHORS' CONCLUSIONS

More than half of results from abstracts, and almost a third of randomized trial results initially presented as abstracts fail to be published in full and this problem does not appear to be decreasing over time. Publication bias is present in that 'positive' results were more frequently published than 'not positive' results. Reports of methodology research written in English showed that a higher proportion of abstracts had been published in full, as did those from native English-speaking countries, suggesting that studies from non-native English-speaking countries may be underrepresented in the scientific literature. After the considerable work involved in adding in the more than 300 additional studies found by the February 2016 searches, we chose not to update the search again because additional searches are unlikely to change these overall conclusions in any important way.

Publication rates of abstracts presented at the Royal Australian and New Zealand College of Radiologists annual scientific meetings: Any change since 2004?

INTRODUCTION

The aim of this study was to determine the publication rate of abstracts presented at annual Royal Australian and New Zealand College of Radiologists (RANZCR) conferences.

METHODS

The College's Annual Scientific Meetings (ASMs) from 2010 to 2013 were examined, with the goal of comparing these results to the findings of an earlier identical study that examined RANZCR ASMs from 1996 to 1999.

RESULTS

Of the 1152 research abstracts presented, 468 (41%) had been published as full articles. The overall abstract to publication ratio (APR) for radiology was 34% and for radiation oncology was 57%. For oral presentations, these were 44% for radiology and 55% for radiation oncology. Papers were published in a wide variety of journals but 23% of articles appeared in the College's journal, the Journal of Medical Imaging and Radiation Oncology. The mean time between presentation and publication was 16.5 months (median 17 months).

CONCLUSION

Publication rates are comparable with international reports. The APR has increased since the previous study for both disciplines, but more so for Radiation Oncology.

Analysis of reporting quality for oral presentations of observational studies at 19th National Surgical Congress: Proposal for a national evaluation system

OBJECTIVE

To compare the quality of oral presentations presented at the 19^th National Surgical Congress with a national evaluation system with respect to the applicability of systems, and consistency between systems and reviewers.

MATERIAL AND METHODS

Fifty randomly selected observational studies, which were blinded for author and institute information, were evaluated by using the Strengthening the Reporting of Observational Studies (STROBE), Timmer Score, and National Evaluation System by two reviewers. Abstract scores, evaluation periods, and compatibility between reviewers were compared for each evaluation system. Abstract scores by three different evaluation systems were regarded as the main outcome. Wilcoxon matched-pairs signed rank and Friedman tests for comparison of scores and times, kappa analysis for compatibility between reviewers, and Spearman correlation for analysis of reviewers based on pairs of evaluation systems were used.

RESULTS

There was no significant difference between abstract scores for each system (p>0.05). A significant difference for evaluation period of reviewers was detected for each system (p<0.05). Compatibility between reviewers was the highest for the Timmer Score (medium, κ=0.523), and the compatibility for STROBE and National Evaluation System was regarded as acceptable (κ=0.394 and κ=0.354, respectively). Assessment of reviewers for pairs of evaluation systems revealed that scores increased in the same direction with each other significantly (p<0.05).

CONCLUSION

The National Evaluation System is an appropriate method for evaluation of conference abstracts due to the consistent results between the referees similarly with the current international evaluation systems and ease of applicability with regard to evaluation period.

Discrepancies Between Plastic Surgery Meeting Abstracts and Subsequent Full-Length Manuscript Publications

BACKGROUND

The purpose of this bibliometric study was to assess the discrepancies between plastic surgery meeting abstracts and subsequent full-length manuscript publications.

METHODS

Abstracts presented at the Brazilian Congress of Plastic Surgery from 2010 to 2011 were compared with matching manuscript publications. Discrepancies between the abstract and the subsequent manuscript were categorized as major (changes in the purpose, methods, study design, sample size, statistical analysis, results, and conclusions) and minor (changes in the title and authorship) variations.

RESULTS

The overall discrepancy rate was 96 %, with at least one major (76 %) and/or minor (96 %) variation. There were inconsistencies between the study title (56 %), authorship (92 %), purpose (6 %), methods (20 %), study design (36 %), sample size (51.2 %), statistical analysis (14 %), results (20 %), and conclusions (8 %) of manuscripts compared with their corresponding meeting abstracts.

CONCLUSION

As changes occur before manuscript publication of plastic surgery meeting abstracts, caution should be exercised in referencing abstracts or altering surgical practices based on abstracts' content.

LEVEL OF EVIDENCE V

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

The fate of research abstracts submitted to a national surgical conference: a cross-sectional study to assess scientific impact

BACKGROUND

Conference abstracts often lack rigorous peer review, but potentially influence clinical thinking and practice. To evaluate the quality of abstracts submitted to a large surgical conference, presentation and publication rates were investigated to assess scientific impact.

METHODS

A Cross-sectional study of abstracts submitted to Dutch Surgical Society meetings from 2007 to 2012 was conducted. Presentation rates, publication rates in MEDLINE-indexed journals using PubMed Central database, and actuarial times to subsequent publication were investigated.

RESULTS

Of 2,174 submitted abstracts, 1,305 (60%) abstracts were accepted for presentation. Actuarial 1, 3, and 5-year publication rates were 22.4%, 62.2%, and 68.6% for presented abstracts, compared with 20.9%, 50.3%, and 57.7% for rejected abstracts, respectively (log-rank x(2) 23.728, df1, P < .001). Publications resulting from abstracts presented at the conference had a significantly higher mean (±standard error) impact factor (4.4 ± .2 vs 3.4 ± .1, P < .001), compared with publications from previously rejected abstracts.

CONCLUSIONS

We advocate critical appraisal of the use of findings of scientific abstracts and conference presentations. The 5-year abstract-to-publication ratio is proposed as a novel quality indicator to allow objective comparison between scientific meetings.

Evaluation of reporting quality of the 2010 and 2012 National Surgical Congress oral presentations by CONSORT, STROBE and Timmer criteria

OBJECTIVE

This study aimed to evaluate the abstracts of oral presentations that were accepted to the National Surgical Congress by CONSORT, STROBE and Timmer criteria and to recommend development of a national abstract assessment system.

MATERIAL AND METHODS

Presentation scores were calculated for oral presentations that have been accepted to the 2010 and 2012 National Surgical Congresses and have been included in the digital congress abstract booklets by two independent reviewers who were blinded to information regarding both the author and the institution. The CONSORT and Timmer criteria were used for randomized controlled trials, and for observational studies the STROBE and Timmer criteria were used. The presentation score that was obtained by three different evaluation systems was accepted as the main variable. The score changes according to the two congresses, the influence of the reviewers on the presentation scores, and compatibility between the two reviewers were evaluated. Comparisons regarding study types and total presentation number were made by using the chi-square test, the compatibility between the total score of the presentations were made by the Mann-Whitney U test and the compatibility between the reviewers were evaluated by the Wilcoxon signed ranks test.

RESULTS

There was no difference between the two Congresses in terms of study type distribution and total number of accepted presentations (p=0.844). The total scores of randomized controlled trials and observational studies from the 2010 and 2012 National Surgical Congresses that were evaluated by two independent reviewers with different assessment tools did not show any significant difference (p>0.05). A significant difference was observed between the reviewers in their evaluation by CONSORT, STROBE and Timmer criteria (p<0.05).

CONCLUSION

Implementation of standard criteria for the evaluation of abstracts that are sent to congresses is important in terms of presentation reporting quality. The existing criteria should be revised according to national factors, in order to reduce the significant differences between reviewers. It is believed that discussions on a new evaluation system will be beneficial in terms of the development of a national assessment system.

Selecting the best clinical vignettes for academic meetings: should the scoring tool criteria be modified?

BACKGROUND

The performance of scoring tools to select clinical vignettes for presentation at academic meetings has never been assessed.

OBJECTIVE

To measure the psychometric properties of two scoring tools used to select clinical vignettes and to determine which elements are most helpful.

DESIGN

Prospective observational study.

PARTICIPANTS

Participants submitting clinical vignette abstracts, Society of General Internal Medicine annual meetings (2006-2007).

MAIN MEASURES

The 2006 scoring tool had three criteria (clarity, significance, and relevance) with brief general descriptors. The 2007 modified tool had five criteria (clarity, significance, relevance, teaching value, and overall assessment) with more detailed descriptors.

KEY RESULTS

A total of 938 clinical vignette abstracts were submitted (484 in 2006; 454 in 2007); 59.5% (n=288) were accepted for presentation. Cronbach's alpha was 0.81 for the 2006 three-item tool and 0.95 for the 2007 modified five-item tool. Simplifying the five-item 2007 tool to three items (relevance, teaching value, overall assessment) yielded a Cronbach's alpha of 0.95. The agreement between the number of clinical vignettes accepted for presentation (2007) using the average score of the five items with the number that would have been accepted using the simplified three items (relevance, teaching value, overall assessment) was almost perfect, with kappa 0.89 (95% confidence interval, 0.85 to 0.93).

CONCLUSIONS

Both scoring tools performed well, but a simplified tool with three items (relevance, teaching value, and overall assessment) and detailed descriptors was optimal; the simplified tool could improve the reviewer efficiency and quality of clinical vignettes presented at national meetings.