The peer review process: the art of judgment

The effect of continuing professional development on public complaints: a case-control study

OBJECTIVES

This study aimed to investigate the relationship between participation in different types of continuing professional development (CPD), and incidences and types of public complaint against physicians.

METHODS

Cases included physicians against whom complaints were made by members of the public to the medical regulatory body in Ontario, Canada, the College of Physicians and Surgeons of Ontario (CPSO), during 2008 and 2009. The control cohort included physicians against whom no complaints were documented during the same period. We focused on complaints related to physician communication, quality of care and professionalism. The CPD data included all Royal College of Physicians and Surgeons of Canada (RCPSC) and College of Family Physicians of Canada (CFPC) CPD programme activities reported by the case and control physicians. Multivariate logistic regression models were used to determine if the independent variable, reported participation in CPD, was associated with the dependent variable, the complaints-related status of the physician in the year following reported CPD activities.

RESULTS

A total of 2792 physicians were included in the study. There was a significant relationship between participation in CPD, type of CPD and type of complaint received. Analysis indicated that physicians who reported overall participation in CPD activities were significantly less likely (odds ratio 0.604; p = 0.028) to receive quality of care-related complaints than those who did not report participating in CPD. Additionally, participation in group-based CPD was less likely (OR 0.681; p = 0.041) to result in quality of care-related complaints.

CONCLUSIONS

The findings demonstrate a positive relationship between participation in the national CPD programmes of the CFPC and RCPSC, and lower numbers of public complaints received by the CPSO. As certification bodies and regulators alike are increasingly mandating CPD, they are encouraged to continually evaluate the effectiveness of their programmes to maximise programme impact on physician performance at the population level.

Peer review of grant applications: criteria used and qualitative study of reviewer practices

BACKGROUND

Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.

METHODS AND FINDINGS

We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.

CONCLUSIONS

Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.

Assessment of family physicians' performance using patient charts: interrater reliability and concordance with chart-stimulated recall interview

Peer-assessment processes with chart review have been used for many years to assess the clinical performance of physicians. The Quebec medical licensing authority has been required by provincial law to assess the practicing Quebec physicians on a nonvoluntary basis. During the period from January 2001 to November 2004, 25 family physicians in active practice were randomly selected from a pool of about 300. For each physician, 25 to 40 patients' medical charts were randomly selected to evaluate the interrater reliability of peer-review assessment of medical charts and to compare ratings based on chart review with a chart-stimulated recall interview to those based on chart review alone. The concordance between chart review alone and that of chart review with chart-stimulated recall interview was 75% for chart keeping, 69% for clinical investigation, 81% for diagnostic accuracy, and 74% for treatment plan. Ratings based on chart review alone achieve moderate levels of reliability (Kappa = 0.44 to 0.56). It appears that some important information about quality of care is missed when only chart review is used.

System review: a method for investigating medical errors in healthcare settings

System analysis is a process of evaluating objectives, resources, structure, and design of businesses. System analysis can be used by leaders to collaboratively identify breakthrough opportunities to improve system processes. In healthcare systems, system analysis can be used to review medical errors (system occurrences) that may place patients at risk for injury, disability, and/or death. This study utilizes a case management approach to identify medical errors. Utilizing an interdisciplinary approach, a System Review Team was developed to identify trends in system occurrences, facilitate communication, and enhance the quality of patient care by reducing medical errors.

Developing a cooperative provider-based statewide peer review service: early experience

The purpose of this study was to demonstrate the feasibility of a model of overcoming local barriers to physician peer review through development of a statewide provider-based physician peer review service. For this purpose, the cooperative demonstration project of the Institute for Quality Healthcare, The University of Iowa and The Robert Wood Johnson Foundation, was used. A consortium of 43 Iowa hospitals developed a physician peer review service utilizing a pool of physician reviewers from member hospitals. Thirty-six peer reviews were conducted in 23 different hospitals by 37 different reviewers throughout the state of Iowa in the first 2 years of operation. Reviews of surgical specialists, psychiatrists, and psychiatric services were requested most frequently. The satisfaction of hospitals with the physician peer review service has thus far been gratifying. The long-term financial viability of the physician peer review service has yet to be demonstrated. This cooperative organizational model of a provider-based physician peer review service may be reproducible and valuable to health care providers in other parts of the United States.