Public reports on surgical outcomes and performance (2015)

Rigorous statistical analysis is required to avoid the misrepresentation of the quality performance of institutions, teams and surgeons in public reports. Reports should include a common set of clinically relevant measures that have been properly evaluated for fairness and accuracy so data can be compared across broad populations for meaningful analysis. The methodology for collection of data should be publicly available and include a detailed description of any data that is used to estimate performance, the use of statistical risk adjustment techniques, the selection of performance measures and how surgical performance was categorised. It must be possible to audit the report results.

Reports should be subject to independent analysis to ensure their validity and reliability. Details of volume should be included to ensure that reported data are statistically robust. Where reports contain risk-adjusted data, minimum volume levels should be reported to ensure that the data is excessively representative of the surgical group. Reports on institutions with too few procedures to accurately characterise performance should not be included in public reporting. Such reports should include a statement that an inadequate number of procedures does not allow a meaningful analysis, and in no way reflects upon the performance of the institution.

Reports must be transparent about the observation period and information in quality measures, including an explanation of the observation period, clearly differentiating between long-term follow-up and short-term outcomes. Focus on short-term outcomes may be inappropriate for some procedures and may not help to meet patient expectations. For example, 30-day mortality may be appropriate for some cardiovascular procedures but not for otolaryngology head and neck procedures that would normally carry a low 30-day mortality. Results from these two groups may therefore not be comparable. Similarly the short term follow-up of major joint replacements may not reflect the long-term outcome.

Data within reports should be timely and reflective of current performance of standards of care. Out-dated material may not reflect current performance because of changes in technique or technology. Some measurements can become irrelevant after new evidence and research is published. There should be a statute of limitations within a public report and out-dated reports must be removed from circulation. As a minimum all public reports should clearly indicate the following information in this statute of limitations: Date of creation; date of the most recent update; date of expiry; inclusive dates for the data used in the report; and, what data has been included or excluded from the report.

Reports should use recognised risk adjustment methodology, as determined by the appropriate specialty society, to ensure ongoing accuracy for patients who are at higher risk of complications and poor outcomes. Conditions that increase the complexity of surgery are difficult to convey accurately in performance reports and there can be significant differences in the course of disease or outcome between groups of patients with the same diagnosis. Factors that influence the outcome not only include patient comorbidities but also socio-economic status, health system structures and health resources available in that community. Considering the health system’s balance of resources the most urgent cases and those in most need are usually dealt with in the public hospital sector. These patients may not be comparable with the population group operated on in the private sector and this may be reflected in the outcome data.

Surgeons are the most familiar with the scope of practice, clinical management and published research associated with their craft group. Specialty societies should have the opportunity to provide input regarding institutional or team measures chosen for public reporting. To ensure that measurement outcomes and data accurately reflect surgical care it would be appropriate for the various surgical groups to review the measures to be reported. Each subspecialty can decide and rank the importance of which quality measures best reflect their care.

Report format and content should be standardised in order to provide clear and comprehensible presentation of data. In the absence of standardised formatting patients will not be able to make comparisons for well-informed decisions about their care.

Adequate time frames should be allowed for institutions or teams to review reports prior to publication. Mechanisms should be in place to allow institutions or teams to verify the content of reports, submit feedback and to allow reports to be amended prior to publication. In addition to reviewing data contained within the report, an explanation of the methodology should also be provided. If individual physicians or surgeons are reviewing reports the individual’s data should be made available to that individual so that their data can be validated appropriately.

Pilot tests to determine usefulness and effectiveness of reports should be conducted. Reports should be highlighted to test the effectiveness of the report on the intended audience. Highlighting of reports will allow provision of confidential feedback to providers to build provider trust. It is likely that clinicians will be more likely to release data if they have trust and confidence in the methodology and results.

Reports should be evaluated to ensure that its stated purpose has been achieved and to identify any unintended consequences. Importantly results must not identify individual practitioners as evidence suggests that the publication of individual performance data may lead to risk-averse behaviour. The best clinicians may be those who manage the most complex and difficult patients and this may be reflected in the performance outcome measures.

Public reporting should not be used to establish the standard of care or the duty of care of a healthcare provider. The standard of care should be clearly defined prior to the release of any public reports. While it is expected that in most cases the quality of care will exceed that, the defined standard and the identification and management of outliers will generally increase the general standard of care across all groups. It is recognised that not every individual, team or institution can be the best performer so their performance should not be the benchmark for all other providers.

RACS considers that the purpose of publication of performance outcomes data is to improve the quality of medical care and to improve the public trust and confidence in the delivery of medical care. Importantly public reporting of outcome data should include all sectors of health care delivery and not be confined to the surgical specialties. RACS recognises that the public release of surgical data may have unintended consequences that may impact on the delivery of quality and safe healthcare and affirms the importance of thorough evaluation of reports to minimise this risk. RACS encourages all organisations and individuals involved in the production of reports on surgical outcomes and performance to utilise these principles to guide their analysis and presentation of data.

• Medicare’s Physician Value-Based Payment Modifier – Will the Tectonic Shift Create Waves? Chien AT & Rosenthal MB. N Engl J Med 369;22, November 28, 2013 p 2076-8
• Grading a Physician’s Value – The Misapplication of Performance Management. Berensen RA & Kaye DR. New Engl J Med 369;22 November 28, 2013 p 2079-81
• The Professional Attributes of Surgeons. Clare Marx Ann R Coll Surg Engl (Suppl) 2014;96:220-222
• Individualised surgical outcomes: please look the other way. S R Moonesinghe. Postgraduate medical Journal December 2013 volume 89 number 1058 page 677 – 678.
• Developing Observational Measures of Performance in Surgical Teams. A N Healy, S Undre, C A Vincent. Qual Saf Health Care 2004 13 (suppl) i33-i40
• Publishing individual surgeons’ death rates prompts risk adverse behaviour. S Westaby BMJ 2014;349:g5026 doi: 10.1136/bmj.g5-26 (published 12 August 2014)
• Analyzing :failure to rescue” Is this an opportunity for outcome improvement in cardiac surgery .Reddy HG, Shih t, Englesbe MJ, Shannon FL, Theurer PF, Herbert MA. Et al. Ann Thorac Surge. 2013;95:1976-81
• Berwick DM A Primer on leading the improvement of systems. BMJ 1996;312:619-22 Agency for Healthcare Research and Quality. US Department of Health & Human Services. www.qualitymeasures.ahrq.gov/content.aspz?id=47545
• Complications, failure to rescue, and mortality with major inpatient surgery in medicare patients. Ghaferi AA(1), Birkmeyer JD, Dimick JB. Ann Surg. 2009 Dec;250(6):1029-34.
• Uses and Abuses of Performance Data in Healthcare. Shaw J, Taylor R, Dix K, Dr Foster Intelligence in Healthcare. April 2015