ASERNIP-S Review Process

Procedure assessment process
Review group process
Review group membership and roles
Classification system

Procedure assessment process

The College established ASERNIP-S in January 1998. The role of ASERNIP-S is to collect and assess evidence-based information in regard to the safety and efficacy of selected new surgical procedures. Recommendations on the safety and efficacy of the new surgical techniques and technologies are then produced.

The flow chart below illustrates the process adopted by ASERNIP-S to assess new surgical procedures. The process commences with nomination of procedures from a variety of sources including:

• the divisions/Sections of the College
• specialist societies
• hospitals
• consumer complaints commissions
• consumers health forum
• and from individuals.

The ASERNIP-S Advisory Committee endorses the nominations and assessment of the procedure commences when time is available. The output of the process is a draft review, recommendations and a safety and efficacy classification, which is submitted to the ASERNIP-S Advisory Committee for ratification. Once the review is endorsed by the College Council, the final document is then disseminated to relevant groups of the College, hospital credentials committees, consumer groups and any other interested parties. Reviews are also made available on the website.

Review group process

The process for systematic review of each procedure is as follows:

1. Review group establishment
   • ASERNIP-S establishes the review group to assess the procedure.
2. Review protocol manual development- The purpose of the protocol is to establish the criteria for the systematic review.
   • The Protocol Surgeon and ASERNIP-S Researcher consult regarding the search terms, study design, comparators and the inclusion/exclusion criteria.
   • The ASERNIP-S Researcher conducts a comprehensive search of the literature, and references are input into a database.
   • The finalised draft protocol is forwarded to all members of the review group and a protocol teleconference is arranged.
   • Following the teleconference any necessary changes to the protocol are made and the protocol for the systematic review is complete.
3. Systematic review preparation
   • Based on the inclusion/exclusion criteria stated in the protocol, 2 ASERNIP-S Researchers independently assess abstracts of articles for inclusion in the review. Any discrepancies are discussed and an agreement reached.
   • The researcher obtains copies of all articles to be included in the review.
   • The ASERNIP-S Researcher extracts data from all articles into tables on the safety and efficacy of the procedure and assesses the methodological validity of the studies using checklists for both randomized controlled trials and non-randomised studies.
   • The ASERNIP-S Researcher produces a draft systematic review on the safety and efficacy of the procedure. The Advisory surgeon is consulted during this period if necessary. At this stage, no recommendation or classification of safety and efficacy is made.
   • All members of the review group are sent the draft systematic review. They critique the draft systematic review and provide comments, which are then circulated to all other members of the review group. This forms a basis for discussions at the teleconference.
   • The review group is also asked to provide a preferred safety and efficacy classification for the procedure.
4. Review group teleconference
   • The review group meets to discuss the review and classification via a teleconference.
   • The review group reaches consensus on the safety and efficacy classification of the procedure and provide clinical recommendation(s) for its use in practice.
5. Review ratification by ASERNIP-S Management Committee
   • A representative of the review group presents the ASERNIP-S systematic review, which includes the safety and efficacy classification and clinical recommendation(s), to the management committee.
   • The management committee ratifies the ASERNIP-S systematic review.
6. Consideration of the ASERNIP-S systematic review by RACS Council
   
   • The Council of the College endorses the ASERNIP-S systematic review.
7. Dissemination of the systematic review
   • The ASERNIP-S systematic review is forwarded to the relevant Section/division of the College and/or specialist society, and released into the public domain.
   • Executive summaries of systematic reviews are forwarded to credentials committees at all surgical hospitals in Australia.
8. Mediation
   • Should an individual or group of persons wish to appeal against the contents or recommendation of the ASERNIP-S review, a petition is first directed to the respective review group.
   • The review group meets again by teleconference to discuss the concerns raised.
   • This appeal is reported to the management committee as part of a mediation process.
   • If the review group agrees it cannot resolve the appeal, then the appeal proceeds through the College external appeals process.

ASERNIP-S review re-appraisal process

Approximately 1 to 2 years after the initial review scoping searches are conducted to determine the worth of commencing a full re-appraisal and update of the original systematic review.

If re-appraisal is warranted, the review group is contacted to inform them of the re-appraisal. Complete searches are conducted and the abstracts of retrieved articles assessed independently by both the researcher and the Protocol Surgeon regarding their inclusion for the review re-appraisal. The finalised database of new articles for review is sent to all members of the review group along with the protocol. If necessary a pre-assessment teleconference is held; however, in most cases, this will not be required. The researcher conducts the re-appraisal. The updated systematic review, a copy of the new articles and a draft review appraisal form are forwarded to the review group. The review group decides if the new evidence requires any changes to be made to the classification and recommendations of the systematic review, and these are updated if necessary. The ASERNIP-S Management Committee considers the re-appraisal for ratification and then the College Council is forwarded the re-appraisal documentation. Relevant groups of the College and any other interested stakeholders are forwarded a copy of the review re-appraisal and the updated version is posted on the ASERNIP-S website.

Review group membership and roles

In addition to the review group members outlined below other people who have expertise in the area under review may be invited.

ASERNIP-S surgical director

The Surgical Director is the Chairperson of the review group.

ASERNIP-S researcher

The ASERNIP-S Researcher coordinates the systematic review process and organises the review group. The researcher drafts the protocol for the systematic review, in conjunction with the Protocol Surgeon, and searches appropriate databases for peer-reviewed literature on the procedure. Relevant outcomes and other data are extracted from the articles and tabulated. A meta-analysis is conducted, where appropriate ie. Where data from individual studies is able to be combined and analysed statistically. The information is summarised, and along with the tabulated information is provided to the review group as a draft systematic review.

Protocol surgeon

The Protocol Surgeon is familiar with the literature relating to the procedure. The Protocol Surgeon assists the researcher to draft the protocol for the systematic review.

Advisory surgeon(s)

The Advisory Surgeon is an expert from the relevant specialty group but is not intimately involved in the procedure. The Advisory Surgeon is available for guidance, if required, by the researcher during the preparation of the draft systematic review.

Surgeon from another specialty section or division

The surgeon from another specialty provides balance to the review group. He or she receives a copy of the draft systematic review for comment.

Other invited members

To maintain a balance in terms of perceived bias, and to include expertise from other related areas, other persons will be invited to join the review group to provide their input.

Each member of the group receives a copy of the protocol and the draft systematic review compiled by the ASERNIP-S Researcher. The draft review is critiqued and comments are forwarded to ASERNIP-S for dissemination prior to the review group teleconference. The review group discusses the systematic review and reaches a consensus on the safety and efficacy classification for the procedure and formulates any clinical recommendations.

Classification system

Following the systematic review of a new surgical procedure a statement is prepared covering the evidence rating and the safety and efficacy of the procedure. If further research is required to obtain data on either the safety and/or efficacy of the procedure then recommendations will be given regarding the most appropriate method for doing this.

Evidence rating

• poor
• average
• good

(This gives an idea of the strength, quality, precision and magnitude (where appropriate) of the evidence-base)

The evidence for ASERNIP-S systematic review is classified as Good, Average or Poor, based on the quality and availability of this evidence. High-quality evidence is defined here as having a low risk of bias and no other significant flaws. While high-quality randomised controlled trials are regarded as the best kind of evidence for comparing interventions, it may not be practical or ethical to undertake them for some surgical procedures, or the relevant randomised controlled trials may not yet have been carried out. This means that it may not be possible for the evidence on some procedures to be classified as good.

Safety

The surgical procedure under review will be given one of the following safety classifications:

• at least as safe compared to comparator procedure(s) - A comparator may be the current "gold standard" procedure, an alternative procedure, a non-surgical procedure or no treatment (natural history)
• safety cannot be determined
• less safe compared to comparator* procedure(s).

Efficacy

The surgical procedure under review will be given one of the following efficacy classifications:

• at least as efficacious compared to comparator* procedure(s)
• efficacy cannot be determined
• less efficacious compared to comparator* procedure(s).

Recommendations regarding the need for further research

In order to strengthen the evidence base regarding the procedure it may be recommended that either:

• an audit be undertaken, or
• a controlled clinical trial, ideally with random allocation to an intervention and control group, be conducted.

The College recognises that it may not always be possible to undertake a controlled clinical trial. Under such circumstances, it is recommended that, at the very least, data be contributed to an audit for further assessment, in collaboration with ASERNIP-S, until such time as a controlled clinical trial is undertaken.