Since 1993, the Institute for Clinical Systems Improvement (ICSI) has developed more than 60 evidence-based health care guidelines that support best practices for the prevention, diagnosis, treatment, or management of a given symptom, disease, or condition for patients. The rigor of ICSI's development and review process has helped make ICSI guidelines a resource accessed in more than 130 countries, according to Google Web site usage data. In this commentary, we highlight the process, as well as address the challenges, of providing relevant guidelines for the future in light of emerging care delivery models and technological advances.
Potential Conflict of Interest
To maintain the integrity of our documents, ICSI has long implemented a policy of transparency, requiring disclosure of potential conflicts and competing interests by all individuals who participate in the development, revision, and approval of ICSI scientific documents. In 2009, ICSI established a Conflict of Interest Review Committee to review these disclosures and make recommendations to the ICSI Board of Directors regarding further steps that could be taken to mitigate potential conflicts. The ICSI board ultimately determines what steps to take to ensure the integrity and reliability of the ICSI guidelines.
The ICSI guideline development process starts with the Committee for Evidence-based Practice selecting topics based on the following criteria:
- Frequency of the topic in the care provided by ICSI members (50+ medical group and hospital members representing about 9,000 physicians)
- Probability of achieving change (closing the gap), including the likelihood of reaching agreement on the recommendations and of implementing the provisions of the document
- Anticipated improvements, including outcomes and waste reduction
- Health care environment, regulatory requirements, and other initiatives
To give ICSI a broad and timely perspective on which new guidelines to develop, topic ideas are solicited annually from all ICSI member groups and guideline work groups.
Document Development and Revision Process
Each guideline work group consists of 10 to 15 members, including physicians, nurses, pharmacists, other health care professionals relevant to the topic, and two ICSI staff members who facilitate the process, collate content components, and draft implementation recommendations and measures for work group consideration.
The development process is based on a number of long-proven approaches (1,2). ICSI staff first conducts a literature search to identify pertinent clinical trials, meta-analyses, systematic reviews, regulatory statements, and other professional guidelines. The literature is reviewed and graded based on the ICSI Evidence Grading System (see Appendix A below).
A key building block to guideline development is defining its clinical scope. Once the scope is defined and relevant literature is assembled, ICSI facilitators identify gaps between current and optimal practices (3). They then guide the work group through development of the clinical flow and algorithm, drafting of annotations and identification of the literature citations. ICSI staff reviews existing regulatory and standard measures and drafts outcome and process measures for work group consideration. The work group considers the importance of changing systems and physician behavior so that outcomes such as health status, patient and provider satisfaction, and cost/utilization (4) are maximized.
A distinctive component of the ICSI guidelines is their implementation recommendations, which are based on ICSI member experiences as well as on best practices identified in the literature. As members of ICSI, medical groups are committed to review ICSI guidelines up for revision. The medical groups provide feedback on new literature, identify areas needing clarification, offer recommended changes, outline successful implementation strategies, and list barriers to implementation. A summary of the feedback from all medical groups is provided to the guideline work group for use in the revision of the guideline.
A nominal group technique (5) is used to build work group consensus around all the guideline components. This allows for the free exchange of opinions and the generation of ideas within a structured and nonhierarchical discussion forum.
Critical Review Process
This step is unique, as it provides an opportunity for clinicians in integrated systems, hospitals, rural/metro independent practices and specialty groups to review the clinical recommendations, evidence, and clinical flow of the guideline prior to publishing. All ICSI member organizations are expected to critically review newly developed guidelines prior to final approval. This is an opportunity to review the science behind the recommendations and focus on the content of the guideline. Each organization speaks with one voice, submitting a consensus report of responses to all guideline components. This critical review process also allows each organization to consider system changes needed to implement the guideline.
As in the revision process, responses are submitted to the guideline development work group for consideration. The work group seeks additional literature as needed, revises the draft document, and responds to all the feedback, outlining why they did or did not integrate the recommended revisions into the guideline. This process challenges the work group's interpretation of the evidence and requires a clear presentation of the content.
Implementation Recommendations and Measures
Each ICSI guideline includes a set of implementation recommendations and measures. These are derived from the implementation strategies and barriers submitted by the ICSI membership as well as from the lessons learned through ICSI collaboratives (multi-stakeholder clinical improvement initiatives involving several medical groups). Each guideline includes implementation strategies related to key clinical recommendations. In addition, ICSI offers guideline-derived measures. The presence of measure consultants on the guideline development panels ensures that ICSI's measures flow from each guideline's clinical recommendations and implementation strategies. All regulatory and publicly reported measures are included but, more important, measures are recommended to assist medical groups with implementation; thus, both process and outcomes measures are offered. We provide complete data specifications for at least one measure, making clear what is "in" the measure and what is excluded (see Appendix B below). This provides medical groups with a common measure to use right away and also provides a reference for developing other measures. ICSI updates the measures along with the guidelines if new evidence emerges.
The ICSI Committee for Evidence-based Practice ultimately approves each document. Approval criteria include:
- Member comments must be addressed reasonably.
- ICSI member organizations express consensus on the guideline's content.
- To the best knowledge of the reviewer, the scientific recommendations within the guideline are current.
- When evidence for a particular recommendation in a guideline has not been well established, the work group provides consensus statements that were developed based on community standard of practice and work group expert opinion (6).
Document Revision Cycle
Scientific documents are revised every 12 to 24 months as indicated by changes in clinical practice and literature. Each ICSI staff person monitors major, peer-reviewed journals every month for the guidelines for which he or she is responsible. Work group members are also asked to provide any pertinent literature through check-ins with the ICSI staff mid-cycle and annually to determine if there have been changes in the evidence significant enough to warrant document revision earlier than scheduled. This process complements the exhaustive literature search that is done on the subject prior to development of the first version of a guideline.
Member organizations provide feedback on existing guidelines according to ICSI's revision cycle. They provide input on new evidence and practice standards as well as implementation recommendations based on their experience with care delivery and system redesign.
During revisions, ICSI staff identifies pertinent new evidence through a literature search. The work group reviews this literature, responds to comments, and revises the document as appropriate. A second review by the broader ICSI membership is indicated if there are changes or additions to the document that would be unfamiliar or deviate from previous standards of practice. If this additional review is not needed, the document goes directly to the steering committee for approval.
Key Challenges Going Forward
Although the ICSI process has evolved and remained extremely effective over the years, it faces a number of challenges. One is the turnaround time for content drafts, guideline revisions, and the approval process, exacerbated in part by the fact that reviewers are volunteers. This challenge is overcome by constant oversight and artful follow-up by ICSI facilitators. Because turnaround timelines are tight, ICSI is piloting the use of Wikis and other technologies to streamline the development and revision process.
Like all guideline developers, ICSI is faced with creating guidelines that can be used for patients with co-morbid conditions. While speaking at the 2009 ICSI Reinertsen Lecture, David Eddy, MD, noted that health care guideline developers are challenged by creating guidelines that offer individualized treatment plans. Eddy has developed programming in which researchers can conduct trials of virtual patients to figure out the best treatments for real patients. The goal is to develop "personalized treatment plans that would identify the patients who could most benefit from a given medical treatment or intervention that could lessen the expense and time-consuming process of clinical trials" (7).
ICSI will continue to explore how to create guidelines that are effective for "real patients" in a format functional for providers, capitalizing on existing and new technology.
According to ICSI members, this rigorous process for developing evidence-based guidelines that outlines the evidence in a usable, understandable manner supports their implementation and quality improvement initiatives. Feedback from our guideline users identifies the algorithm, implementation recommendations, tools, and measures of success as keys to the adoption of the ICSI guidelines.
Individual research reports are assigned a letter indicating the class of report based on design type: A, B, C, D, M, R, X.
|Primary Reports of New Data Collections
||Randomized, controlled trial
||Non-randomized trial with concurrent or historical controls
- Case-control study
- Study of sensitivity and specificity of a diagnostic test
- Population-based descriptive study
|Reports That Synthesize or Reflect upon Collections of Primary Reports
- Systematic review
- Decision analysis
- Cost-effectiveness analysis
- Consensus report
- Narrative review
Appendix B - Example of Data Specification for Measures*
Possible Success Measure #3b
Percentage of adult heart failure patients with documentation that LVS function was evaluated or will be evaluated (primary care and outpatient cardiology).
Adult heart failure patients who had a clinic visit during the month in question.
Data of Interest
Numerator: Number of adult heart failure patients with documentation that left ventricular systolic (LVS) function was evaluated or will be evaluated.
- Patients who are less than 18 years of age
- Patients with reason(s) documented by a physician, nurse practitioner or physician assistant for no LVS function evaluation
Denominator: Number of adult heart failure patients with a clinic visit during the month in question
ICD-9 codes: 402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, 404.93, 428.0, 428.1, 428.20, 428.21, 428.22, 428.23, 428.30, 428.31, 428.32, 428.33, 428.40, 428.41, 428.42, 428.43, 428.9.
The measurement period is monthly. Monthly data will be submitted quarterly. For the patients who are in the monthly sample, the data cover the entire span of the patients' primary care and outpatient cardiology histories.
Definition of Terms
The most useful evaluation of left ventricular function is the comprehensive two-dimensional echocardiogram coupled with Doppler flow studies. Radionuclide ventriculography can also be performed to evaluate left ventricular ejection fraction and volume. Description of left ventricular systolic function may be quantitative (i.e., ejection fraction) or qualitative (e.g., "moderately depressed" or visually estimated ejection fraction).
The clinic visit is defined as an office visit with a physician, nurse practitioner or physician assistant. Education office visits may include a visit with a nurse. Exclude visits for the purpose of testing or device checks only.
Suggested Sample Size
The minimum sample size is 20 patients per month.
*Source: Institute for Clinical Systems Improvement (ICSI). Heart failure in adults. Bloomington (MN): Institute for Clinical Systems Improvement (ICSI); 2007 Aug. 119 p. [217 references]
Cally Vinz, RN
ICSI, Bloomington, MN
Joann Foreman, RN
ICSI, Bloomington, MN
Kathy Cummings, RN
ICSI, Bloomington, MN
The views and opinions expressed are those of the authors and do not necessarily state or reflect those of the National Quality Measures Clearinghouse™ (NQMC), the Agency for Healthcare Research and Quality (AHRQ), or its contractor, ECRI Institute.
Potential Conflicts of Interest
Ms. Vinz, Ms. Foreman, and Ms. Cummings declare that none of them has received any grants or research support from any organization within the last 12 months.
- Goldman RS and Haley W. A decade of clinical practice guideline development. Part 1: rationale. Nephrol News, 2003. Oct; 17(11): 42-45.
- Goldman RS and Haley W. A decade of clinical practice guideline development. Part II: practical application. Nephrol News, 2003. Nov; 17(12): 45, 49-52.
- Handley MR and Stuart ME. An evidence-based approach to evaluating and improving clinical practice: guideline development. HMO Pract. 1994. Mar, 8(1): 10-19.
- Eccles M and Mason J. How to develop cost-conscious guidelines. Health Technol Assess. 2001. Vol. 5(16).
- Allen J, Dyas J, and Jones M. Building consensus in health care: a guide to using the nominal group technique. Br J Community Nurses. 2004; 9(3): 110-114.
- Marshall T, Solomon P and Steber SA. Implementing best practice models by using a consensus-building process. Adm Policy Ment Health. 2001. Nov, 29(2): 105-116.
- Kahn J. The body synthetic. Wired, Dec. 2009.
The National Guideline Clearinghouse™ (NGC), sponsored by the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services, welcomes responses to our posted expert commentaries on the NGC Web site.
Please complete the form below if you wish to have your response made accessible to our users or if you wish for NGC to act on your response. NOTE: Providing such information however, does not ensure that your comment will be published or made available or that NGC will take any course of action. That will be determined by NGC staff in conjunction with our Editorial Board after review of the information you supply. Criteria for publication of responses to commentaries include:
Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-0106) AHRQ, 540 Gaither Road, Room # 5036, Rockville, MD 20850.