Skip to main content
  • Expert Commentary
  • May 17, 2010

An Integrated Approach to Developing Health Care Guidelines and Measures


Since 1993, the Institute for Clinical Systems Improvement (ICSI) has developed more than 60 evidence-based health care guidelines that support best practices for the prevention, diagnosis, treatment, or management of a given symptom, disease, or condition for patients. The rigor of ICSI's development and review process has helped make ICSI guidelines a resource accessed in more than 130 countries, according to Google Web site usage data. In this commentary, we highlight the process, as well as address the challenges, of providing relevant guidelines for the future in light of emerging care delivery models and technological advances.

Potential Conflict of Interest

To maintain the integrity of our documents, ICSI has long implemented a policy of transparency, requiring disclosure of potential conflicts and competing interests by all individuals who participate in the development, revision, and approval of ICSI scientific documents. In 2009, ICSI established a Conflict of Interest Review Committee to review these disclosures and make recommendations to the ICSI Board of Directors regarding further steps that could be taken to mitigate potential conflicts. The ICSI board ultimately determines what steps to take to ensure the integrity and reliability of the ICSI guidelines.

Selection Process

The ICSI guideline development process starts with the Committee for Evidence-based Practice selecting topics based on the following criteria:

  • Frequency of the topic in the care provided by ICSI members (50+ medical group and hospital members representing about 9,000 physicians)
  • Probability of achieving change (closing the gap), including the likelihood of reaching agreement on the recommendations and of implementing the provisions of the document
  • Anticipated improvements, including outcomes and waste reduction
  • Health care environment, regulatory requirements, and other initiatives

To give ICSI a broad and timely perspective on which new guidelines to develop, topic ideas are solicited annually from all ICSI member groups and guideline work groups.

Document Development and Revision Process

Each guideline work group consists of 10 to 15 members, including physicians, nurses, pharmacists, other health care professionals relevant to the topic, and two ICSI staff members who facilitate the process, collate content components, and draft implementation recommendations and measures for work group consideration.

The development process is based on a number of long-proven approaches (1,2). ICSI staff first conducts a literature search to identify pertinent clinical trials, meta-analyses, systematic reviews, regulatory statements, and other professional guidelines. The literature is reviewed and graded based on the ICSI Evidence Grading System (see Appendix A below).

A key building block to guideline development is defining its clinical scope. Once the scope is defined and relevant literature is assembled, ICSI facilitators identify gaps between current and optimal practices (3). They then guide the work group through development of the clinical flow and algorithm, drafting of annotations and identification of the literature citations. ICSI staff reviews existing regulatory and standard measures and drafts outcome and process measures for work group consideration. The work group considers the importance of changing systems and physician behavior so that outcomes such as health status, patient and provider satisfaction, and cost/utilization (4) are maximized.

A distinctive component of the ICSI guidelines is their implementation recommendations, which are based on ICSI member experiences as well as on best practices identified in the literature. As members of ICSI, medical groups are committed to review ICSI guidelines up for revision. The medical groups provide feedback on new literature, identify areas needing clarification, offer recommended changes, outline successful implementation strategies, and list barriers to implementation. A summary of the feedback from all medical groups is provided to the guideline work group for use in the revision of the guideline.

A nominal group technique (5) is used to build work group consensus around all the guideline components. This allows for the free exchange of opinions and the generation of ideas within a structured and nonhierarchical discussion forum.

Critical Review Process

This step is unique, as it provides an opportunity for clinicians in integrated systems, hospitals, rural/metro independent practices and specialty groups to review the clinical recommendations, evidence, and clinical flow of the guideline prior to publishing. All ICSI member organizations are expected to critically review newly developed guidelines prior to final approval. This is an opportunity to review the science behind the recommendations and focus on the content of the guideline. Each organization speaks with one voice, submitting a consensus report of responses to all guideline components. This critical review process also allows each organization to consider system changes needed to implement the guideline.

As in the revision process, responses are submitted to the guideline development work group for consideration. The work group seeks additional literature as needed, revises the draft document, and responds to all the feedback, outlining why they did or did not integrate the recommended revisions into the guideline. This process challenges the work group's interpretation of the evidence and requires a clear presentation of the content.

Implementation Recommendations and Measures

Each ICSI guideline includes a set of implementation recommendations and measures. These are derived from the implementation strategies and barriers submitted by the ICSI membership as well as from the lessons learned through ICSI collaboratives (multi-stakeholder clinical improvement initiatives involving several medical groups). Each guideline includes implementation strategies related to key clinical recommendations. In addition, ICSI offers guideline-derived measures. The presence of measure consultants on the guideline development panels ensures that ICSI's measures flow from each guideline's clinical recommendations and implementation strategies. All regulatory and publicly reported measures are included but, more important, measures are recommended to assist medical groups with implementation; thus, both process and outcomes measures are offered. We provide complete data specifications for at least one measure, making clear what is "in" the measure and what is excluded (see Appendix B below). This provides medical groups with a common measure to use right away and also provides a reference for developing other measures. ICSI updates the measures along with the guidelines if new evidence emerges.

Document Approval

The ICSI Committee for Evidence-based Practice ultimately approves each document. Approval criteria include:

  • Member comments must be addressed reasonably.
  • ICSI member organizations express consensus on the guideline's content.
  • To the best knowledge of the reviewer, the scientific recommendations within the guideline are current.
  • When evidence for a particular recommendation in a guideline has not been well established, the work group provides consensus statements that were developed based on community standard of practice and work group expert opinion (6).

Document Revision Cycle

Scientific documents are revised every 12 to 24 months as indicated by changes in clinical practice and literature. Each ICSI staff person monitors major, peer-reviewed journals every month for the guidelines for which he or she is responsible. Work group members are also asked to provide any pertinent literature through check-ins with the ICSI staff mid-cycle and annually to determine if there have been changes in the evidence significant enough to warrant document revision earlier than scheduled. This process complements the exhaustive literature search that is done on the subject prior to development of the first version of a guideline.

Member organizations provide feedback on existing guidelines according to ICSI's revision cycle. They provide input on new evidence and practice standards as well as implementation recommendations based on their experience with care delivery and system redesign.

During revisions, ICSI staff identifies pertinent new evidence through a literature search. The work group reviews this literature, responds to comments, and revises the document as appropriate. A second review by the broader ICSI membership is indicated if there are changes or additions to the document that would be unfamiliar or deviate from previous standards of practice. If this additional review is not needed, the document goes directly to the steering committee for approval.

Key Challenges Going Forward

Although the ICSI process has evolved and remained extremely effective over the years, it faces a number of challenges. One is the turnaround time for content drafts, guideline revisions, and the approval process, exacerbated in part by the fact that reviewers are volunteers. This challenge is overcome by constant oversight and artful follow-up by ICSI facilitators. Because turnaround timelines are tight, ICSI is piloting the use of Wikis and other technologies to streamline the development and revision process.

Like all guideline developers, ICSI is faced with creating guidelines that can be used for patients with co-morbid conditions. While speaking at the 2009 ICSI Reinertsen Lecture, David Eddy, MD, noted that health care guideline developers are challenged by creating guidelines that offer individualized treatment plans. Eddy has developed programming in which researchers can conduct trials of virtual patients to figure out the best treatments for real patients. The goal is to develop "personalized treatment plans that would identify the patients who could most benefit from a given medical treatment or intervention that could lessen the expense and time-consuming process of clinical trials" (7).

ICSI will continue to explore how to create guidelines that are effective for "real patients" in a format functional for providers, capitalizing on existing and new technology.


According to ICSI members, this rigorous process for developing evidence-based guidelines that outlines the evidence in a usable, understandable manner supports their implementation and quality improvement initiatives. Feedback from our guideline users identifies the algorithm, implementation recommendations, tools, and measures of success as keys to the adoption of the ICSI guidelines.

Appendix A

Evidence Grading

Individual research reports are assigned a letter indicating the class of report based on design type: A, B, C, D, M, R, X.

Class Description
Primary Reports of New Data Collections
A Randomized, controlled trial
B Cohort-study
C Non-randomized trial with concurrent or historical controls

  • Case-control study
  • Study of sensitivity and specificity of a diagnostic test
  • Population-based descriptive study
D Cross-sectional study

  • Case series
  • Case report
Reports That Synthesize or Reflect upon Collections of Primary Reports
M Meta-analysis

  • Systematic review
  • Decision analysis
  • Cost-effectiveness analysis
R Consensus statement

  • Consensus report
  • Narrative review
X Medical opinion

Appendix B - Example of Data Specification for Measures*

Possible Success Measure #3b

Percentage of adult heart failure patients with documentation that LVS function was evaluated or will be evaluated (primary care and outpatient cardiology).

Population Definition

Adult heart failure patients who had a clinic visit during the month in question.

Data of Interest

Numerator: Number of adult heart failure patients with documentation that left ventricular systolic (LVS) function was evaluated or will be evaluated.


  • Patients who are less than 18 years of age
  • Patients with reason(s) documented by a physician, nurse practitioner or physician assistant for no LVS function evaluation

Denominator: Number of adult heart failure patients with a clinic visit during the month in question

ICD-9 codes: 402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, 404.93, 428.0, 428.1, 428.20, 428.21, 428.22, 428.23, 428.30, 428.31, 428.32, 428.33, 428.40, 428.41, 428.42, 428.43, 428.9.

Measurement Period

The measurement period is monthly. Monthly data will be submitted quarterly. For the patients who are in the monthly sample, the data cover the entire span of the patients' primary care and outpatient cardiology histories.

Definition of Terms

The most useful evaluation of left ventricular function is the comprehensive two-dimensional echocardiogram coupled with Doppler flow studies. Radionuclide ventriculography can also be performed to evaluate left ventricular ejection fraction and volume. Description of left ventricular systolic function may be quantitative (i.e., ejection fraction) or qualitative (e.g., "moderately depressed" or visually estimated ejection fraction).

The clinic visit is defined as an office visit with a physician, nurse practitioner or physician assistant. Education office visits may include a visit with a nurse. Exclude visits for the purpose of testing or device checks only.

Suggested Sample Size

The minimum sample size is 20 patients per month.

*Source: Institute for Clinical Systems Improvement (ICSI). Heart failure in adults. Bloomington (MN): Institute for Clinical Systems Improvement (ICSI); 2007 Aug. 119 p. [217 references]


Cally Vinz, RN
ICSI, Bloomington, MN

Joann Foreman, RN
ICSI, Bloomington, MN

Kathy Cummings, RN
ICSI, Bloomington, MN


The views and opinions expressed are those of the authors and do not necessarily state or reflect those of the National Quality Measures Clearinghouse™ (NQMC), the Agency for Healthcare Research and Quality (AHRQ), or its contractor, ECRI Institute.

Potential Conflicts of Interest

Ms. Vinz, Ms. Foreman, and Ms. Cummings declare that none of them has received any grants or research support from any organization within the last 12 months.


  1. Goldman RS and Haley W. A decade of clinical practice guideline development. Part 1: rationale. Nephrol News, 2003. Oct; 17(11): 42-45.

  2. Goldman RS and Haley W. A decade of clinical practice guideline development. Part II: practical application. Nephrol News, 2003. Nov; 17(12): 45, 49-52.

  3. Handley MR and Stuart ME. An evidence-based approach to evaluating and improving clinical practice: guideline development. HMO Pract. 1994. Mar, 8(1): 10-19.

  4. Eccles M and Mason J. How to develop cost-conscious guidelines. Health Technol Assess. 2001. Vol. 5(16).

  5. Allen J, Dyas J, and Jones M. Building consensus in health care: a guide to using the nominal group technique. Br J Community Nurses. 2004; 9(3): 110-114.

  6. Marshall T, Solomon P and Steber SA. Implementing best practice models by using a consensus-building process. Adm Policy Ment Health. 2001. Nov, 29(2): 105-116.

  7. Kahn J. The body synthetic. Wired, Dec. 2009.


Submitted 09/07/2015

I read this article in my final term of school for my BSN. I have been a nurse for over 17 years and never really thought about the entire process of implementing a new Health Care Guideline start to finish. In my Capstone Research Project, we did this entire process, start to finish, just the same way this article describes. I am now more appreciative of the quantity of and quality of work that is put into ensuring these practices are derived from evidence-based, peer-reviewed, current standards of practice. Thank you for the article.

Christia Pries, BSN

Registered Nurse, Nurse Manager

Oregon Department of Corrections

Disclosure/Conflict of Interest: None

Submitted 11/19/2012

I thought the article was very interesting how guidelines for better healthcare are developed and how much time is involved. Very good article.

Joanne Williams,

Staff Nurse

BJC Healthcare Systems

Disclosure/Conflict of Interest: None stated

Make a Comment

All submitted comments will be reviewed by NGC Staff and NGC's Editorial Board who will elect what to publish on this web site.

* indicates required field

Submit a Comment

Do you have any Disclosures and/or Conflicts of Interest to report? *

Optional Information

About Commenting

The National Guideline Clearinghouse ™ (NGC), sponsored by the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services, welcomes responses to our posted expert commentaries on the NGC Web site.

All responses will be reviewed by NGC Staff and NGC's Editorial Board. Please keep your response brief and to the point. Responses must be presented in clear statements. We reserve the right to 1) publish your response, 2) copyedit your response, and 3) invite the author of the Expert Commentary to reply to a posted response. Excessively long or offensive and nonobjective responses will be disregarded.

Please complete the form if you wish to have your response made accessible to our users or if you wish for NGC to act on your response. NOTE: Providing such information however, does not ensure that your comment will be published or made available or that NGC will take any course of action. That will be determined by NGC staff in conjunction with our Editorial Board after review of the information you supply. Criteria for publication of responses to commentaries include:

  • Relevance of response to the subject of the commentary
  • Responses that provide a unique perspective to the commentary
  • Responses that offer an alternative view than the commentary
  • Responses that are clearly articulated
  • Responses that are neither excessively long or offensive and/or nonobjective
Form Approved    OMB No. 0935-0106    Exp. Date 11/30/2017

Public reporting burden for this collection of information is estimated to average 90 seconds per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-0106) AHRQ, 5600 Fishers Lane, Rockville, MD 20857.

Note: Responses accepted for publication will be posted to the NGC Web site along with the submitter's name. Current position or occupation, organizational affiliation, and disclosed potential conflicts of interest are optional and will be posted when provided. Email and other address are for internal use only and will not be published with responses or made publicly available.