Skip to main content
  • Expert Commentary
  • January 02, 2012

Consensus-based Selection Criteria for Accountability Measures: Viewpoint from the National Guideline Clearinghouse/National Quality Measures Clearinghouse Editorial Board

Note: This is the second consensus viewpoint by the Editorial Board of the National Guideline Clearinghouse/National Quality Measures Clearinghouse (NGC/NQMC). (1) Serving in an advisory capacity to NGC/NQMC, the Editorial Board brings diverse expertise to the development and implementation of practice guidelines and quality measures in health care.

Use of quality measures to assess health care delivered by hospitals, health systems, and other providers in the United States (U.S.) has grown in recent years and will likely become increasingly common in the years to come. The purpose of this viewpoint by NGC/NQMC's Editorial Board is to endorse a proposal for consensus-based selection criteria for quality measures used for the purpose of accountability.

In 2010, Chassin et al. called for stakeholders in the quality measurement enterprise to adopt rigorous selection criteria for process measures used to achieve accountability—i.e., for public reporting, payment, or accreditation. (2) (Those authors noted that outcome measures were important but raise further issues to be discussed separately.) Accountability measures, they proposed, should focus explicitly on maximizing net health benefits to patients. To advance this goal, the authors recommended that measures employed for accountability meet the following criteria: (2)

  1. The measure has a strong evidence base showing that the care process improves outcomes.
  2. The measure accurately captures whether the evidence-based care process has, in fact, been provided.
  3. The measure addresses a care process that has few intervening processes that must occur before the improved outcome is realized.
  4. Implementing the measure has little or no chance of inducing unintended adverse consequences.

Data on more than 2,000 quality measures inventoried in the NQMC provide insight into why explicit, well-defined selection criteria are needed to choose accountability measures. Stakeholder organizations developing or selecting measures for broad use generally agree conceptually about the characteristics that are desirable in a quality measure. Although their terminology may differ, most organizations group these characteristics under three domains reflecting a measure's clinical importance, scientific soundness, and feasibility. Organizations seek to optimize these properties in developing and selecting measures.

For each quality measure, NQMC provides its users with information on many of these attributes. Examples of these characteristics (and the domains to which they belong) in NQMC include:

  • The measure's basis in research evidence (Clinical Importance)
  • Study of the measure's reliability and validity (Scientific Soundness)
  • Sources of data needed to construct the measure, reflecting its data-collection burden (Feasibility).

Although organizations agree conceptually about desirable attributes, the measures they develop or select vary widely with regard to these properties. In addition to the inherent difficulty of developing good measures, other reasons contribute to this variation. First, simultaneously optimizing the clinical importance, scientific soundness, and feasibility of a measure may not be possible. Trade-offs frequently need to be made in one domain to achieve the goals of another. For example, measures that address important, evidence-based processes often require clinically richer, but more burdensome, data sources. As a result, measures of greater clinical importance can prove less feasible to implement. Second, selecting measures requires that organizations set priorities among attributes; those representing different stakeholders may not share the same priorities. Third, even when an attribute is widely considered a priority, organizations may have different thresholds for considering it fulfilled. An example is the attribute "evidence-based". Developers provide NQMC with information on research evidence supporting their measures. Across measures submitted, we see wide variation in the types of evidence provided, the aspects of a measure that the evidence supports, and the quality and quantity of studies that make up the relevant body of evidence.

This variability illustrates the need for explicit selection criteria for the evidence base of any measure considered for accountability purposes. The first of the four proposed criteria defines the type of evidence needed: evidence linking the process measured with improved outcomes. Further details will be needed that define the type and number of studies required for this criterion to be met. In an important step, the National Quality Forum has begun to develop criteria for evaluating and testing the scientific acceptability of quality measures. (3)

We endorse the development of criteria along the lines proposed by Chassin et al., even as we recognize that they may present challenges to the development of accountability measures of certain types of clinical processes and, possibly, for some specialty areas in health care.

The clinical processes most commonly evaluated in clinical trials are treatment interventions—predominantly pharmacological. Thus, measures assessing the performance of these interventions are those most likely to have an evidence base qualifying them as an accountability measure. Research studies are less likely to be available for other processes that may be essential for delivering an effective intervention, such as detection of the condition requiring treatment, access to the treatment, and continuity of care. The gap between the importance of these processes and the absence (or indirectness) of research evidence to support them may be a consideration when organizations try to operationalize criteria for accountability measures.

According to Chassin et al., a substantial proportion of current Joint Commission core measures presently used by most U.S. hospitals for accreditation meet all four of the criteria they propose. This welcome assessment, however, does not apply to all major areas of health care. For instance, only one of the seven measures in the Joint Commission's core measure set for inpatient psychiatry is based on research evidence from clinical trials. (4) Establishing criteria for accountability measures will challenge organizations to work even harder than in the past to ensure the development of measures that assess all major health care services.

These concerns should inform rather than delay development and implementation of uniform selection criteria for accountability measures. Through their linkage to reimbursement, accreditation, and other oversight functions, accountability measures are intended to influence patient outcomes. Criteria proposed for selecting accountability measures thus raise the same kinds of questions for quality measures that are routinely considered when evaluating a new clinical intervention: Has the change in a clinical process encouraged by a proposed accountability measure been studied and proven to provide benefit? Does the process measured have fidelity to the process shown to provide that benefit? And can these benefits be realized without causing undue harm?


Potential Conflicts of Interest

All Core Editorial Board members complete Conflict of Interest disclosure forms annually for the Agency of Healthcare Research and Quality (AHRQ). With regard to this viewpoint, Dr. Loeb notes he was an author on the 2010 Chassin et al., article. All other Core Editorial Board members declared no potential conflicts of interest with respect to this viewpoint.

References

  1. NGC/NQMC Editorial Board. Promoting transparent and actionable clinical practice guidelines: viewpoint from the National Quality Measures Clearinghouse/National Guideline Clearinghouse (NQMC/NGC) Editorial Board. Available at: http://www.guideline.gov/expert/expert-commentary.aspx?id=24556.
  2. Chassin MR, Loeb JM, Schmaltz SP, Wachter RM. Accountability measures -- using measurement to promote quality improvement. N Engl J Med 2010;363:683-688.
  3. National Quality Forum. Measure Evaluation Criteria. January 2011. Available at: http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx External Web Site Policy. Accessed: August 7, 2011.
  4. Goren JL, Parks JJ, Ghinassi FA, Milton CG, Oldham JM, Hernandez P, Chan J, Hermann RC. When is antipsychotic polypharmacy supported by research evidence? Implications for quality improvement. The Joint Commission Journal on Quality and Patient Safety 2008; 34(10):571-582.

Comments

Submitted 01/10/2012

All segments of the health care system are not equally engaged in all measures. For any given measure, there is an entity with primary accountability and other entities that play supporting roles to greater or lesser degrees. By this we mean the term "entity" may apply to clinician, hospital, PAC/LTC/SNF, health plan or other specific entity. Only some of published clinical measures indicate the accountable entity for which the measure was designed. In the past, measures have been used by various groups to rate any or all other stakeholder entities without regard to accountability. We believe that all measures published through the NQMC/NGC should have a designated stakeholder as primarily accountable, and other stakehold groups that may have contributing roles. In that way, the primary and secondary lines of accountability may be determined; and stakeholder goups may focus on those aspects of care for which they hold primary or secondary accountability.

Tom James, MD

Medical Director, National Network Operation

Humana, Louisville, Kentucky

Disclosure/Conflict of Interest: Employee of Humana, Inc, and part-time physician with Norton Healthcare

Make a Comment

All submitted comments will be reviewed by NGC Staff and NGC's Editorial Board who will elect what to publish on this web site.

* indicates required field

Submit a Comment

Do you have any Disclosures and/or Conflicts of Interest to report? *

Optional Information

About Commenting

The National Guideline Clearinghouse ™ (NGC), sponsored by the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services, welcomes responses to our posted expert commentaries on the NGC Web site.

All responses will be reviewed by NGC Staff and NGC's Editorial Board. Please keep your response brief and to the point. Responses must be presented in clear statements. We reserve the right to 1) publish your response, 2) copyedit your response, and 3) invite the author of the Expert Commentary to reply to a posted response. Excessively long or offensive and nonobjective responses will be disregarded.

Please complete the form if you wish to have your response made accessible to our users or if you wish for NGC to act on your response. NOTE: Providing such information however, does not ensure that your comment will be published or made available or that NGC will take any course of action. That will be determined by NGC staff in conjunction with our Editorial Board after review of the information you supply. Criteria for publication of responses to commentaries include:

  • Relevance of response to the subject of the commentary
  • Responses that provide a unique perspective to the commentary
  • Responses that offer an alternative view than the commentary
  • Responses that are clearly articulated
  • Responses that are neither excessively long or offensive and/or nonobjective
Form Approved    OMB No. 0935-0106    Exp. Date 11/30/2017

Public reporting burden for this collection of information is estimated to average 90 seconds per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-0106) AHRQ, 5600 Fishers Lane, Rockville, MD 20857.

Note: Responses accepted for publication will be posted to the NGC Web site along with the submitter's name. Current position or occupation, organizational affiliation, and disclosed potential conflicts of interest are optional and will be posted when provided. Email and other address are for internal use only and will not be published with responses or made publicly available.