Each year the National Guideline Clearinghouse (NGC) and the National Quality Measures Clearinghouse (NQMC) conduct a user survey on their respective Web Sites to determine how well we are fulfilling our mission of providing an accessible mechanism for obtaining objective, detailed information on clinical practice guidelines and quality measures and to furthering their dissemination, implementation, and use in order to inform health care decisions. In the past the data has been used internally and has guided decisions regarding enhancements to content and the Web Sites. This year, we thought it would be interesting to share the Survey results with our users and examine the differences between the two clearinghouses.
The 2008 NGC/NQMC User Satisfaction Survey was completed in the fall of 2008. This was the 8th such survey of NGC and the 4th of NQMC. Overall, a total of 9,945 NGC users and 722 NQMC users provided feedback to the Agency for Healthcare Research and Quality (AHRQ) and ECRI Institute teams. Surveys like these allow AHRQ and the NGC/NQMC teams to keep abreast of who is using the databases, how they are being used, how satisfied users are, and what can be done to ensure continued usefulness and satisfaction with these critical resources. Below, we describe general findings from the 2008 survey.
Who is Using These Clearinghouses?
For NGC, about two-thirds of users have direct clinical responsibilities: physicians, physician assistants (P.A.), nurses, nurse practitioners, pharmacists, dentists, or other clinical professionals. By contrast, such individuals comprise only one-fourth of NQMC users. Other prevalent occupations in the NQMC user base are Quality Manager (20%), Administrator or Manager of Hospital, Health Plan, or Medical Group (13%), and Health Services/Clinical Researcher or Measure Developer (10%). Figure 1 below provides a more detailed breakdown of both clearinghouse audiences.
Figure description: Figure 1 is a bar graph depicting the distribution of job titles for respondents to the NGC and NQMC surveys. Of those who responded the percentage value for each category of job title is as follows: Nurse/Nurse Practitioner/Physician Assistant was 29.6% for NGC, 11.2% for NQMC; Physician was 25.5% for NGC, 11.8% for NQMC; Student (Medical, Nursing, or Pharmacy) was 16.8% for NGC, 6.8% for NQMC; Pharmacist or other clinical specialist was 9.8% for NGC, 1.9% for NQMC; Other was 9.1% for NGC, 14.7% for NQMC; Medical Librarian or other Information Specialist or Medical Writer/Editor was 3.1% for NGC, 2.2% for NQMC; Quality Manager was 1.8% for NGC, 19.5% for NQMC; Health Services or Clinical Researcher or Measure Developer was 1.5% for NGC, 9.5% for NQMC; Individual Patient or Consumer was 1.2% for NGC, 3.7% for NQMC; Administrator or Manager of Hospital or Health Plan or Medical Group was 1.2% for NGC, 13.4% for NQMC; Public or Private Sector Health Care Purchaser or Employer or Policymaker was 0.3% for NGC, 2.9% for NQMC; and Federal/State Policymaker was 0.2% for NGC, 2.5% for NQMC.
Many current users are new to the clearinghouses. In NGC, 56% of those who filled out the survey did so at their very first visit to the Web site. The percentage for NQMC was similarly high (49%). We also noticed that the NGC population is more international: 35% of NGC users were from outside the U.S., as compared to only 12% of the NQMC population. Both sites have similar proportions of content from non-U.S. developers; however, NGC has a greater diversity of developers representing the non-U.S. guidelines, and this may help to explain why NGC appears to receive more visits from non-U.S. users. In NGC 29% of the content (714 out of 2441 guidelines) was produced by 44 different non-U.S. developers. In NQMC, 24% of the content (415 out of 1754 measures) was produced by 4 different non-U.S. developers.
How Satisfied are Users?
Overall, 45% of NGC users said they were "Very Satisfied" and another 43% said they were "Fairly Satisfied." Another 9% were "Neutral," and the other 2.5% were either "Fairly Dissatisfied" or "Very Dissatisfied." These responses indicate general satisfaction, but there clearly is room for improvement. Although most of the general comments from NGC users were positive, the most common complaint regarded the NGC search function. Specifically, users found the resulting list of guidelines to be cumbersome, and often irrelevant or incorrect. Several users requested a more sophisticated search engine, and/or more explicit instructions on how new users are to successfully conduct a guideline search. For NQMC, users were asked a different question ("Would you recommend the NQMC Web site to anyone else?"), and virtually all (99%) responded "Yes."
In addition, both groups of users were asked to rate the breadth or comprehensiveness of the clearinghouse on a 1-5 scale (5 being the best), and the average ratings were 4.1 and 4.4 for NGC and NQMC, respectively. Also, ease of navigation was generally favorably assessed, with average ratings of 3.7 and 3.8 for NGC and NQMC, respectively. These ratings, however, suggest there is room for improvement, particularly in ease of navigation. For the latter issue, 37% of NGC users gave ratings of either "Neutral," "Difficult," or "Very Difficult."
Which Parts of the Web Sites are Most Useful?
Each clearinghouse Web site provides many options for accessing content, but which are actually used the most? For NGC, the top three responses were Basic Search (cited by 57% of respondents), Disease/Condition Browse (47%), and Treatment/Intervention Browse (33%). Further, the relevance of results of text keyword searches in NGC was rated an average of 3.9 on a scale from 1-5 (5 being the best). The NQMC users listed the same top three access methods, but the order was different: most often cited was Disease/Condition (45%), followed by Basic Search (40%), and Treatment/Intervention Browse (29%). Like the NGC users, text keyword searches were generally quite relevant, with an average of 3.9 on the 1-5 scale.
How do NGC and NQMC Users Compare?
NQMC users were much more likely (52%) than NGC users to learn about the clearinghouse via a link from another health-related web site, and they were also twice as likely to subscribe to the What's New email (36% for NQMC as compared to 17% for NGC). Regarding the What's New email, both groups reported that the most useful part was the notification of recent updates posted to the Web site.
Overall, Web site ratings were slightly more favorable for NQMC. The difference between the two clearinghouses was 9% of the scale range for breadth/comprehensiveness, 11% for relevance of search results, and 2% for ease of navigation. (For example, if a scale ranges from 0 to 4, then 9% of the scale range is (4-0)*0.09 or 0.36 points, similar to going from 3.00 to 3.36.) One possible reason for this difference is the contrasting user bases between NGC (mostly clinicians) and NQMC (mostly managers/administrators), as noted above.
What are the Trends Over Time?
In all eight NGC surveys, users were asked "Overall, how would you rate your satisfaction with NGC?" (we used a 1-5 scale, with 5 being the best). The average response has been consistently high, with averages ranging from 4.29 to 4.42. There is some suggestion of a small decrement over time in these ratings, with the two highest ratings of 4.42 occurring in 2002 and 2004 and the two lowest ratings of 4.29 and 4.31 occurring in 2008 and 2007. These are quite small differences, however, representing only about 3% of the scale range.
Two other items included in all eight NGC surveys are 1) "How would you rate the breadth and comprehensiveness of NGC's guideline collection?" and 2) the level of agreement with the statement "NGC's guideline summaries present all the important attributes of a guideline, when this information is available." As with overall satisfaction, responses to these items have stayed consistently positive. Figure 2 below provides a historical view of this data.
Figure description: Figure 2 is a graph comparing data from 2001 to 2008 on how respondents to the Survey rated NGC in three areas: 1) satisfaction with NGC; 2) breadth and comprehensiveness of NGC's guideline collection; and 3) whether NGC's guideline summaries present all the important attributes of a guideline, when this information is available. Respondents used a Likert scale to rate NGC's performance in each of these three areas with 1 = worst and 5 = best and a mean score was derived. For rating satisfaction with NGC, the mean scores by year were as follows: 4.29 for 2008; 4.31 for 2007; 4.35 for 2006; 4.4 for 2005; 4.42 for 2004; 4.41 for 2003; 4.42 for 2002; and 4.31 for 2001. For the breadth and comprehensiveness of NGC's guideline collection, the mean scores by year were: 4.1 for 2008; 4.06 for 2007; 4.11 for 2006; 4.17 for 2005; 4.17 for 2004; 4.12 for 2003; 4.13 for 2002; and 4.04 for 2001. For the question on whether NGC's guideline summaries present all the important attributes of a guideline, when this information is available, the mean scores by year were: 4.24 for 2008; 4.16 for 2007; 4.24 for 2006; 4.28 for 2005; 4.29 for 2004; 4.33 for 2003; 4.36 for 2002; and 4.3 for 2001.
We also performed trend analyses for NQMC and found no changes in users' ratings over the past 4 years (see Figure 3 below). This was true for recommending NQMC to others, ratings of breadth/comprehensives of the quality measures collection, and completeness of the available information on a given quality measure.
Figure description: Figure 3 is a graph comparing data from 2005 to 2008 on how respondents to the Survey rated NQMC in two areas: 1) completeness of …overall description of the measure; and 2) breadth and comprehensiveness of NQMC's quality measure collection. Respondents used a Likert scale to rate NQMC's performance in each area with 1 = worst and 5= best and a mean score was derived. For rating the completeness of …overall description of the measure, the mean scores by year were as follows: 4 for 2008; 4.1 for 2007; 4.03 for 2006; and 4.01 for 2005. For the breadth and comprehensiveness of NQMC's quality measure collection, the mean scores by year were: 4.44 for 2008; 4.48 for 2007; 4.36 for 2006; and 4.4 for 2005.
Generally, the users of the NGC and NQMC clearinghouses are satisfied with the guidelines and measures included, as well as the content of that material. One area identified for future improvement is ease of navigation. NGC user comments suggest that improvements need to be made to increase relevance of search results. NGC users tend to be more clinically oriented and are more dispersed internationally than NQMC users. Also, both Web sites are attracting a large number of new users, as evidenced by the continued finding that about half of the survey respondents had never previously visited the Web site.