Methods Used in the Website of the National Healthcare Quality and Disparities ReportsLast Updated: September 2017
- Organization of the NHQR/NHDR Measures Within the Integrated NHQR/NHDR Website
- Available Comparisons
- National and State-Level Comparisons With Achievable Benchmarks
- National Trends Over Time
- State Snapshot Comparisons
- National View
- State View
- Data Query
- Internet Citation
- MEPS Methods
- HCUP Methods
The integrated National Healthcare Quality and Disparities Reports (NHQR/NHDR) website provides comprehensive information about health care developments and overviews for policymakers, legislators, and reporters. It also contains detailed data tables for researchers. Comparative information, such as trends over time and current distance from achievable benchmarks, is easily accessible. The user can also “drill down” to more detailed information to demonstrate connections between the broader and more elemental levels.
The design of this website was based on recommendations from the Institute of Medicine (IOM) on how to improve upon the NHQR/NHDR reports and related products. The aim was to facilitate greater impact regarding quality improvement and disparities elimination1. A few key goals served as the foundation of the integrated NHQR/NHDR website design:
- Bring together the NHQR/NHDR and related products in a way that produces a cohesive story about health care quality
- Present information in forward-looking and action-oriented formats
- Highlight equity as an integral part of overall quality, rather than as a separate story line
- Present data as a solution to health care disparities, with prospects for evidence-based practices
- Improve overall navigation and usability of the site for various audiences
The NHQR/NHDR website integrates information that was previously accessible from three different websites:
The following additional features were added to address the IOM recommendations:
- Benchmarks based on performance of the top 10 percent of States, to encourage achievable goals
- Displays that provide consistent definitions and comparisons across national and State levels
- Displays of racial and ethnic comparisons, as a part of the larger quality evaluation
- Provision of subject areas and topics that simplify access to the data and information
This Methods Report includes the following sections:
- An overview of the organization and presentation of materials
- An introduction to each of the five sections of the integrated NHQR/NHDR Web site—Reports, National View, State View, Data Query, and Resources
- A description of the methods behind the various graphics and data presentations within the five sections
The NHQR/NHDR reports include approximately 300 different measures that are collected from more than three dozen organizations, including the Agency for Healthcare Research and Quality (AHRQ), the Centers for Disease Control and Prevention (CDC), the National Center for Health Statistics (NCHS), and the Centers for Medicare & Medicaid Services (CMS). The term measure is used to define a specific metric, such as “Adults who received a blood cholesterol measurement in the last 5 years” or “Deaths per 1,000 adult hospital admissions with congestive heart failure.”
To facilitate the evaluation of comprehensive overviews and trends, the 300 individual measures are categorized into seven subject areas-
- Diseases and conditions
- Priority populations
- Health insurance
- Access to care
- Types of care
- Settings of care
Each of the seven subject areas is further divided into three or more topics, which are provided in Table 1.
|Diseases and Conditions|
|Chronic Kidney Disease||Diabetes|
|HIV and AIDS||Mental Health and Substance Abuse|
|Musculoskeletal Disease||Respiratory Diseases|
|High Income||Low Income|
|Non-Hispanic White||Residents of Rural Areas|
|Adults with Basic Activity Limitations||Adults with Complex Activity Limitations|
|Health Insurance, age under 65|
|Access to Care|
|Type of Care|
|Acute Care||Chronic Care|
|Setting of Care|
|Patient Safety||Person-Centered Care|
|Care Coordination||Effective Treatment|
|Healthy Living||Care Affordability|
A measure is often reported under more than one subject. For example, the measure “Adults who received a blood cholesterol measurement in the last 5 years” is included under the Cardiovascular Disease within Diseases and Conditions, under Prevention within Types of Care, and under Ambulatory within Setting of Care.
For all of the subject areas except Priority Populations and Health Insurance, the collection of measures is based on chapters in the NHQR/NHDR Report. The measures that are reported under Priority Population and Health Insurance are limited to those that have sufficient data to include them as a reporting category. For example, for “Adults who received a blood cholesterol measurement in the last 5 years,” an estimate for females is included under the Priority Population of Women; however, this measure is not included under the Priority Population of NHOPI, because the data source does not provide an estimate for this racial group. In other words, only a subset of NHQR/NHDR measures has sufficient data to support reporting under Priority Populations and Health Insurance.
Within the different subject areas and topics, the integrated NHQR/NHDR website presents three types of comparative information-
- National and State-level comparisons with achievable benchmarks
- National trends over time
- State-level comparisons with the natinal overall estimates (i.e., State Snapshots)
National trends are always highlighted in the NHQR/NHDR reports. State-level comparisons with all-State averages had been highlighted in the State Snapshot. The addition of the comparison with achievable benchmarks was based on the IOM recommendations to promote best-in-class achievement at both the national and State levels.
The incorporation of comparisons with achievable benchmarks sets standards based on the performance of the top 10 percent of States. These standards are considered achievable because they have already been attained by the best performing States.
State-level benchmarks are only determined for NHQR/NHDR measures that have State-level data for at least 30 States. The benchmark for a measure is calculated as a straight average of the top 10 percent of reporting States. For example, if data from 50 States are available for a measure, then the benchmark is the average of the measure estimate across the best five States. Since the 2016 report, States without 2013 or later data have been excluded from the benchmark calculation.
Measure estimates for the Nation, topics, State overall, and topics within each State are then compared with the State-level benchmark for that measure. Based on this comparison, national and State-level measure estimates are assigned to one of three categories: achieved the benchmark or better, close to the benchmark, or far away from the benchmark. Since the 2016 report, measures and topics without 2013 or later data have been excluded from the benchmark comparison. These categories are defined below.
- Achieved the benchmark or better. The value for a measure is no worse than 90 percent of the benchmark value. This includes the case in which the measure’s value is equal to or better than the benchmark.
- Close to the benchmark. The value for a measure is between 50 percent and 90 percent of the benchmark (i.e., the value is worse than the benchmark; it has achieved at least half but not as much as 90 percent of the benchmark).
- Far away from the benchmark. The value for a measure has not achieved 50 percent of the benchmark.
When comparing the estimates to the benchmarks, one critical consideration is the direction of the outcome. Specifically, a low value is the desired outcome for some measures, such as mortality, unmet needs, and communication problems. In contrast, a high value is the desired outcome for other measures, such as receiving recommended tests or reporting good communication. Consider a measure in which a low number is desired, such as “Adults who needed to see a specialist in the last 12 months who found it difficult to see a specialist.” If the benchmark is 18.1 and the national estimate is 14.6, then the measure achieved the benchmark—its value is 19 percent lower than the benchmark (the preferred direction). If the national estimate is 22.0, then it would be considered close to the benchmark—its value is 20 percent higher than the benchmark. If the national estimate is greater than 27.2, the measure would be considered far away from the benchmark—its value is more than 50 percent larger.
National trends are determined for NHQR/NHDR measures with at least 4 years of data. The time span can vary across measures. Since the 2016 report, the average annual percent change (AAPC) has been estimated using unweighted log-linear regression.
Model: ln(M) = β0 + β1Y,
where ln(M) is the natural logarithm of the aligned rate, β0 is the intercept or constant, and β1 is the coefficient corresponding to year Y
- Improving = Average annual percentage change >1% per year in a favorable direction and p <0.10.1.
- Worsening = Average annual percentage change >1% per year in an unfavorable direction and p <0.10.
- No Change = Average annual percentage change ≤1% per year or p ≥0.10.
Under State Snapshot, each State’s overall performance is compared with the national overall estimate. State by topic performance is also compared with the national overall estimate, not the national estimate for a particular topic. This is in contrast to the comparison with an achievable benchmark in which the comparison is with the average of the top 10 percent of States.
The national overall estimate is the estimate from micro data. If it is not available, all-State average is calculated as the national overall estimate.
For each measure, State performance is categorized as better-than-average, average, or worse-than-average relative to the all-State average.
These comparison categories are defined based on a statistical test for differences:
- Better-than-average. The State rate on an NHQR measure is better than the all-State average, and that difference is statistically significant.
- Average. The State rate on an NHQR measure is not statistically different from the all-State average.
- Worse-than-average. The State rate on an NHQR measure is worse than the all-State average, and that difference is statistically significant.
Since the 2016 report, two criteria have been used to define the difference between the State rate and the national average:
- The absolute difference must be statistically significant with p <0.05 on a two-tailed test.
- The relative difference must be at least 10% when framed negatively.
Across a group of measures within a subject area and topic, each State receives a performance meter score. First, points are assigned to each measure as follows:
- 1 point for each State-level measure that was better than the all-State average
- 0.5 points for each State-level measure that was average, relative to the all-State average
- 0 points for each State-level measure that was worse than the all-State average
Next, the points are combined into a meter score:
A = number of better-than-average NHQR measures in the summary
B = number of average NHQR measures in the summary
C = number of worse-than-average NHQR measures in the summary
Meter scores range from 0 (all measures are worse than average) to 100 (all measures are better than average). Scores between 0 and 100 represent the mix of measures that are worse than average, average, and better than average. Higher scores represent better performance, because the score increases with the number of measures that are average and increases more rapidly with the number of measures that are better than average.
A 180-degree colored semicircle that is divided into five categories is used for visual presentation of the data. The five categories are:
- Very Weak: 0 ≤ score < 20
- Weak: 20 ≤ score < 40
- Average: 40 ≤ score < 60
- Strong: 60 ≤ score < 80
- Very Strong: 80 ≤ score ≤ 100
The meter score for a given measure is depicted on the semicircle as a performance meter arrow as shown in Figure 1. A solid arrow is used for the most recent year of available data, if there are at least five measures available. A dashed arrow is used to show performance for the baseline year, when the baseline has more than two-thirds of the measures available in the most recent year. This criterion is applied to ensure similar comparisons between the baseline and the most recent year. For the 2016 report, measures with the latest year before 2013 were excluded from the most recent year’s comparison.
The State overall meter scores are more reliable and stable than the State by topic scores mainly because the number of measures with valid data for each topic within a State is usually much smaller than that for the State overall. The number of data years and measure types with valid data may also affect the meter scores. Please keep these factors in mind when interpreting the meter scores and check the underlying data for details.
The Reports section of the integrated NHQR/NHDR website allows access to the following:
There are web-enabled tables of contents for the NHQR/DR reports, which link directly to the most current report that is viewable on the AHRQ website. The reports provide the foundation on which all other sections of the integrated NHQR/DR website stand.
The section on Related Reports provides links to supplemental reports from the IOM, AHRQ, CDC, and National Academy for State Health Policy (NASHP).
The Fact Sheet section includes additional information on AHRQ research topics and programs: aging, children’s health, computers and medical informatics, coronary artery disease, diabetes, health care costs, health information technology, health literacy and cultural competency, health services research, hypertension, managed care, Medicare and Medicaid, medical errors and patient safety, mental health, minority health, outcomes research, patient-centered care, pharmaceutical research, pneumonia, primary care, health care quality, translating research into practice, women’s health, AHRQ tools, and a purchaser-provider synergies overview.
The National View section of the NHQR/NHDR integrated website provides an overview of performance across NHQR/NHDR measures. This perspective shows areas of strengths and weaknesses at a glance. There are two types of comparative information provided in this section—
- Comparison with “achievable benchmarks” for current data
- Trends over time
This information is available across all NHQR/NHDR measures and by subject area and topic.
On the National View page, there are comparisons with achievable benchmarks: across all measures which are organized in several categories, and by measure areas and topics.The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons With Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement (Figure 2). If a measure does not have an available benchmark (i.e., there are no State data from which to calculate a benchmark), it is not represented in the summary graphic. In the example in Figure 2, benchmarks are available for 124 measures: 29 measures that are far away from the benchmark, 58 measures that are close to the benchmark, and 37 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 3). For measures that report data, the comparison to benchmarks is provided for individuals who are Hispanic and non-Hispanic White, Black, and Asian and Pacific Islander (API).
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 4). For measures that report data, the comparison with benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 5.
Within the National View, the user may also view summary results of trending over time over all measures and by measure areas and topics. The methods for the trend comparison are detailed under the section of this report titled National. Since the 2016 report, the trending method is the same as for the report.
The trends summary graphic displays the total frequencies of each category of achievement. Measures that do not have at least four years of data to document the trend are shown in the “data not available” column. Selecting topic, to get the trends summary graphics for each topic. In Figure 6, there are 84 measures showing improvement over time, 58 measures with no change, and 22 measures worsening over time. There is no trending information for 130 measures, which represent 44 percent of all of the measures in this figure— 130/(130 + 84+58+22).
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the annual average percent change, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 7.
The State View section of the NHQR/NHDR integrated website is similar to the National View in that it provides an overview of performance across NHQR/NHDR measures and by measure areas and topics, but the overview provides a State’s overall performance and a State’s performance for each topic. This perspective shows each State’s areas of strengths and weaknesses at a glance. Two types of comparative information are provided in this section—
- Comparison with “achievable benchmarks” for current data
- State Snapshot comparisons with all-State averages
This information is available across all State-level measures and by subject area and topic.
Users begin by selecting a State from the selection map or by using the drop-down box (Figure 8).
Similar to the National View page, the first comparative graphic under State View is a summary of quality measures compared with achievable benchmarks. Three summaries are available: across all State-specific measures, by race and ethnicity, and by community income. Summary by race/ethnicity is also available by default. Select a measure area then a topic for summary results for a particular topic.
The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons with Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement, where State-specific data and benchmarks are available (Figure 9). If a measure does not have an available benchmark, it is not represented in the summary graphic. In the example in Figure 8, benchmarks are available for 155 measures reported for the State: 37 measures that are far away from the benchmark, 70 measures that are close to the benchmark, and 48 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 10). For measures that report data for the selected State, the comparison with benchmarks is provided for individuals who are White, Black, Hispanic, and Asian and Pacific Islander (API).
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 11). For measures that report data for the selected State, the comparison with benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 12.
The State Snapshot provides insight into a selected State’s overall performance or by topic within a State by comparing it to the national average for the available measures. The methods for the State Snapshot are detailed under the section of this report titled State Snapshot Comparisons.
Figure 13 shows the State Snapshot performance meter for a sample State. By comparing the solid blue arrow to the dashed arrow on the figure, the user can see that the State performed better in the most recent data year than it had performed in the past (baseline year). The most recent score was 43.75 across all measures, which is better than its baseline score of 41.9. In addition, the State’s performance is in the “average” range relative to the comparisons with the all-State averages. Beneath the performance meter, there is a table that lists the meter score for all States. The list can be sorted by State or by descending meter score. Select a measure area and a topic unver “State View” for summary results for a particular topic.
The user can obtain information about (1) which measures are included in the performance meter score and (2) the average annual percent change, by clicking on any of the bar graphics or on the link labeled “Review underlying data.” An example is provided in Figure 14.
The State Dashboard provides “snapshot” information across all subject areas and topics on one web page. The first graphic is the State Snapshot performance meter across all available State-level measures. Next, rectangular versions of the performance meter present information on each subject area and topic. The methods for the State Snapshot meters are detailed under the section of this report titled State Snapshot Comparisons.
Figure 15 shows a portion of a sample State Dashboard. The solid blue triangles indicate performance in the most recent data year; the white triangles indicate performance in the baseline year. The State’s performance has improved in the areas of cancer, cardiovascular disease, chronic kidney disease, and diabetes. For chronic kidney disease, the State’s performance improved from the average range to the strong range from the baseline to the most recent year. Quality performance in the area of HIV/AIDS has not changed from the baseline year to the most recent year, as indicated by two arrows in the same location. The meters for mental health and substance abuse and for musculoskeletal disease are portrayed in gray because there are no data on these measures for the selected State.
The data query section of the NHQR/NHDR integrated website provides detailed analytic capabilities that allow the user to view individual measures rather than summaries of performance. Figure 16 contains a copy of the data query selection form. The user must complete the following hierarchy to narrow the selection:
- Geographic area—national or a State
- Subject area—diseases and conditions, priority populations, health insurance, access to care, types of care, or settings of care
- Topic within the subject area.
These selections define the list of available measures. Only one measure can be selected at a time. The user can choose to examine the measure over time or by one or two categories. The available categories are determined by the measure and depend on the available data reported to the NHQR/NHDR from the original data source.
Based on the user’s selections, the data query will return a data table of the information. The information is a subset of the rows in the full NHQR/NHDR data table for the chosen measure. The results of a query on the national level for the cardiovascular measure “Deaths per 1,000 hospital admissions with congestive heart failure (CHF)” are shown in Figure 17, by ethnicity and gender. The full NHQR/NHDR table is available and can be saved by clicking on the link of “Excel — Open/Save File.” .
A short description about the source data is available by clicking on the link of “More Information” under the selected data table. Click “Measure Specification” under the table for the definition of the selected measure.
A graphic depiction of the selected data query is available by selecting the “Graphic”tab above the data table. When the query is specific to trends over time, a line graph is displayed (Figure 18). When one or two categories are selected, bar graphs are used to depict the data (Figure 19).
The Resources section of the NHQR/NHDR integrated website contains a catalog of publications and web tools. These resources provide information that is useful to quality improvement and disparities reduction activities. They are categorized into seven focal areas—
- Raising awareness
- Collecting data
- Analyzing data
- Reporting data
- Identifying best practices
- Tracking success
- Focusing on specific vulnerable populations
The source, title, and a brief description are provided for each resource. Figure 20 shows an example of the resources under Focusing on Specific Vulnerable Populations.
Methods Used in the Website of the National Healthcare Quality and Disparities Reports. September 2017. Rockville, MD: Agency for Healthcare Research and Quality.