Methods Used on the Website of the National Healthcare Quality and Disparities ReportsLast Updated: May 2019
- What’s New
- Organization of the NHQDR Measures Within the Integrated Website
- Available Comparisons
- National and State-Level Comparisons With Achievable Benchmarks
- National Trends Over Time
- State Snapshot Comparisons
- National View
- State View
- Data Query
The integrated National Healthcare Quality and Disparities Reports (NHQDR, previously NHQR/NHDR) website provides comprehensive information about healthcare developments and overviews for policymakers, legislators, and reporters. It also contains detailed data tables for researchers. Comparative information, such as trends over time and current distance from achievable benchmarks, is easily accessible. The user can also “drill down” to more detailed information to demonstrate connections between the broader and more elemental levels.
The design of this website was based on recommendations from the National Academy of Medicine (NAM) (formerly Institute of Medicine) on how to improve upon the NHQDR and related products. The aim was to facilitate greater impact regarding quality improvement and disparities elimination. A few key goals served as the foundation of the integrated NHQDR website design:
- Bring together the NHQDR and related products in a way that produces a cohesive story about healthcare quality
- Present information in forward-looking and action-oriented formats
- Highlight equity as an integral part of overall quality, rather than as a separate story line
- Present data as a solution to health care disparities, with prospects for evidence-based practices
- Improve overall navigation and usability of the site for various audiences
The NHQDR website integrates information that was previously accessible from three different websites (NHQR/NHDR, State Snapshots, and Data Query). The following features were added to address the NAM recommendations:
The following additional features were added to address the NAM recommendations:
- Benchmarks based on performance of the top 10 percent of States, to encourage achievable goals
- Displays that provide consistent definitions and comparisons across national and State levels
- Displays of racial and ethnic comparisons, as a part of the larger quality evaluation
- Provision of subject areas and topics that simplify access to the data and information
This Methods Report includes the following sections:
- An overview of the organization and presentation of materials
- An introduction to each of the five sections of the integrated NHQDR website—Reports, National View, State View, Data Query, and Resources
- A description of the methods behind the various graphics and data presentations within the five sections
This section covers major changes to the website, data, and analysis methods for 2018 and 2017.
The “Opioids” panel is new for the 2018 report. This panel provides data, analytic results, charts, and State quartile maps for measures related to opioid use or abuse. Users can select a measure and look for more details by selecting demographic areas. The 2018 report has seven opioid measures (two core measures and five supplemental measures). The supplemental measures can be accessed on the website.
Data Limitations and Changes. Before 2018, the NHQDR had about 50 core measures that used Healthcare Cost and Utilization Project (HCUP) data. However, HCUP State data and trend data are not available for the 2018 report due to a change from International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes to Tenth Revision (ICD-10) codes and AHRQ Quality Indicator changes. Only 2016 national data are included.
Data for about 20 nursing home care measures and 30 home health care measures were not available for the 2016 and 2017 reports. For the 2018 report, 2013-2016 data are available and are included in the comparisons and trend analyses.
Data availability may affect the analysis due to differences in properties of measures and source data, as well as population subgroup sizes.
Consistency Between the Report and the Website. Since 2016, the NHQDR team has been working to make the analysis methods and results consistent between the website and the report. However, inconsistencies still exist. For the 2018 report, nursing home data became available after the report was completed. Therefore, the report findings do not include the nursing home measures, but they can be accessed on the website. All available national data for the opioid-related healthcare supplemental measures are found in the website data analysis, while only newly added opioid-related healthcare measures are discussed in the report.
Benchmark Year. The benchmark year for the 2018 report did not move forward. In previous years, the benchmark year usually moved forward by a year. The benchmark year was 2014 for the 2016 report and 2015 for the 2017 report. For 2018, the benchmark year is still 2015. Therefore, there are more measures for which States and subgroups reached the benchmark in 2018 compared with previous years.
For the 2017 report, data for about 200 noncore measures are included on the Data Query page. Composite measures and other primary measures are integrated into the existing seven subject areas. Supplemental measures are organized into six topics under a new subject are of Supplemental Measures.
- Core Measures
- Composite Measures
- Other Measures
- Supplemental Measures
- Access to Care
- Person-Centered Care
- Care Coordination
- Effective Treatment
- Healthy Living
- Care Affordabilit
Data for opioid-related measures are included in all types of data analysis. Data for other noncore measures are excluded from all types of data analysis.
The NHQDR reports include approximately 300 different measures that are collected from more than three dozen organizations, including the Agency for Healthcare Research and Quality (AHRQ), the Centers for Disease Control and Prevention (CDC), the National Center for Health Statistics (NCHS), and the Centers for Medicare & Medicaid Services (CMS). The term measure is used to define a specific metric, such as “Adults who received a blood cholesterol measurement in the last 5 years” or “Deaths per 1,000 adult hospital admissions with congestive heart failure.”
To facilitate the evaluation of comprehensive overviews and trends, the 300 individual measures are categorized into seven areas:
- Priority Areas
- Diseases and conditions
- Priority Populations
- Health Insurance
- Access to Care
- Type of Care
- Setting of Care
Each of the seven areas is further divided into three or more topics:
Diseases and Conditions
- Cardiovascular Disease
- Chronic Kidney Disease
- HIV and AIDS
- Musculoskeletal Disease
- Mental Health and Substance Abuse
- Respiratory Diseases
- High Income
- Low Income
- Residents of Rural Areas
- Adults With Basic Activity Limitations
- Adults With Complex Activity Limitations
- Older Adults
- Native Hawaiians/Pacific Islanders (previously Native Hawaiians and Other Pacific Islanders)
- American Indians/Alaska Natives
- Non-Hispanic Whites
Access to Care
- Structural Access
- Patient Centeredness
Type of Care
- Acute Care
- Chronic Care
- Patient Safety
Setting of Care
- Home Health and Hospice
- Nursing Home
A measure is often reported under more than one subject. For example, the measure “Adults who received a blood cholesterol measurement in the last 5 years” is included under Cardiovascular Disease within Diseases and Conditions, under Prevention within Type of Care, and under Ambulatory within Setting of Care.
For all of the subject areas except Priority Populations and Health Insurance, the collection of measures is based on topic areas in the NHQDR. The measures reported under Priority Populations and Health Insurance are limited to those with sufficient data to include them as a reporting category.
For example, for “Adults who received a blood cholesterol measurement in the last 5 years,” an estimate for females is included under the Priority Population of Women; however, this measure is not included under the Priority Population of NHPI (Native Hawaiian/Pacific Islander), because the data source does not provide an estimate for this racial group. In other words, only a subset of NHQDR measures has sufficient data to support reporting under Priority Populations and Health Insurance.
Within the different subject areas and topics, the integrated NHQDR website presents three types of information:
- National and State-level comparisons with achievable benchmarks
- National trends over time
- State-level comparisons with the natinal overall estimates (i.e., State Snapshots)
The addition of the comparison with achievable benchmarks was based on the IOM recommendation to promote best-in-class achievement at both the national and State levels. National trends are always highlighted in the NHQDR. State-level comparisons with all-State averages are highlighted in the State Snapshots.
The incorporation of comparisons with achievable benchmarks sets standards based on the performance of the top 10 percent of States. These standards are considered achievable because they have already been attained by the best performing States.
State-level benchmarks are only determined for NHQDR measures that have State-level data for at least 30 States. The benchmark for a measure is calculated as a straight average of the top 10 percent of reporting States. For example, if data from 50 States are available for a measure, then the benchmark is the average of the measure estimate across the best five States.
States without 2013 or later data were excluded from the benchmark calculation for the 2016 report. For the 2017 and 2018 reports, 2015 data were selected first if available; otherwise, 2016 or 2014 data were selected.
Measure estimates for the Nation, topics, State overall, and topics within each State are then compared with the State-level benchmark for that measure. Based on this comparison, national and State-level measure estimates are assigned to one of three categories: achieved the benchmark or better, close to the benchmark, or far away from the benchmark. These categories are defined below:
- Achieved the benchmark or better. The value for a measure is no worse than 90 percent of the benchmark value. This including cases in which the measure’s value is equal to or better than the benchmark.
- Close to the benchmark. The value for a measure is between 50 percent and 90 percent of the benchmark (i.e., the value is worse than the benchmark; it has achieved at least half but not as much as 90 percent of the benchmark).
- Far away from the benchmark. The value for a measure has not achieved 50 percent of the benchmark.
When comparing the estimates to the benchmarks, one critical consideration is the direction of the outcome. Specifically, a low value is the desired outcome for some measures, such as mortality, unmet needs, and communication problems. In contrast, a high value is the desired outcome for other measures, such as receiving recommended tests or reporting good communication.
For example, a measure could have a low number desired, such as “Adults who needed to see a specialist in the last 12 months who found it difficult to see a specialist.” If the benchmark were 18.1 and the national estimate were 14.6, then the measure would achieve the benchmark—its value would be 19 percent lower than the benchmark (the preferred direction). If the national estimate were 22.0, then it would be considered close to the benchmark—its value would be about 20 percent higher than the benchmark. If the national estimate were greater than 27.2, the measure would be considered far away from the benchmark—its value would be more than 50 percent larger.
National trends are determined for NHQDR measures with at least 4 years of data. The time span can vary across measures. Since the 2016 report, the average annual percent change (AAPC) has been estimated using unweighted log-linear regression.
Model: ln(M) = β0 + β1Y,
where ln(M) is the natural logarithm of the aligned rate, β0 is the intercept or constant, and β1 is the coefficient corresponding to year Y
- Improving = Average annual percentage change >1% per year in a favorable direction and p <0.10.1.
- Worsening = Average annual percentage change >1% per year in an unfavorable direction and p <0.10.
- No Change = Average annual percentage change ≤1% per year or p ≥0.10.
Under the State Snapshot, each State’s overall performance is compared with the national overall estimate. State performance by topic is also compared with the national overall estimate, not the national estimate for a particular topic. This approach contrasts with the comparison with an achievable benchmark in which the comparison is with the average of the top 10 percent of States.
The national overall estimate is the estimate from micro data. If these data are not available, the all-State average is calculated as the national overall estimate.
For each measure, State performance is categorized as better-than-average, average, or worse-than-average relative to the all-State average.
These comparison categories are defined based on a statistical test for differences:
- Better-than-average. The State rate on an NHQDR measure is better than the all-State average, and that difference is statistically significant.
- Average. The State rate on an NHQDR measure is not statistically different from the all-State average.
- Worse-than-average. The State rate on an NHQDR measure is worse than the all-State average, and that difference is statistically significant.
Since the 2016 report, two criteria have been used to define the difference between the State rate and the national average:
- The absolute difference must be statistically significant with p <0.05 on a two-tailed test.
- The relative difference must be at least 10% when framed negatively.
Across a group of measures within a subject area and topic, each State receives a performance meter score. First, points are assigned to each measure as follows:
- 1 point for each State-level measure that was better than the all-State average
- 0.5 points for each State-level measure that was average, relative to the all-State average
- 0 points for each State-level measure that was worse than the all-State average
Next, the points are combined into a meter score:
A = number of better-than-average NHQDR measures in the summary
B = number of average NHQDR measures in the summary
C = number of worse-than-average NHQDR measures in the summary
Example: North Dakota has 37 better-than-average measures, 61 average measures, and 11 worse-than-average measures. Thus, A = 37, B = 61, and C = 11.
(37 * 1) + (61 * 0.5) + (11 * 0)) * 100/(37 + 61 + 11)) = (37 + 30.5 + 0) * 100/109 = 67.5 * 100/109 = 6,750/109 = 61.93
Meter scores range from 0 (all measures are worse than average) to 100 (all measures are better than average). Scores between 0 and 100 represent the mix of measures that are worse than average, average, and better than average. Higher scores represent better performance, because the score increases with the number of measures that are average and increases more rapidly with the number of measures that are better than average.
A 180-degree colored semicircle divided into five categories is used for visual presentation of the data. The five categories are:
- Very Weak: 0 ≤ score < 20
- Weak: 20 ≤ score < 40
- Average: 40 ≤ score < 60
- Strong: 60 ≤ score < 80
- Very Strong: 80 ≤ score ≤ 100
The meter score for a given measure is depicted on the semicircle as a performance meter arrow as shown in Figure 1. A solid arrow is used for the most recent year of available data, if at least five measures are available. A dashed arrow is used to show performance for the baseline year, when the baseline has more than two-thirds of the measures available in the most recent year. This criterion is applied to ensure similar comparisons between the baseline and the most recent year. For the 2016 report, measures with the latest year before 2013 were excluded from the most recent year’s comparison.
The State overall meter scores are more reliable and stable than the State topic scores mainly because the number of measures with valid data for each topic within a State is usually much smaller than that for the State overall. The number of data years and measure types with valid data may also affect the meter scores. These factors should be considered when interpreting the meter scores, and underlying data should be checked for details.
The National View section of the NHQDR integrated website provides an overview of performance across NHQDR measures. This perspective shows strengths and weaknesses at a glance.
This section provides two types of comparative information:
- Comparison with “achievable benchmarks” for current data
- Trends over time
This information is available across all NHQDR measures and by subject area and topic.
The National View page provides comparisons with achievable benchmarks across all measures, which are organized in several categories; and by measure areas and topics. The methods for the benchmark comparison are detailed in the section above called National and State-Level Comparisons With Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement (Figure 2). If a measure does not have an available benchmark (i.e., no State data from which to calculate a benchmark), it is not represented in the summary graphic. In the example in Figure 2, benchmarks are available for 124 measures: 29 measures that are far away from the benchmark, 58 measures that are close to the benchmark, and 37 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 3). For measures that report data, the comparison to benchmarks is provided for individuals who are Hispanic and non-Hispanic White, Black, and Asian and Pacific Islander.
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 4). For measures that report data, the comparison with benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 5.
Within the National View, the user may also view summary results of trending over time for all measures and by measure areas and topics. The methods for the trend comparison are detailed under the section above called National Trends Over Time. Since the 2016 report, the trending method is the same as for the report.
The trends summary graphic displays the total frequencies of each category of achievement. Measures that do not have at least 4 years of data to document the trend are shown in the “data not available” column. Select topic to get the trend summary graphic for each topic. In Figure 6, 84 measures show improvement over time, 58 measures show no change, and 22 measures show worsening over time. No trending information is available for 130 measures, which represent 44 percent of all of the measures in this figure—130/(130 + 84+58+22).
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the annual average percent change, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 7.
The State View section of the NHQDR integrated website is similar to the National View in that it provides an overview of performance across NHQDR measures and by measure areas and topics, but the overview provides a State’s overall performance and a State’s performance for each topic. This perspective shows each State’s strengths and weaknesses at a glance.
This section provides two types of comparative information:
- Comparison with “achievable benchmarks” for current data
- State Snapshot comparisons with all-State averages
This information is available across all State-level measures and by subject area and topic.
Users begin by selecting a State from the selection map or by using the drop-down box (Figure 8).
Similar to the National View page, the first comparative graphic under State View is a summary of quality measures compared with achievable benchmarks. Three summaries are available: across all State-specific measures, by race and ethnicity, and by community income. Summary by race/ethnicity is also available by default. Select a measure area then a topic for summary results for a particular topic.
The methods for the benchmark comparison are detailed under the section above called National and State-Level Comparisons with Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement, where State-specific data and benchmarks are available (Figure 9). If a measure does not have an available benchmark, it is not represented in the summary graphic. In the example in Figure 8, benchmarks are available for 155 measures reported for the State: 37 measures that are far away from the benchmark, 70 measures that are close to the benchmark, and 48 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 10). For measures that report data for the selected State, the comparison with benchmarks is provided for individuals who are White, Black, Hispanic, and Asian and Pacific Islander.
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 11). For measures that report data for the selected State, the comparison with benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 12.
The State Snapshot provides insight into a selected State’s performance overall or by topic within a State by comparing it with the national average for the available measures. The methods for the State Snapshot are detailed under the section above called State Snapshot Comparisons.
Figure 13 shows the State Snapshot performance meter for a sample State. By comparing the solid blue arrow to the dashed arrow on the figure, the user can see that the State performed better in the most recent data year than it had performed in the past (baseline year). The most recent score was 43.75 across all measures, which is better than its baseline score of 41.9. In addition, the State’s performance is in the “average” range relative to the comparisons with the all-State averages.
Beneath the performance meter is a table that lists the meter score for all States. The list can be sorted by State or by descending meter score. Select a measure area and a topic under “State View” for summary results for a particular topic.
The user can obtain information about (1) which measures are included in the performance meter score and (2) the average annual percent change, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 14.
The State Dashboard provides “snapshot” information across all subject areas and topics on one web page. The first graphic is the State Snapshot performance meter across all available State-level measures. Next, rectangular versions of the performance meter present information on each subject area and topic. The methods for the State Snapshot meters are detailed under the section above called State Snapshot Comparisons.
Figure 15 shows a portion of a sample State Dashboard. The inverted triangles (blue) indicate performance in the most recent data year; the regular triangles (white) indicate performance in the baseline year. The State’s performance improved in the areas of cancer and chronic kidney disease. Performance declined for cardiovascular disease and diabetes.
For chronic kidney disease, the State’s performance improved from the average range to the strong range from the baseline to the most recent year. Quality performance in the areasof mental health and substance abuse and respiratory diseases has not changed from the baseline year to the most recent year, as indicated by two triangles in the same location. The meters for HIV/AIDS and for musculoskeletal disease are in gray because the State has no data on these measures.
The data query section of the NHQDR integrated website provides detailed analytic capabilities that allow the user to view individual measures rather than summaries of performance. Figure 16 shows the data query selection form. The user must complete the following fields to narrow the selection:
- Geographic area—national or a State
- Subject area—diseases and conditions, priority populations, health insurance, access to care, types of care, or settings of care
- Topic within the subject area.
These selections define the list of available measures. Only one measure can be selected at a time. The user can choose to examine the measure over time or by one or two categories. The available categories are determined by the measure and depend on the available data reported to the NHQDR from the original data source.
Based on the user’s selections, the data query will return a data table of the information. The information is a subset of the rows in the full NHQDR data table for the chosen measure. The results of a query on the national level for the cardiovascular measure “Deaths per 1,000 hospital admissions with congestive heart failure (CHF)” are shown in Figure 17. The full NHQDR table is available and can be saved by selecting “Excel — Open/Save File.”
A short description about the source data is available beneath the table. Users can get additional details by selecting “More Information” at the end of the source description. Select “Measure Specification” under the table for information on how the measure is defined (e.g., numerator, denominator).
Users can view a graphic depiction of the selected data query by selecting the “Graphic” tab above the data table. When the query is specific to trends over time, a line graph is displayed (Figure 18). When one or two categories are selected, bar graphs are used to depict the data (Figure 19).
The Reports section of the integrated NHQDR website includes the following:
This section links to the files of the most recent NHQDR and to the web pages for previous reports. Chartbooks include focused data on the NHQDR priority areas and various priority populations. The section on Related Reports links to reports from IOM, AHRQ, CDC, and the National Academy for State Health Policy that relate to access, disparities, and other issues germane to the NHQDR.
Data Spotlights present data on specific areas of interest, such as infant mortality. Fact Sheets include additional information on AHRQ research topics and programs, such as children’s health, computers and medical informatics, coronary artery disease, diabetes, healthcare costs, health literacy and cultural competency, hypertension, and patient-centered care.
The Resources section of the NHQDR integrated website contains a catalog of publications and web tools. These resources provide information that is useful to quality improvement and disparities reduction activities. They are categorized into seven focal areas—:
- Raising awareness
- Collecting data
- Analyzing data
- Reporting data
- Identifying best practices
- Tracking success
- Focusing on specific vulnerable populations
The source, title, and a brief description are provided for each resource. Figure 20 shows an example of the resources under Focusing on Specific Vulnerable Populations.
Methods Used on the Website of the National Healthcare Quality and Disparities Reports. August 2019. Rockville, MD: Agency for Healthcare Research and Quality.