The integrated National Healthcare Quality and Disparities Reports (NHQDR, previously NHQR/NHDR) website provides comprehensive information about health care developments and overviews for policymakers, legislators, and reporters. It also contains detailed data tables for researchers. Comparative information, such as trends over time and current distance from achievable benchmarks, is easily accessible. The user can also "drill down" to more detailed information to demonstrate connections between the broader and more elemental levels.
The design of this website was based on recommendations from the National Academy of Medicine (NAM) (formerly Institute of Medicine) on how to improve upon the NHQDR reports and related products. The aim was to facilitate greater impact regarding quality improvement and disparities elimination1. A few key goals served as the foundation of the integrated NHQDR website design:
The NHQDR website integrates information that was previously accessible from three different websites:
The following additional features were added to address the NAM recommendations:
This Methods Report includes the following sections:
This section covers major changes on the website, data, and analysis methods, with the newest changes first.
Data Limitations and Changes: Before 2018, the NHQDR had about 50 core measures using Healthcare Cost and Utilization Project (HCUP) data. However, the HCUP State data and trend data were not available for the 2019 report due to the change from ICD-91 diagnosis codes to ICD-10 codes and changes to the AHRQ Quality Indicators (same as the 2018 report). Different from the 2018 report, HCUP risk-adjusted benchmarks were used for some of the Quality Indicators for subgroup comparisons under the National View panel. National data and State data are available for the two HCUP opioid measures.
Behavioral Risk Factor Surveillance System data became available for the 2019 report and are included in all analyses.
Medical Expenditure Panel Survey activity limitation data were not available for the 2019 report or for the website.
The "Opioids" panel was new for the 2018 report. This panel provides data, analytic results, charts, and State quartile maps for measures related to opioid use or abuse. Users can select a measure on the first page and look for more details by selecting demographic areas. The 2018 report has seven opioid measures; two are core measures and five are supplemental measures. The supplemental measures are included in the website analysis, not in the report.
Data Limitations and Changes: Before 2018, the NHQDR had about 50 core measures using Healthcare Cost and Utilization Project (HCUP) data. However, the HCUP State data and trend data were not available for the 2018 report due to the change from ICD-9 diagnosis codes to ICD-10 codes and Quality Indicator changes. Only 2016 national data were included on the website.
Data for about 20 nursing home care measures and 30 home health care measures were not available for the 2016 and 2017 reports. For the 2018 report, 2013-2016 data were available and were included in the comparisons as well as trend analysis.
Data availability may affect the analysis results for various reasons, including differences in properties of measures and source data and in population subgroups in the data tables.
Consistency Between the Report and the Website: Since 2016, we have been trying to make the analysis methods and results consistent between the website and the report. However, inconsistencies still exist. For 2018, nursing home data became available after the report was prepared and we decided to include them in the website data analysis. We also decided to include all data for the opioid supplemental measures in the website data analysis because opioid-related issues are a Department of Health and Human Services priority.
Benchmark Year: The benchmark year for the 2018 report did not move forward by 1 year. In previous years, the benchmark year usually moved forward by a year. The benchmark year was 2014 for the 2016 report, and 2015 for the 2017 report. For 2018, the benchmark year was still 2015. Because of this change, more measures, States, and subgroups reached the benchmark in 2018 compared with previous years.
From the 2017 report, data for about 200 supplemental measures are included on the Data Query page. These measures are grouped in two additional subject areas and a number of topics:
1 ICD-9 is the International Classification of Diseases, Ninth Revision. ICD-10 is the 10th revision.
The NHQDR reports include approximately 300 different measures that are collected from more than three dozen organizations, including the Agency for Healthcare Research and Quality (AHRQ), the Centers for Disease Control and Prevention (CDC), the National Center for Health Statistics (NCHS), and the Centers for Medicare & Medicaid Services (CMS). The term measure is used to define a specific metric, such as “Adults who received a blood cholesterol measurement in the last 5 years” or “Deaths per 1,000 adult hospital admissions with congestive heart failure.”
To facilitate the evaluation of comprehensive overviews and trends, the 300 individual measures are categorized into seven areas:
Each of the seven subject areas is further divided into three or more topics, which are provided in Table 1.
Table 1. NHQDR Integrated Website Subject Areas and Topics
|Diseases and Conditions|
|Chronic Kidney Disease||Diabetes|
|HIV and AIDS||Mental Health and Substance Abuse|
|Musculoskeletal Disease||Respiratory Diseases|
|Residents of Rural Areas|
|Adults With Basic Activity Limitations|
|Adults With Complex Activity Limitations|
|NHPI (previously NHOPI)|
|Access to Care|
|Types of Care|
|Acute Care||Chronic Care|
|Setting of Care|
|Ambulatory||Home Health and Hospice|
A measure is often included in more than one section. For example, the measure “Adults who received a blood cholesterol measurement in the last 5 years” is included under the Cardiovascular Disease within Diseases and Conditions, under Prevention within Type of Care, and under Ambulatory within Setting of Care.
For all of the subject areas except Priority Populations and Health Insurance, the collection of measures is based on topic areas in the NHQDR. The measures that are reported under Priority Populations and Health Insurance are limited to those that have sufficient data to include them as a reporting category. For example, for “Adults who received a blood cholesterol measurement in the last 5 years,” an estimate for females is included under the Priority Population of Women; however, this measure is not included under the Priority Population of NHPI (Native Hawaiian/Pacific Islander), because the data source does not provide an estimate for this racial group. In other words, only a subset of NHQDR measures has sufficient data to support reporting under Priority Populations and Health Insurance.
Within the different subject areas and topics, the integrated NHQDR website presents three types of comparative information-
National trends are always highlighted in the NHQDR reports. State-level comparisons with all-State averages had been highlighted in the State Snapshot. The addition of the comparison with achievable benchmarks was based on the IOM recommendations to promote best-in-class achievement at both the national and State levels.
The incorporation of comparisons with achievable benchmarks sets standards based on the performance of the top 10 percent of States. These standards are considered achievable because they have already been attained by the best performing States.
State-level benchmarks are only determined for NHQDR measures that have State-level data for at least 30 States. The benchmark for a measure is calculated as an average of the top 10 percent of reporting States. For example, if data from 50 States are available for a measure, then the benchmark is the average of the measure estimate across the best five States. States without 2013 or later data were excluded from the benchmark calculation for the 2016 report. For the 2017 and 2018 report, 2015 data were selected first if available; otherwise, 2016 or 2014 data were selected.
Measure estimates for the Nation, topics, State overall, and topics within each State are then compared with the State-level benchmark for that measure. Based on this comparison, national and State-level measure estimates are assigned to one of three categories: achieved the benchmark or better, close to the benchmark, or far away from the benchmark. These categories are defined below.
When comparing estimates with the benchmarks, one critical consideration is the direction of the outcome. Specifically, a low value is the desired outcome for some measures, such as mortality, unmet needs, and communication problems. In contrast, a high value is the desired outcome for other measures, such as receiving recommended tests or reporting good communication. Consider a measure in which a low number is desired, such as “Adults who needed to see a specialist in the last 12 months who found it difficult to see a specialist.” If the benchmark is 18.1 and the national estimate is 14.6, then the measure achieved the benchmark—its value is 19 percent lower than the benchmark (the preferred direction). If the national estimate is 22.0, then it would be considered close to the benchmark—its value is 20 percent higher than the benchmark. If the national estimate is greater than 27.2, the measure would be considered far away from the benchmark—its value is more than 50 percent larger.
National trends are determined for NHQDR measures with at least 4 years of data. The time span can vary across measures. Beginning with the 2016 report, the average annual percent change (AAPC) has been estimated using unweighted log-linear regression.
Model: ln(M) = β0 + β1Y,
where ln(M) is the natural logarithm of the aligned rate, β0 is the intercept or constant, and β1 is the coefficient corresponding to year Y
Under State Snapshot, each State’s overall performance is compared with the national overall estimate. State performance by topic is also compared with the national overall estimate not the national estimate for a particular topic. This approach is in contrast to the comparison with an achievable benchmark in which the comparison is with the average of the top 10 percent of States.
The national overall estimate is the estimate from micro data. If it is not available, the all-State average is calculated as the national overall estimate.
For each measure, State performance is categorized as better-than-average, average, or worse-than-average relative to the all-State average.
These comparison categories are defined based on a statistical test for differences:
Beginning with the 2016 report, two criteria have been used to define the difference between the State rate and the national average:
Across a group of measures within a subject area and topic, each State receives a performance meter score. First, points are assigned to each measure as follows:
Next, the points are combined into a meter score:
A = number of better-than-average NHQDR measures in the summary
B = number of average NHQDR measures in the summary
C = number of worse-than-average NHQDR measures in the summary
Example: North Dakota has 37 better-than-average measures, 61 average measures, and 11 worse-than-average measures. Thus, A = 37, B = 61, and C = 11.
(37 * 1) + (61 * 0.5) + (11 * 0)) * 100/(37 + 61 + 11)) = (37 + 30.5 + 0) * 100/109 = 67.5 * 100/109 = 6,750/109 = 61.93
Meter scores range from 0 (all measures are worse than average) to 100 (all measures are better than average). Scores between 0 and 100 represent the mix of measures that are worse than average, average, and better than average. Higher scores represent better performance, because the score increases with the number of measures that are average and increases more rapidly with the number of measures that are better than average.
A 180-degree colored semicircle divided into five categories is used for visual presentation of the data. The five categories are:
The meter score for a given measure is depicted on the semicircle as a performance meter arrow as shown in Figure 1. A solid arrow is used for the most recent year of available data, if at least five measures are available. A dashed arrow is used to show performance for the baseline year, when the baseline has more than two-thirds of the measures available in the most recent year. This criterion is applied to ensure similar comparisons between the baseline and the most recent year. For the 2016 report, measures with the latest year before 2013 were excluded from the most recent year’s comparison.
The State overall meter scores are more reliable and stable than the State topic scores mainly because the number of measures with valid data for each topic within a State is usually much smaller than that for the State overall. The number of data years and measure types with valid data may also affect the meter scores. Please keep these factors in mind when interpreting the meter scores and check the underlying data for details.
The National View section of the NHQDR integrated website provides an overview of performance across NHQDR measures. This perspective shows areas of strengths and weaknesses at a glance. There are two types of comparative information provided in this section—
This section provides two types of comparative information:
This information is available across all NHQDR measures and by subject area and topic.
On the National View page, there are comparisons with achievable benchmarks: across all measures, which are organized in several categories; and by measure areas and topics. The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons With Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement (Figure 2). If a measure does not have an available benchmark (i.e., no State data from which to calculate a benchmark), it is not represented in the summary graphic. In the example in Figure 2, benchmarks are available for 124 measures: 29 measures that are far away from the benchmark, 58 measures that are close to the benchmark, and 37 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 3). For measures that report data, the comparison to benchmarks is provided for individuals who are Hispanic and non-Hispanic White, Black, and Asian and Pacific Islander.
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 4). For measures that report data, the comparison with benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 5.
Within the National View, the user may also view summary results of trending over time over all measures and by measure areas and topics. The methods for the trend comparison are detailed under the section of this report titled National. Since the 2016 report, the trending method is the same as for the report.
The trends summary graphic displays the total frequencies of each category of achievement. Measures that do not have at least 4 years of data to document the trend are shown in the “data not available” column. Select topic to get the trend summary graphic for each topic. In Figure 6, there are 84 measures showing improvement over time, 58 measures with no change, and 22 measures worsening over time. There is no trending information for 130 measures, which represent 44 percent of all of the measures in this figure— 130/(130 + 84+58+22).
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the annual average percent change, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 7.
The State View section of the NHQDR integrated website is similar to the National View in that it provides an overview of performance across NHQDR measures and by measure areas and topics but, in addition, provides an overview of a State’s overall performance and a State’s performance for each topic. This approach allows the user to see each State’s areas of strengths and weaknesses at a glance. Two types of comparative information are provided in this section—
This section provides two types of comparative information:
This information is available across all State-level measures and by subject area and topic.
Users begin by selecting a State from the selection map or by using the drop-down box (Figure 8).
Similar to the National View page, the first comparative graphic under State View is a summary of quality measures compared with achievable benchmarks. Three summaries are available: across all State-specific measures, by race and ethnicity, and by community income. Summary by race/ethnicity is also available by default. Select a measure area, then a topic for summary results for a particular topic.
The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons With Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement, where State-specific data and benchmarks are available (Figure 9). If a measure does not have an available benchmark, it is not represented in the summary graphic. In the example in Figure 8, benchmarks are available for 155 measures reported for the State: 37 measures that are far away from the benchmark, 70 measures that are close to the benchmark, and 48 measures that have achieved or performed better than the benchmark.
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 10). For measures that report data for the selected State, the comparison with benchmarks is provided for individuals who are White, Black, Hispanic, and Asian and Pacific Islander.
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 11). For measures that report data for the selected State, the comparison with benchmarks is provided for low-income and high-income communities.
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 12.
The State Snapshot provides insight into a selected State’s performance overall or by topic within a State by comparing it with the national average for the available measures. The methods for the State Snapshot are detailed under the section of this report titled State Snapshot Comparisons.
Figure 13 shows the State Snapshot performance meter for a sample State. By comparing the solid blue arrow to the dashed arrow on the figure, the user can see that the State performed better in the most recent data year than it had performed in the past (baseline year). The most recent score was 43.75 across all measures, which is better than its baseline score of 41.9. In addition, the State’s performance is in the “average” range relative to the comparisons with the all-State averages. Beneath the performance meter, there is a table that lists the meter score for all States. The list can be sorted by State or by descending meter score. Select a measure area and a topic under “State View” for summary results for a particular topic.
The user can obtain information about (1) which measures are included in the performance meter score and (2) the average annual percent change, by selecting any of the bar graphs or the link labeled “Review underlying data.” An example is provided in Figure 14.
The State Dashboard provides “snapshot” information across all subject areas and topics on one web page. The first graphic is the State Snapshot performance meter across all available State-level measures. Next, rectangular versions of the performance meter present information on each subject area and topic. The methods for the State Snapshot meters are detailed under the section above called State Snapshot Comparisons.
Figure 15 shows a portion of a sample State Dashboard. The inverted triangles (blue) indicate performance in the most recent data year; the regular triangles (white) indicate performance in the baseline year. The State’s performance improved in the areas of cancer and chronic kidney disease. Performance declined for cardiovascular disease and diabetes.
For chronic kidney disease, the State’s performance improved from the average range to the strong range from the baseline to the most recent year. Quality performance in the areasof mental health and substance abuse and respiratory diseases has not changed from the baseline year to the most recent year, as indicated by two triangles in the same location. The meters for HIV/AIDS and for musculoskeletal disease are in gray because the State has no data on these measures.
The data query section of the NHQDR integrated website provides detailed analytic capabilities that allow the user to view individual measures rather than summaries of performance. Figure 16 contains a copy of the data query selection form. The user must complete the following hierarchy to narrow the selection:
These selections define the list of available measures. Only one measure can be selected at a time. The user can choose to examine the measure over time or by one or two categories. The available categories are determined by the measure and depend on the available data reported to the NHQDR from the original data source.
Based on the user’s selections, the data query will return a data table of the information. The information is a subset of the rows in the full NHQDR data table for the chosen measure. The results of a query on the national level for the cardiovascular measure “Deaths per 1,000 hospital admissions with congestive heart failure (CHF)” are shown in Figure 17, by ethnicity and gender. The full NHQDR table is available and can be saved by selecting “Excel — Open/Save File.”
A short description about the source data is available by selecting “More Information” under the selected data table. Select “Measure Specification” under the table for the definition of the selected measure.
Users can view a graphical depiction of the selected data query by selecting the “Graph” tab above the data table. When the query is specific to trends over time, a line graph is displayed (Figure 18). When one or two categories are selected, bar graphs are used to depict the data (Figure 19).
The Reports section of the integrated NHQDR website allows access to the following:
This section links to the files of the most recent NHQDR and to the web pages for previous reports. Chartbooks include focused data on the NHQDR priority areas and various priority populations. The section on Related Reports links to reports from IOM, AHRQ, CDC, and the National Academy for State Health Policy that relate to access, disparities, and other issues germane to the NHQDR.
Data Spotlights present data on specific areas of interest, such as infant mortality. Fact Sheets include additional information on AHRQ research topics and programs, such as children’s health, computers and medical informatics, coronary artery disease, diabetes, healthcare costs, health literacy and cultural competency, hypertension, and patient-centered care.
The Resources section of the NHQDR integrated website contains a catalog of publications and web tools. These resources provide information that is useful to quality improvement and disparities reduction activities. They are categorized into seven focal areas—
The source, title, and a brief description are provided for each resource. Figure 20 shows an example of the resources under Focusing on Specific Vulnerable Populations.
Methods Used on the Website of the National Healthcare Quality and Disparities Reports. December 2020. Rockville, MD: Agency for Healthcare Research and Quality.