Mortality and Complications Outcomes Methodology
TABLE OF CONTENTS
- Methodology Updates
- Data Acquisition
- Data Quality Checks
- Cohort Definitions and Outcomes
- Defining In-Hospital Complications
- Diagnosis Records With Unknown or Empty POA Indicator
- Diagnosis Records With POA Indicators
- Independent vs. Dependent Complications
- Multivariate Logistic Regression Models
- Statistically Significant Risk Factors
- Model Coefficient Summary and Fit Statistics Tables
- Adjustment for POA Fill Rate as an Additional Model Variable
- Hospital Performance
- Limitations of the Data Models
- Mortality and Complications Outcomes Methodology PDF with Appendices
At Healthgrades, our mission is to give consumers the confidence to make meaningful healthcare decisions by providing them with transparent, actionable information. We help millions of consumers each month find and connect with the right doctor, right hospital, and right care. Since 1998, Healthgrades has been dedicated to making healthcare more accessible and transparent by differentiating providers on the basis of patient satisfaction, physician experience, and hospital quality outcomes.
Our hospital quality ratings and awards give consumers the transparency they deserve—and have come to expect—when making informed choices about where to seek care. Healthgrades leverages a rigorous and impartial methodology that evaluates hospital quality based solely on clinical outcomes in the most common procedures and conditions. With our quality ratings and awards, Healthgrades helps hospitals promote their achievements, measure clinical trends, and work to continuously improve patient care and outcomes.
In accordance with our mission to help consumers make more educated healthcare decisions and to help hospitals improve the care they deliver to patients, Healthgrades analyzes clinical outcomes data for nearly every hospital in the country on an annual basis. Healthgrades helps consumers evaluate and compare hospital performance for care provided during a hospital stay for specific conditions or procedures.
Healthgrades analyzes clinical outcomes data based on Medicare-patient records in 31 common conditions and procedures, also known as cohorts, for nearly 4500 short-term acute care facilities nationwide. Additionally, for two cohorts – appendectomy and bariatric surgery – where Medicare data does not adequately represent the patient population, Healthgrades uses all-payer data available for 16 states.
The Healthgrades methodology uses a statistically rigorous approach, which includes risk adjustment, to account for differences in patient populations and adjust for specific patient conditions seen at individual facilities that can influence outcomes in significant ways.
As the health of patient populations differs from hospital to hospital, Healthgrades uses risk adjustment to account for these differences. This allows Healthgrades to obtain fair statistical comparisons of mortality and complication rates. Patient risk factors may include comorbid conditions (such as hypertension and diabetes), age, sex, and specific procedure performed.
Individual risk models are constructed and refined for each of the 33 conditions or procedures relative to each specific outcome. For each cohort, the clinical outcome is measured by either mortality (in-hospital mortality and 30-day post-admission mortality) or in-hospital complications.
Every year, Healthgrades reviews our methodology to ensure that it is up to date both clinically and statistically. As the practice of medicine changes, we incorporate changes into our methodology. Reviewers include Healthgrades physicians, nurses, and statisticians, as well as our external physician clinical advisory board. Once our clinicians approve all proposed changes, we analyze all changes for statistical relevance in the model. See our Healthgrades 2023 Methodology Updates for an in-depth explanation of all our annual methodology changes.
Healthgrades used the following data sources in the analysis of hospital patient records:
Medicare inpatient data from the Medicare Provider Analysis and Review (MedPAR) file purchased from the Centers for Medicare and Medicaid Services (CMS) for years 2019 through 2021—the most recent data set available. MedPAR data were selected for several reasons:
- Almost every hospital in the country is included in the database, with the exception of military and Veterans Administration hospitals.
- Accuracy is regulated. Hospitals are required by law to submit complete and accurate information for fee-for-service Medicare patient hospital care with substantial penalties for those that report inaccurate or incomplete data.
- The Medicare population represents most of the patients for the clinical categories studied.
- Inpatient all-payer data for the Appendectomy and Bariatric Surgery cohorts were provided by the following 16 states where state data are available:
Since the appendectomy and bariatric surgery cohorts include very few patients older than 65 years of age, all-payer data are used to evaluate clinical outcomes. Consequently, Appendectomy and Bariatric Surgery cohorts are based exclusively on the most recent state data set available (years 2018 through 2020). State data years are always one year behind MedPAR data years.
Healthgrades also uses all-payer data to evaluate and compare hospital performance in labor and delivery and gynecologic surgeries and releases those findings with our Obstetrics & Gynecology Care Excellence Awards and Ratings. For more information, see our Obstetrics & Gynecology Care Excellence Awards and Ratings methodologies.
Healthgrades conducts a series of data quality checks to preserve the integrity of the analyses. Through these analyses, patient records are identified and excluded for various reasons. For example, patient records are excluded when patients are not a clinical like-group (e.g., transplant patients are more at-risk for complications and are thus removed). Healthgrades excludes:
- Patients under the age of 65 found in the Medicare inpatient data set.
- Patients who left the hospital against medical advice.
- Patients who were transferred to another acute care hospital. The transferred patient is included in the data of the hospital that receives the transferred patient.
- Patients discharged alive with a length of stay that is inconsistent with the reason for admission. For example, a patient discharged alive after a one-day length of stay for cranial neurosurgery would be excluded because this procedure requires several days for in-hospital recovery.
- Patients who were still in the hospital when the claim was filed.
- Patients with an invalid gender (e.g., a prostatectomy related to a female patient).
- Patients with an invalid age.
- Patients who have had any organ transplant.
- Patients who are admitted from hospice.
- Patients with metastatic cancers.
In 2020, the global pandemic changed healthcare, including how and when healthcare was delivered and, in many cases, not delivered or delayed. This changed the MedPAR data set with the data showing volume drops in many areas and volume spikes in others (e.g., COVID-19 admissions). After examining the effects to the data set caused by the pandemic, Healthgrades made two changes.
First, we are excluding from our methodology any case that has a diagnosis of COVID in the 2020 and 2021 data years (January 1, 2020 - September 30, 2021). Since Healthgrades cannot currently adequately risk-adjust for the presence of COVID patients, we made the decision to exclude them. Those exclusions encompass the following ICD-10 codes in any position in the data set: U07.1, B97.29, and B34.2
Second, there are certain Healthgrades cohorts that had substantial drops in volume due to how COVID changed the practice of medicine during the pandemic (e.g., cancelling elective surgeries or patients not coming in for service due to fear of COVID transmission). While in general, our methodology delineates that a hospital must have 30 total Medicare inpatient cases over three years and five cases in the most recent data year, we changed that threshold for six cohorts this year due to loss of volume. For Heart Attack Treatment (Stent Treatment Not Available), Total Knee Replacement, Total Hip Replacement, Prostate Removal Surgery, Treatment of COPD (Chronic Obstructive Pulmonary Disease), and Community Acquired Pneumonia, the threshold this year is 15 total cases over three years of data with five cases in the most recent year.
We will continue to evaluate the impact of the pandemic on the data and to make changes as necessary to ensure Healthgrades ratings continue to reflect accurate statistical and clinical outcomes.
Healthgrades analyzes clinical outcomes (mortality and complications) independently for all 33 conditions or procedure cohorts. For each cohort, we determine a list of specific procedures and diagnoses that define the cohort as well as a list of exclusions; exclusions are rare and/or clinically complex diagnoses and procedures that cannot be adequately risk-adjusted. The purpose of the inclusions and exclusions is to create cohorts of like patients. A full list of ICD-10 codes used to define each cohort can be found in Healthgrades ICD-10 Mapping Tool at https://icd10mappingtool.healthgrades.com/.
To be eligible for a rating in each cohort, a hospital must have at least 30 cases across three years of data and at least five cases in the most recent year. Due to the impact of the COVID pandemic, the minimum number of cases over three years was adjusted for six cohorts: Total Knee Replacement, Total Hip Replacement, Prostate Removal Surgery, Heart Attack (Stent Treatment Not Available), Chronic Obstructive Pulmonary Disease (COPD), and Community Acquired Pneumonia. As a result, the number of hospitals that qualified and received a rating ranged from a low of 364 rated for Heart Attack (Stent Treatment Not Available) to a high of 3,818 rated for Community Acquired Pneumonia.
For 17 of the cohorts, clinical outcomes evaluated are in-hospital and 30-day post-admission mortality (see Table 1).
For 16 of the cohorts, the clinical outcome evaluated was the occurrence of one or more in-hospital complications (see Table 1), as defined by clinical and coding experts. A list of the ICD-10 codes that indicate in-hospital complications for each cohort can be found in the Healthgrades ICD-10 Mapping Tool.
For the analyses, clinical outcomes were dichotomous:
- Complications were documented in the patient record as either present or not present.
- Mortality was documented in the patient record as either recorded as alive or deceased. Mortality is considered a complication in all Complication Cohorts.
Table 1. Mortality and Complication Cohorts
Chronic Obstructive Pulmonary Disease (COPD)
Coronary Artery Bypass Graft (CABG) Surgery
Coronary Interventional Procedures
Upper Gastrointestinal Surgery
In-Hospital Complication Cohorts
|Abdominal Aortic Aneurysm Repair |
Back and Neck Surgeries (Without Spinal Fusion)
Gallbladder Removal Surgery
|Hip Fracture Treatment|
Peripheral Vascular Bypass
Prostate Removal Surgery
Total Knee Replacement
Transurethral Prostate Resection Surgery
For 16 cohorts, Healthgrades evaluates in-hospital complications. Complications are determined by our internal and external clinical experts, including physicians, nurses, coders, and other clinicians. Healthgrades considers a patient to have a complication if they have one or more complications during their hospital stay. Many of these complications may be preventable and usually cause a prolonged hospital stay, additional and costly medical treatments, harm, and sometimes even death.
Healthgrades assigns a complication if a diagnosis is not present upon admission. The Present on Admission (POA) indicator from the data set is used to distinguish complications that occur during a patient’s hospital stay from those present at the time the patient was admitted to the hospital.
Table 2 shows the POA indicators listed in the MedPAR Codebook (Chronic Condition Warehouse Codebook: MedPAR, November 2020, Version 3.0) and how they are used by Healthgrades to determine in-hospital complications in the current 2023 award year analyses. These indicators are identical to those in the all-payer data file.
Table 2. Present On Admission Indicators
POA Indicator (Interpretation)
Use by Healthgrades
|Present at the time of inpatient admission||Utilized in the risk-adjustment process|
(No, Not Present)
|Not present at the time of inpatient admission||Used to identify in-hospital complications|
|U, [blank], 1, Other (Unknown)||Documentation is insufficient to determine if condition was present on admission|
Unreported/not used—exempt from POA reporting
May require the co- occurrence of other codes to be a complication, see Records with No POA Indicator section below.
If not considered a complication, it may be used in risk adjustment.
|Provider is unable to clinically determine whether condition was present on admission or not.||Not considered a complication nor used in risk adjustment|
When utilizing administrative patient records, Healthgrades identifies diagnosed conditions as either pre-existing or hospital-acquired. Only the hospital-acquired conditions are considered as in-hospital complications in our analyses.
To differentiate between pre-existing and hospital-acquired conditions when there is no POA indicator or the POA indicator is U (unknown), Healthgrades uses the presence of a postoperative complication code to identify a documented occurrence of a complication.
For example, in the case where a patient record contains ICD-10 I48.0 (Atrial Fibrillation) without a POA indicator, that code is considered a comorbid risk factor if it occurs by itself and an in-hospital complication if there is one of several ICD-10 cardiac complication codes, including I97.89 (Other Postprocedural Complications and Disorder of Circulatory System, NEC), also present in the patient record.
Healthgrades does not require the presence of a postoperative complication code for diagnoses that were clearly hospital-acquired (e.g., heart attack in an elective procedure such as total knee replacement).
When there is a POA indicator present, a diagnosis is only considered an in-hospital complication if the POA indicator is set to No. This means that the condition was not present at the time of admission and was acquired during the hospitalization episode of care.
Complications are “independent” if the condition clearly occurred during a patient’s hospital stay or if the condition is defined as postoperative by the coding definition. These conditions only require a single code to be present in order to be counted as a complication. They are not counted as a complication if the POA indicator was “Yes” or “Clinically Undetermined.”
Complications described as “dependent” are conditions that must either have POA indicator set to “No” or if the POA indicator is set to “Unknown” or is missing, there must also be listed an ICD-10 postoperative complication code present in the patient record. These conditions require two codes—the condition ICD code and the ICD-10 complication code—to be considered a complication. A full list of ICD-10 codes, including postoperative complication codes used, can be found in Healthgrades ICD-10 Mapping Tool.
Healthgrades constructs and refines individual risk models for each of the 33 conditions or procedures relative to each specific outcome. The methodology uses backward stepwise multivariate logistic regression to adjust for patient risk factors that influence patient outcomes in significant and systematic ways. Risk factors may include age, sex, specific procedure performed, and comorbid conditions (e.g., hypertension, diabetes, or heart failure).
For each cohort, comorbid diagnoses, demographic characteristics (age and sex), source of patient admission, and specific procedures (e.g., percutaneous coronary intervention in heart bypass surgery) are classified as potential risk factors. Healthgrades uses logistic regression to determine which of these potential risk factors are statistically significant in predicting the outcome measure (e.g., mortality). All risk factors that remain in the final model are statistically significant in predicting the clinical outcome (alpha = .05).
Risk factors with an odds ratio less than 1.0 are removed from the final model. There are exceptions to this rule, as determined by statistical and clinical expert review. For example, risk factors that are part of the cohort definition (inclusion codes) remain in the model even if the odds ratio is less than 1.0 (e.g., laparoscopic cholecystectomy in gallbladder surgery).
Appendix B lists the coefficient summary table for each of the 17 mortality-based cohorts. Appendix C lists the coefficient summary table for each of the 16 complication-based cohorts. Included in the summary tables are the risk factor descriptions, the model coefficients, standard errors, z-statistics, and odds ratios.
(See Appendix B and Appendix C for coefficient summary tables for 33 logistic regression models. For each model, tables include the following items:
- Model (factor) Coefficient – This represents the increase or decrease to the patient level log odds when the patient has the associated factor.
- Standard Error – This is a measure of variation for the coefficient.
- Z-Statistic – This is a test statistic which provides a measure of the strength of the relationship between the factor and the outcome.
- Odds Ratio – This is the most commonly interpreted component of a logistic regression model. This indicates the relative increase in the likelihood of a negative outcome (mortality or complication) when a patient has the risk factor relative to a patient who does not.
Appendix D, Model Fit Statistics contains the c-stat with a 95% confidence interval for each risk-adjustment model by cohort. The statistical models were checked for predictive ability and finalized. All of the models were predictive of the outcome being measured, with c-statistics ranging from .657 to .949. These cohort-specific and outcome-specific models were then used to estimate the probability of the outcome for each patient in the cohort (predicted probability of mortality or complications).
POA fill rates are included as an additional independent variable for each year of data. The POA fill rate is calculated as the percent of diagnosis codes having a known POA indicator value of Yes, No or Clinically Undetermined among patients in that cohort at that hospital.
This additional risk adjustment is necessary because critical access hospitals are exempt from POA reporting. Thus, these hospitals have lower rates of detectable in-hospital complications and conversely, more conditions potentially factored into the acuity adjustment. Testing verified that the inclusion of POA fill rates in the logistic regression model adequately adjusts for the differing overall rates of complications between hospitals reporting POA and the exempt hospitals.
Once the regression models were developed, Healthgrades stratified hospital performance for each of the 33 conditions or procedures into three categories:
★★★★★ Better Than Expected – Actual performance was better than predicted and the difference was statistically significant at alpha = 0.1.
★★★ As Expected – Actual performance was not statistically significantly different from what was predicted at alpha = 0.1.
★ Worse Than Expected – Actual performance was worse than predicted and the difference was statistically significant at alpha = 0.1.
Developing the Healthgrades hospital performance categories involves four steps:
- The hospital predicted value (predicted number of mortalities or complications at each hospital) is calculated by summing the individual patient record predicted values determined from the logistic regression models discussed above.
- The hospital predicted value is compared with the actual or observed value (the actual number of patient mortalities or patients with complications at each hospital).
- A test is conducted to determine whether the difference between the predicted and actual values is statistically significant. This test is performed to make sure that differences are unlikely to be caused by chance alone. A z-score is used to establish an approximate 90% confidence interval.**
- Hospital performance star-levels are determined based upon the outcome of the test for statistical significance.
The performance category results of five-, three-, and one-star ratings are designed to create a transparent, easy-to-understand measure that consumers can readily use in evaluating their options for where to receive care for their specific health needs.
** In stratifying hospital performance categories, Healthgrades establishes a 90% confidence interval for z-score distribution with cut-offs of 1.645 standard deviations above and below the mean. The five-star hospital performance category for a specific condition or procedure includes scores that are greater than 1.645 standard deviations above the mean. The one-star hospital performance category for a specific condition or procedure includes scores that are less than 1.645 standard deviations below the mean. Additionally, the Hosmer-Lemeshow patient level variance estimates are used to calculate hospital specific standard deviations for test of statistical significance.
While logistic regression models are valuable in identifying hospitals that perform better than others for care provided during a hospital stay, there are some limitations to these models. The models are limited by the following factors:
- Cases may have been coded incorrectly or incompletely by the hospital.
- The models can only account for risk factors that are documented and recorded into the medical claims record. If a particular risk factor is not coded into the billing data, (such as a patient’s socioeconomic status or health behavior) then it is not accounted for within these models.
Please note that hospitals grouped into the five-star category for performance in a specific cohort is not a recommendation or endorsement by Healthgrades for any hospital; it simply means that the data associated with a particular hospital has met the foregoing qualifications. Individual patients should work with their doctor to decide if a particular hospital is suited for their unique needs.
Also note that if more than one hospital reported to CMS under a single Medicare Provider ID, Healthgrades analyzed patient outcomes data for those hospitals as a single unit. Throughout this document, therefore, “hospital” refers to one hospital or a group of hospitals reporting under a single provider ID. Some hospitals that operate as a single unit submit data under multiple provider IDs. In this case, the data from the multiple provider IDs are combined and reported under the parent hospital provider ID.