Objectives: Increased demand for quality health care has led to lay-press ranking systems, such as the ranking from US News and World Report (US News). Their “Best Hospitals” publication advertises itself as the go-to resource for patients seeking care in a number of specialty areas. We sought to test the relationship between US News rankings and transplant outcomes.
Materials and Methods: Using data from 2014 to 2018, we compared outcomes from the Scientific Registry of Transplant Recipients database for liver and kidney transplants against US News-ranked centers using the categories “Nephrology” and “GI Surgery and Gastroenterology” as substitutes, as US News does not rank transplant centers specifically. P < .05 was set as significant.
Results: Using hazard ratio data, we found that kidney transplant center rank had only a small impact on postoperative outcomes in terms of patient survival (hazard ratio = 0.996, P = .049) but had no impact on graft survival (hazard ratio = 0.997, P = .077). In addition, liver transplant center rank had no impact on liver graft survival (hazard ratio = 1.003, P = .304). The impact of hospital ranking on survival was minimal compared with other variables.
Conclusions: The US News rankings for “Nephrology” and “GI Surgery and Gastroenterology” have minimal values as a measure of liver and kidney transplant outcomes, highlighting that these lay press rankings are not useful to the unique transplant patient population and that providers should help guide patients to transplant-specific resources.
Key words : Kidney transplantation, Liver transplantation, Outcomes, Quality of health care
Increased national attention toward the cost and quality of health care, as well as the advent of value-based purchasing by the Centers for Medicare and Medicaid Services (CMS), has led to an increase in the availability of hospital rankings and outcomes data. The US News and World Report (US News) is one of the most recognized and publicly available sources for consumer hospital ratings and has been noted to be more responsive to changes than other public ranking platforms.1-4 Testing these relationships reveals mixed results, although they generally favor a positive correlation between quality and rank.1,4-9 Despite this, examination of the various nationally published hierarchies has demonstrated little agreement pertaining to which institutions are “best.” Each agency measures centers on differing criteria, a fact that has gained the national spotlight with a popular New York Times article.5,8,10,11
For some specialty services, such as liver and kidney transplantation, the lay literature may not correlate with measured outcomes. Although there is no direct source for transplant center rankings, US News is commonly used by physicians to guide patients. The field of transplantation, with its long-recognized focus on ethical and efficient use of precious and limited resources, is ripe for review.
Quality measurements for reimbursement from the CMS are flawed. Jackson and colleagues revealed that regional referral centers are unfairly biased due to nonmedical-related factors, such as travel distance.12 This is of particular importance in transplantation, where a majority of care is handled by relatively few specialty centers.12 Previous reports have evaluated the relationship between US News rankings and patient outcomes for other areas of complex surgery and revealed strong correlations. However, none specifically addressed liver and kidney transplantation, and no press ranking systems have established correlations in transplant outcomes data.1
Materials and Methods
The Scientific Registry of Transplant Recipients (SRTR) database was queried to identify kidney and liver transplant outcomes and hazard ratios (HR) for patient and graft survival at 1 year, grouped into rolling 2.5-year cohorts. This time period best matches temporally with the US News 3-year data evaluation period. In addition, 1-year survival is a common outcome measure evaluated by 2 major oversight agencies.13 The annual US News “Best Hospitals” issues were obtained from the publisher. Due to the nature of this analysis, because we only collected secondary data without identifiers, there was no need for informed consent. These data were merged using transplant center codes. Data covered transplant information between 2014 and 2018.
The US News rankings are determined by several factors. The hospitals must meet one of the following criteria to be included: it is a teaching hospital, it is affiliated with a medical school, it has at least 200 beds, or it has at least 100 beds in addition to offering at least 4 medical technologies deemed significant by US News. If a hospital has a specialty ranked, additional criteria are required. Hospitals must meet a volume or discharge threshold that varies by specialty. The underlying methodology of these rankings involves a composite score based on structure, process, and outcomes. Structure refers to hospital resources related directly to patient care. Examples include intensity of nursing staff and availability of certain technologies and services. Process is the way in which a hospital delivers care, treatment, and education. The most obvious measure from the outcomes data is death, which is measured by risk-adjusted mortality. This accounts for the complexity of the individual and their current condition. As part of its ratings, US News also considers patient survival and safety for a 3-year period ending 2 years prior to each publication. For this reason, US News rankings from 2014 to 2018 were paired with SRTR data from June 2008 to December 2015, so that each agency was evaluating 1-year survival averaged over the same 3-year period. From these data, our outcome variables were patient and graft survival after kidney and liver transplantation.
US News only provides numerical rankings for the top 50 programs. US News Hospital rankings for “GI Surgery and Gastroenterology” and “Nephrology” were substituted for liver and kidney transplant programs, respectively, as US News does not specifically score or rank hospitals for transplantation.
The basic patient, donor, and transplant characteristics were compared across high-, medium-, and low-ranking transplant centers using t or Wilcoxon-Mann-Whitney tests and chi-square or Fisher exact tests, depending on the sample size and the distribution of the variables used. Patient and graft survival rates (1- and 2-years posttransplant) were analyzed. Survival curves and the estimates for transplant outcomes were obtained using the Kaplan-Meier product limit method. A series of log-rank tests were performed to evaluate whether the graft survival rate and the patient mortality rate varied depending on the transplant center’s ranking. In the survival analysis of transplant outcomes, patient death and graft failure were the endpoints, thereby generating death-censored survival estimates. Death data are measured by risk-adjusted mortality. This accounts for the complexity of the individual and their current condition. To control for patient- and donor-level risk factors, we performed multivariate Cox regressionto determine statistically significant associations between rankings of transplant center and graft survival as well as patient mortality for both kidney and liver transplant. Statistical significance was defined as P < .05 in the analysis unless noted otherwise. Stata version 15 (StataCorp LLC) was used for the analysis.
Kidney transplant outcomes according to nephrology rankings
Rankings of transplant centers were coded, with higher numbers indicating a better position in the ranking (ie, the rank of 50 is the top-ranked program). We grouped the transplant centers into 3 cohorts: high rank (>35), medium rank (35 to 20), and low rank (<20). Table 1 compares the demographic, clinical, and donor-level factors across the 3 groups for the kidney transplant cohort.
Kidney transplant: patient survival
Figure 1 demonstrates that there was no clear relationship between hospital ranking and patient survival. This did not control for baseline differences between hospital demographics. We estimated the Spearman correlation coefficient between ranking and most recent patient status (based on composite death date) (n = 34 946, Spearman rho = -0.015). The Spearman correlation coefficients for ranking and patient death status were negative and significant, indicating that higher-ranking transplant centers have lower probability of patient death at the time of the observation. However, this correlation did not account for other demographic, clinical, and donor-level factors that may determine the transplant outcomes.
Table 2 presents the Cox regression results, where the dependent variable was risk of patient mortality (based on composite death date) and the primary independent variable was the ranking of transplant centers. Our results indicated that higher-ranking transplant centers were associated with lower risk of patient mortality, which was shown to be a significant association (HR = 0.996, P = .49). This was further illustrated in Figure 2, which shows the impact that each factor had on patient mortality risk. As shown in Figure 2, the higher ranking of a transplant center was negatively associated with patient mortality, although the impact was small (<0.01% reduction in risk as the ranking increased by 1). When all other variables were kept constant, older age, male sex, White ethnicity, Black ethnicity, Hispanic ethnicity, prior transplant, being on dialysis, higher HLA mismatch, Kidney Donor Profile Index (KDPI), and having higher body mass index (BMI) were all risk factors for patient mortality, whereas having a local and regional donor reduced the risk of patient mortality compared with having a national donor. Figure 3 presents the Kaplan-Meier patient survival estimates; as shown, a significantly higher number of patients had better survival in the high- and medium-ranking centers compared with the low-ranking centers.
Kidney transplant: graft survival
We estimated the Spearman correlation coefficient between ranking and kidney graft status (n = 34 946, Spearman rho = -0.024). The Spearman correlation coefficients for ranking and kidney graft status were negative and significant, indicating that higher-ranking transplant centers had lower probability of graft failures. However, this correlation did not account for other demographic, clinical, and donor-level factors that may determine transplant outcomes. Similar to Figure 1, there was no clear relationship demonstrated between ranking and graft survival time.
As shown in Table 3 and Figure 4, ranking of the transplant center had no significant effect on graft survival status (HR = 0.997, P = .077). Similar to patient survival results, having a local or regional donor increased graft survival. As shown in the Kaplan-Meier results in Figure 5, higher survival was shown in high-ranking centers compared with medium- and low-ranking centers.
Liver transplant according to gastrointestinal rankings
In the ranking of liver transplant centers, those coded with higher numbers indicated a better position in the ranking (ie, rank of 50 is the top-ranked program), similar to the analyses of kidney transplant centers. We again grouped the transplant centers into 3 cohorts: high rank (>35), medium rank (35 to 20), and low rank (<20). Table 4 compares the demographic, clinical, and donor-level factors across the 3 groups for the liver transplant cohort.
Again, no clear relationship was shown between hospital ranking and graft survival. This did not control for baseline differences between hospital demographics. We estimated the Spearman correlation coefficient between ranking and liver graft status (n = 14 657, Spearman rho = -0.001). The Spearman correlation coefficient for ranking and liver graft status was negative and significant, indicating that higher-ranking transplant centers had lower probability of graft failure. However, this correlation did not account for other demographic, clinical, or donor-level factors that may determine transplant outcomes.
As shown in Table 5, there was no significant effect of hospital ranking on liver graft survival (HR = 1.003, P = .304) at 1 year. Age, time on wait list, retransplantation, being on dialysis before transplant, donor age, and donor sex had significant impacts on graft survival at 1 year. These impacts are further demonstrated in Figure 6, which visually interprets the magnitude of the results from Table 5. As shown in the Kaplan-Meier liver graft survival results in Figure 7, medium-ranking centers had better survival compared with low- and high-ranking centers. When we conducted the same analyses with liver patient outcomes, the results were the same; therefore, we decided to show only graft survival for the sake of clarity and to reduce redundancy.
The relationship between lay press ratings and objective clinical outcomes is complex. When we considered both liver and kidney transplant outcomes after analyzing hazard ratios, a better US News ranking was only slightly correlated with decreased survival of kidney transplant patients, but not with survival of kidney grafts. Similar analyses showed no correlations with liver graft survival or with survival of liver transplant recipients. For this reason, US News rank appears to be a poor tool for a consumer concerned about duration of life and transplanted organs.
It is important that this information be explicitly conveyed to patients, as it has been shown to greatly impact their decision-making process.16 The methodology of US News, which evaluates and scores many aspects of care besides outcomes, may remain a useful tool for consumers who are more concerned about other areas of their care, including the hospital environment, support services, and the prestige of their transplant center.3,8,17 Superior patient and graft survival alone, compared with national average survival rates, would make for excellent hospital press and appear convincing to the public. Health professionals, however, understand that population characteristics play a critical role in outcomes.3,8,18,19 This is even more significant among transplant populations, as their unique risk factors are a huge determinant in organ eligibility and transplant center availability.
The US News and World Report states that the purpose of their annual rankings is to help consumers make a choice about where to find “especially skilled inpatient care” when faced with operations or care that poses “unusual technical challenges or a significantly increased risk.”20 The increasing digitalization of ranking information has given individuals easier access to data.21 Although there are concerns about unintended negative consequences from hospital report cards, they appear at face value to provide transparency and public accountability.3,4,9,18,22,23 How one chooses a transplant center is not perfectly understood and likely differs to some extent from person to person.16,24 Consumers, when presented with quality information in an easy-to-understand format, such as rankings, will consider quality along with cost as major decision-making factors, including choice of surgeon and center.16,25-28 However, the transplant population is not as straightforward as the general elective surgical patient population, as their choices are not simply their choice but depend on organ and transplant center availability. In a study that analyzed how patients choose a kidney transplant center, one of the major themes patients used to make decisions was their perceived reputation of the center, along with reliance on opinions from providers, family, and friends, valued relationships, convenience, and insurance coverage.16 The choice of center may be made entirely by the referring physician with little input from the patient, although the referring physician may consult rankings before making a recommendation.4,24,29 According to one survey given to those choosing a transplant center, the wait list was given the highest priority.30 The wait list was almost twice as likely as outcomes data to be the most important factor.30 The survey also found that many relied on information from physicians to help with their decision, whereas a minority cited transplant-specific organizations.30 Data have also indicated that, while both patients and providers often state that they would like to make the decision based primarily on quality, few actually do.17
Outcomes of transplant centers are more dependent on their referral pool than other factors.31 In kidney transplantation, a high graft failure rate is associated with a decline in wait-list registrations, although it is unclear whether individual choice, education level, referring physician choice, or insurance companies drive this decision.32,33 Hospitals clearly believe that the pool can be influenced by attracting customers with the use of high US News rankings, as evidenced by their use in advertising.8,18,34 To make matters more compelling, both transplant outcomes and “patient perceptions of care” are tied to CMS reimbursement and public funding.18,19,35 Hospitals have a strong financial incentive to recruit the healthy and wealthy, influence their referral pools, and work to increase their perceived reputation. There is a growing concern within the field of transplantation that increased scrutiny may cause centers to avoid high risks and unnecessarily discard organs.3,13,19 If a definitive ranking system were to be created, this could potentially have disastrous negative consequences that could lead to transplant centers rejecting more organs and limiting access in an already limited field.
Our analysis indicated that more research is needed concerning public hospital rankings and how patients are able to interpret these and other outcome databases such as the SRTR. Ratings agencies consider many factors in ranking a hospital beyond treatment outcomes. These include, but are not limited to, reputation, clinical and basic science research, the availability of ancillary services, overall hospital safety measurements, and services designed to increase patient comfort, which have variable levels of impact on the treatment and care actually provided.
One limitation of our study was the lack of US News hospital categories specific to kidney and liver transplantation. This weakness demonstrates the need for dedicated transplant resources that are easy for patients to understand. Our use of “GI Surgery and Gastroenterology” and “Nephrology” subspecialties from the US News report was the best substitute for actual transplant rankings but assumes consumers will arrive at a similar conclusion. We feel this is valid for 2 reasons. First, most referrals for transplantation, almost all pretransplant care, and a large amount of posttransplant care are provided by hepatologists and nephrologists. Second, a plurality of top 50-ranked programs (99.33% for kidney and 78.28% for liver) are transplant centers. However, for a center to be analyzed by US News, it needs to meet specific criteria, including being a teaching hospital, being affiliated with a medical school, having at least 200 beds, or having at least 100 beds in addition to offering at least 4 medical technologies deemed significant by US News. Because not every transplant center meets these requirements, some may have been missed by our analyses, which further demonstrates the need to produce transplant-specific resources for patients. An additional weakness of this study was the recent changes to allocation for liver and kidney transplantation, which may not have been captured in this analysis; these may, in the future, alter outcomes and rankings.
The goal of the data provided by SRTR is to help consumers gain a broad understanding of a transplant program’s general outcomes and trends, which is key for the transplant patient population. It is also useful for comparisons of different programs. The SRTR provides timely and accurate information from programs for various types of transplants. However, the metrics presented in SRTR are calculated for all patients at the program that is searched for and may not represent an individual’s particular need. Each transplant patient has a unique situation and history that needs to be considered holistically to find the “best” fit for them, but this does not lend itself to a simple ranking system.
A strength of this study is that the SRTR 1-year evaluation periods aligned perfectly with the US News evaluation periods. Although longer-term data, for instance 3-year or 5-year survival rates, would be interesting to investigate, it would not have aligned temporally with the US News evaluation period. In addition, the SRTR 1-year survival data are used by the CMS and the Organ Procurement and Transplantation Network Membership and Professional Standards Committee for regulatory review of transplant programs.13 Another strength is the use of hazard ratios, which account for patient mix and are rigorously validated by the SRTR.14,15
The field of transplantation treats a population that is particularly desperate and for whom care is exceedingly complex. Practitioners in the field are responsible for the ethical distribution of a precious and limited resource, donated organs. Transplant center survival results are tied to outcomes and federal funding. It is critical for the public to have an accurate assessment of the quality of their transplant center while also understanding how their unique situation fits with each center. The US News advertises its rankings as the go-to resource for advice on seeking specialized care. However, because we showed these to be poor representations of objective transplant outcomes, such as graft survival, it may be most prudent for providers to guide transplant patients away from simple ranking data and to more detailed resources such as the SRTR, which can be evaluated with the patient and provider together to determine what factors are most critical to each patient. The SRTR is an incredibly valuable tool unique to transplant surgery that is very transparent and meticulous in its methods, which is crucial in such a delicate field with constant vigilance for upholding ethical standards of organ use.
The field of transplantation is complex. With its duty to ethically and efficiently use its donated organs and the rigorous standards that are constantly evaluated and reviewed, along with results from our study that current ranking systems are of little to no use to the consumer, we believe it would be prudent for providers to encourage transplant patients to not rely on a “one size fits all” approach to their care. In this age of readily available information, it is easy for consumers to find a “top 50” list without understanding the details of how these conclusions are formed. Because of this, it is more important now than ever before as physicians to help patients process the data they are finding in a meaningful way. This is even more important in transplantation, as the unique situation of each patient must be accounted for and there is no one best center; rather, it is more important to find the best center for each patient, looking at transplant center volume, risk tolerance, and the patient’s risk factors.
Volume : 19
Issue : 10
Pages : 1014 - 1022
DOI : 10.6002/ect.2021.0043
From the 1University of Toledo Medical Center, Toledo, Ohio; the 2Oregon Health and Science University, Portland, Oregon, the 3George Mason University, Fairfax, Virginia; and the 4Albany Medical Center, Albany, New York, USA
Acknowledgements: The data reported by the authors were supplied by the Scientific Registry of Transplant Recipients (SRTR) and the US News and World Report through their hospital rankings. The analysis and interpretation of the data are the responsibility of the authors alone and are not affiliated with the SRTR. The authors thank Dr. Weikai Qu for his contributions to this paper. The authors have not received any funding or grants in support of the presented research or for the preparation of this work and have no declarations of potential conflicts of interest.
Author contributions: Lindsey Loss and Gavin Kelly provided data collection, manuscript writing, and editing; Naoru Koizumi and Abu Bakkar Siddique provided data analysis and interpretation; Julia Shreve and Stephen Markowiak provided manuscript writing; Munier Nazzal provided research study design; and Jorge Ortiz provided research study design, research concept, data analysis and interpretation, manuscript writing, and editing.
Corresponding author: Lindsey Loss, 3181 SW Sam Jackson Park Road Mail code L223 Portland, OR 97239 USA
Phone: +503 494 7758
Figure 1. Scatterplot Between Ranking and Kidney Patient Survival
Table 1. Descriptive Statistics: Kidney Transplant
Table 2. Cox Regression Results: Kidney Transplant Patient Survival
Figure 2. Coefficient Plot of Cox Regression: Kidney Patient Survival
Figure 3. Kaplan-Meier Survival Analysis: Kidney Transplant Patient Survival
Table 3. Cox Regression Results: Kidney Graft Survival
Figure 4. Coefficient Plot of Cox Regression: Kidney Graft Survival
Figure 5. Kaplan-Meier Survival Analysis: Kidney Graft Survival
Table 4. Descriptive Statistics: Liver Transplant
Table 5. Cox Regression Results: Liver Graft Survival
Figure 6. Coefficient Plot of Cox Regression: Liver Graft Survival
Figure 7. Kaplan-Meier Survival Analysis: Liver Graft Survival