Sometimes you can tell from the very first sentence that a news article is going to have problems. Case in point: a recent New York Times cover story on the Dartmouth Atlas of Health Care, an influential project that reports huge differences in health care costs and treatments around the United States. The article began like this:
In selling the health care overhaul to Congress, the Obama administration cited a once obscure research group at Dartmouth College to claim that it could not only cut billions in wasteful health care spending but make people healthier by doing so.
The tip off is “Once obscure research group.” Technically it’s not totally wrong- the Dartmouth Atlas group was obscure when it formed in 1988, in the same way that the New York Times was a “once obscure” newspaper when it first published back in 1851. But the phrase “once obscure” implies that the Obama administration went way out in left field to find research that it liked. The truth is that you’d be hard pressed to find health policy research more mainstream than the Dartmouth Atlas.
For the past 20 years it’s been a key source of data for anyone trying to figure out how to control health care costs. The group’s work has been cited so often that back in 1997, its co-founder John Wennberg was named “Most Influential Policy Maker of the Past 25 Years” by the policy journal Health Affairs.
It’s a minor complaint, but the phrase “once obscure” starts the article with a questionable tone, and it’s downhill from there. The authors of the article, Reed Abelson and Gardiner Harris, level two major charges against the Dartmouth Atlas:
Criticism #1: The “mistaken belief that the Atlas proves that cheaper care is better care.”
The authors refer to a color-coded map produced by the Dartmouth Atlas researchers showing Medicare costs by county across the United States:
The light blue areas are counties with low per capita Medicare costs and navy blue areas have high Medicare costs. Abelson and Harris write:
For all anyone knows, patients could be dying in far greater numbers in hospitals in the light blue regions than hospitals in the dark blue ones, and Dartmouth’s maps would not pick up that difference. As any shopper knows, cheaper does not always mean better.
All right, so for one, some research has been done to find out whether patients are dying in “far greater numbers” in low cost areas, and… they’re not. A few paragraphs later, the Times article even says “a 2003 study found that patients who lived in places most expensive for the Medicare program received no better care than those who lived in cheaper areas.”
But, the authors say, other studies have suggested that more expensive care is better care. Jonathan Skinner, an economics professor and co-author of the Dartmouth Atlas addressed this criticism in a blog post for the New York Times titled, Is More Care Better Care? Incredibly, his post was published on June 13, 2009, nearly a year before this article appeared. Here’s Professor Skinner:
Critics also cite the one study showing that more state-level spending is associated with better care. This finding raises the concern that if health-care spending were cut in high-cost regions, quality would suffer. But several other studies (like this one and this one) have found negative associations: regions or hospitals spending more actually exhibit slightly worse outcomes.
But we have argued elsewhere that this debate is a red herring. No matter which data you look at, spending and quality are barely correlated. Per capita spending ranges across the country by a factor of nearly three, and quality measures show little relationship to these large spending gaps. Nor do any of these studies tell us whether higher spending “causes” lower (or higher) quality. The good news is that high-quality, low-cost providers do exist — like the Geisenger Clinic or Intermountain Healthcare — and can provide a model for financially sustainable health care.
Also, in the Times article, the line that really bothered us was, “As any shopper knows, cheaper does not always mean better.” On the surface it sounds like common sense– in other words you get what you pay for. But in health care, higher quality means lower total costs (which is what the Dartmouth Atlas is looking at). Health care journalist Merrill Goozner explains:
Take a few additional steps to keep operating rooms germ free and rates of hospital-acquired infections and their attendant higher costs plummet. Do a knee implant right the first time and you don’t have a patient back within a year for a revision.
In other words, “as any shopper knows,” things that are well-made don’t break or need to be replaced as often, saving you money in the long run.
Criticism #2: The Dartmouth Atlas doesn’t take into account the cost of living
From the NY Times article:
Even Dartmouth’s claims about which hospitals and regions are cheapest may be suspect. The principal argument behind Dartmouth’s research is that doctors in the Upper Midwest offer consistently better and cheaper care than their counterparts in the South and in big cities, and if Southern and urban doctors would be less greedy and act more like ones in Minnesota, the country would be both healthier and wealthier.
But the real difference in costs between, say, Houston and Bismarck, N.D., may result less from how doctors work than from how patients live. Houstonians may simply be sicker and poorer than their Bismarck counterparts. Also, nurses in Houston tend to be paid more than those in North Dakota because the cost of living is higher in Houston.
I’ll let Columbia statistics professor Andrew Gelman address this one:
Huh? One of the striking things about the cost-of-care map is how little it corresponds with cost of living. The high-cost regions include most of Texas, just about all of Louisiana, Mississippi, Arkansas, Oklahoma, and Tennessee (as well as some more expensive places such as the Bosnywash area and much of California.) There may be a lot of problems with this study, but I can’t imagine that one of these problems is a lack of accounting for the (relatively) high cost of living in Houston.
Instead of comparing Bismarck to Houston, the authors could have looked at areas with similar costs of living and still found huge differences in Medicare spending. For example, in a much better article examining regional differences in spending, Atul Gawande of the New Yorker compares McCallen, TX, to El Paso, TX. The two cities are nearly identical in terms of demographics, overall health of the population, and cost of living, but McAllen still has much higher Medicare costs per capita. You can see similar discrepancies all over the map.
The NY Times article missed an opportunity
The most frustrating thing about the New York Times article is that despite all their criticism of the Dartmouth Atlas, the authors miss it’s biggest actual shortcomings. Hank Aaron at the Brookings Institute says:
The Dartmouth research, showing huge variation in the use of various medical procedures and large variations in per patient spending under Medicare, has been a revelation and a useful one. There is no way to explain such variation on medical grounds and it is problematic. But readers, including my former colleague Orszag, have taken an over-simplistic view of what the numbers mean and what to do about them.
For example, he points out that knowing that high spending doesn’t lead to high quality isn’t the same as knowing how to reduce spending without sacrificing quality. We can always find wasteful medical spending after the fact, but in order to reduce health care spending in a way that improves quality, we need to know what procedures will be wasteful before doctors start treating a patient. However:
There are some egregious situations where [beforehand] waste can be demonstrated. But for the most part, we just do not have good enough research to predict with sufficient accuracy who will benefit and who will not from doing more of some procedure. That will take a lot of research. It hasn’t yet been done.
In other words, it’s not that the Dartmouth Atlas data is wrong, it’s that we don’t know how to use the data. The real question is whether we can figure out how to separate the good care from the bad. There’s a lot of funding in the health care bill for this type of research. But we won’t know if it’s successful for years to come.
