The Little Flaw in the Longevity-Gene Study That Could Be a Big Problem

Remember that Science study from last week linking a whole bunch of genes—including unexpectedly powerful ones—to extreme old age in centenarians? NEWSWEEK reported that a number of outside experts thought it sounded too good to be true, perhaps because of an error in the way the genes were identified that could cause false-positive results. Since last Thurday, they've been trying to figure out what might be lurking in the data, and now there's a suspect: a DNA chip called the 610-Quad, which is used to identify and sequence the chemical letters of DNA, and which has an apparent tendency to get some small but critical details wrong. The flaw with the chip and the way it was used could cast serious doubt on the study's strongest results, suggesting that they stem from a lab mishap rather than a real link to long life.

The flaw in question could be easily addressed with a little follow-up research. In very simplified terms, all that's needed is for someone to rerun the analysis using a single different DNA chip. But this should have been done already, before publication. The fact that it wasn't raises the question of how a paper with a missing piece like this got approved and published by Science.

The paper—which identified 150 genetic variants that might increase a person's chances of making it to age 100, apparently by protecting the body against disease—had as one of its two principal components a type of research called a genome-wide association study (GWAS). These surveys, the bread and butter of modern genomics, use chips to analyze large amounts of DNA in many people, looking for variants that are more common in "cases" (here, that means centenarians) than "controls" (regular people). The variants that turn up more often in "cases" are the ones linked to the trait the scientists are curious about. The studies are usually very thoughtfully designed and reliable. What happened with this one?

The first thing to know is that not all gene-identifying chips are created equal. They occasionally identify letters of DNA incorrectly, and—to complicate things further—each type of chip makes different errors at different points in the genome. That phenomenon can lead to false-positive results if it's not well-controlled by experimental design, says David Goldstein, the Duke University geneticist who first raised this issue here last week. "Unfortunately, different chips have their own little problems for specific [genetic variants]," he says. The key to keeping false positives at bay is to ensure that cases and control groups are analyzed using exactly the same techniques. If you use one type of chip to analyze your cases and a different type to analyze your control group, "you can see any [variants] that are genotyped differently on the different chips 'lighting up' as apparently associated with the trait," says Goldstein, when in fact that pattern is just an experimental artifact.

All of the chips used in the Science study came from the same manufacturer, Illumina, but they weren't identical. According to a brief description in the paper, the researchers used two different chips to look at their centenarians, analyzing most people with a 370 chip that examines 370,000 genetic variants and a smaller fraction of people with a different chip (the 610-Quad) that examines 610,000 variants. The reason, says Paola Sebastiani, the Boston University biostatistician who led the study, is that at one point the 370,000-variant chip went off the market and the 610-Quad "was the best option for us in terms of costs and coverage." The controls involved an even more varied assortment of technology—some were analyzed with the 370, some with the 610, and some with two other types of chips.

Kári Stefánsson, the Icelandic geneticist who founded deCode Genetics, knows something about the 610-Quad—his company has used it too. He says it has a strange and relevant quirk regarding two of the strongest variants linked to aging in the BU study, called rs1036819 and rs1455311. For any given gene, a person will have two "alleles," or forms of DNA. In the vast majority of people, at the rs1036819 and rs1455311 locations in the genome, these pairs of alleles consist of one "minor" form and one "major" form. But the 610-Quad chip tends to see the wrong thing at those particular locations. It always identifies the "minor" form but not the "major" form, says Stefánsson—even if the latter really is present in the DNA, which it usually is. If you use the error-prone chip in more of your case group than your control group—as the BU researchers did—you're going to see more errors in those cases. And because what you're searching for is unusual patterns in your cases, you could very well mistake all those errors (i.e., false positives) for a genetic link that doesn't actually exist.

Stefánsson says he is "convinced that the reported association between exceptional longevity and most of the 33" variants found in the Science study, including all the variants that other scientists hadn't already found, "is due to genotyping problems." He has one more piece of evidence. Given what he knows about the 610-Quad, he says he can reverse-engineer the math in the BU study and estimate what fraction of the centenarians were analyzed with that chip. His estimate is about 8 percent. The actual fraction, which wasn't initially provided in the Science paper, is 10 percent, the BU researchers tell NEWSWEEK. That's close, given that Stefánsson's calculations look at just two of the variants found in the study and there may be similar problems with others.

One of the oddest things about this potential error is how much it stands out in an otherwise carefully designed study. The BU researchers made a serious attempt to deal with confounding factors—a challenge given that centenarians are by definition different from any possible control group, because they were born earlier—and, Sebastiani says, the team "conducted extensive quality-control procedures and cleaning of the data."

What the group apparently didn't do, however, is obtain a third-party analysis of their centenarians' and controls' DNA using a single chip for everyone. There's "nothing in the world simpler to do," says Goldstein. "We would do this for any 'discovery' we had in this kind of a situation, but when the results themselves are a bit improbable, as the results are here with the exceptional genetic control, then there is all the more necessity for that quality-control step." Goldstein adds that such a step is standard practice for most GWAS research. That's why you can trust many other GWAS papers while withholding judgment on this one. Yes, it's tempting to look at this study and wonder what other flaws may be hiding in other GWAS papers, even those in top-flight publications. But this episode shouldn't be read as evidence that genome-wide association studies are untrustworthy as a rule, because the rigor that seems to be missing from this study is almost always found in others that haven't yielded such dramatic results.

It's possible that when that replication study is done, the genetic associations in the longevity study will hold up. (At least a few of them, such as the link between long life and APOE—which is also linked to Alzheimer's—surely ought to, since they have been found in other studies.) The BU paper's critics aren't out-and-out saying it's wrong. They're just saying it could be.

Still, one has to wonder how the paper wound up in Science, which, along with Nature, is the top basic-science journal in the world. Most laypeople would never catch a possible technical glitch like this—who reads the methods sections of papers this complicated, much less the supplemental material, where a lot of the clues to this mystery were?—but Science's reviewers should have. It's clear that the journal—which hasn't yet responded to the concerns raised here—was excited to publish the paper, because it held a press conference last week and sent a representative to say as much.

The BU scientists are holding a public Web chat today at 1 p.m. ET. Most of the questions they take probably won't concern highly technical stuff like this. Sebastiani would prefer that critics' questions be addressed directly to her in journals rather than, say, relayed to her by NEWSWEEK writers: "So far we were not approached by any of these investigators directly, only by reporters," she says, which is "rather surprising and disappointing to me." The Web conference, Sebastiani adds, is being held primarily to address the fact that several companies are already thinking about selling a test based on the Science paper, a notion that the study's authors abhor and are trying to prevent. "We strongly feel that results of such a test should continue to be for research use and that it is not at all ready for use in the public domain," says Sebastiani. "There are just too many opportunities for misuse and misinterpretation at this very early point." Not at all ready for use in the public domain: that's the one thing that everyone involved with this paper does seem to agree on.

One more important thing: the BU researchers put together a model for predicting whether a given person would live to 100 or not, and it was widely reported that the model had 77 percent accuracy. That was true in the study, where the researchers were applying the model to people from groups of roughly equal size—they had about the same number of centenarians as they did controls. In reality, however, only 1 in every 6,000 people lives past 100, so the real-life sample sizes, if you will, are very different. Both Stefánsson and David Altshuler, a geneticist who leads GWAS research at the Broad Institute, say that fact renders the model much less useful than you might think, because it actually tells you only that your chance of living to 100 is either really small (much less than 1 percent) or really, really small (even less than that). "For most practical purposes," Stefánsson notes, this "makes no difference for an individual." It's a good reason not to rush out and get your longevity genes tested, although at this point, you don't need another one.

UPDATE: Within an hour of this story's publication, the Science study's authors released a statement which a BU spokeswoman described as appearing "because of your inquiry and a similar one from the New York Times concerning methodology used to test 2 of the 150 genetic variants." Here is what the statement says: "Since the publication of our study in Science, which was extensively peer-reviewed, a question has been raised about two elements of the findings. One has to do with two of the 150 genetic variants included in the prediction model, while the other is related to the criteria used to determine the significance of the individual variants. On the first concern, we have been made aware that there is a technical error in the lab test used on approximately 10% of the centenarian sample that involved the two of the 150 variants. Our preliminary analysis of this issue suggests that the apparent error would not effect the overall accuracy of the model. Because the issue has been raised since the publication of the paper, we are now closely re-examining the analysis. Another question that was raised concerns the criteria used to determine if an association between a genetic variant and exceptional longevity was statistically significant. We used standard criteria for the analysis, and we are confident that the appropriate threshold was used."