Here’s the deal. I love genomics. I love playing with numbers. And I still think that unveiling the design of life itself is the most interesting problem in the world. However. I think we have some issues.
The first problem we have is the Epic Multiple Testing Problem. Ok, so I’m sure everyone’s aware of multiple testing corrections within individual experiments – that’s fairly straight-forward, and we correct for those. However, what happens that we don’t correct for, is that in particularly hot fields, lots of labs are working on the same problem. And by chance, some of them get strong, striking results. And the more extreme your results are, really the more dubious you should be about them, but actually, we trust them more, they get published first, etc. Then when people go to replicate the results, they get weaker or no results, but there’s a bias against publishing negative results, so these get disregarded. And it’s only when the results are so entrenched that they become dogma, that this creates a space for people to go in and specifically disprove them.
So, we have a cycle of: great result – mostly unreported weakening – accepted dogma – disproved conclusions. A cycle that possibly takes decades. This is described by John Ioannidis in his excellent article – Why most published research findings are false. And do check out this New Yorker article on the same topic – this is really something that’s happening widely, and across disciplines. Drugs are magically losing efficacy, a large number of textbook scientific claims are gradually being disproven.
I don’t see this as a tragic demise of science, more as a science run-of-the-mill. I believe in Karl Popper’s model of science as relying on falsifiability, rather than being continually right. We make claims, we test them, only the best survive – that’s ok. It does mean a few things though. First of all, because of the high pressure on scientists to publish positive results, there’s a systematic pressure to create and embrace false positives. Because of the Epic Multiple Testing Problem, even with correct stats, most up to date methods, and no bad intentions from anyone – lots of research findings, particularly in hot fields, will be false positives. I think it gets worse in fields where people think they already know the conclusions (hello obesity research), or where there are social beliefs tangled up in the science (e.g. anything ever about the male and female brain, Simon Baron-Cohen’s thing about autism being the ‘extreme male brain’, whatever. Did you know that girls with autism are significantly less likely to be diagnosed, even when displaying the same severity of symptoms as the boys who are diagnosed? Not helpful. Anyway). Not that everything that emerges from those fields will be false, but I would guess that it significantly increases their rate of false positives.
So, this is a problem across the entirety of science as a field. What makes genomics potentially worse than average? Well, as I briefly discussed in my last post, there’s a flood of data, and as a field, we’re really quite unprepared and incompetent with it. I don’t mean that individual people are necessarily bad, but biology isn’t traditionally a particularly numerical field (despite some wonderful exceptions to this generalisation), and I think most undergrad curriculums still reflect this. Mine included no programming, and next to no maths, except what I set out to learn on my own initiative. However, lots of people later end up in positions where they need to make use of high-throughput data – something you can’t meaningfully do without at least a basic understanding of stats. But people need to publish to survive, so what you end up with are widely misapplied statistics and hugely variable quality of bioinformatics methods. And, you know, the times when people accidentally flipped 1’s and 0’s and reversed the outcome of their experiment, shifted everything by just one row, accidentally used the same data twice, or in the wrong genome release, or any number of things that can go wrong. I expect this to change as training catches up with the new requirements biologists are facing, but for now, even from this alone, I’d expect we’re one of the fields with the highest rate of false positive findings in science today.
However, even ignoring the incompetence issues, I think there might be another problem. Even if you know what you’re doing, I think the journey of relating the data back to the biology is a precarious one. It’s possible that the more complicated models are actually overfitting the data – while the numbers are fun to play with, what we’re actually dealing with tends to be very noisy data, so any conclusions beyond the most broad and basic ones are in danger of being just patterns that we see in noise. Even with the best tools available to us.
So I don’t believe in genomics. I believe that most of what we publish today will be proven false down the line. I believe that the pressure to publish exacerbates the problem, and that because of the nature of our field, we’re particularly vulnerable. It’s terrifying, given how much of an impact genomics is already having and will continue to have on medicine. It’s also a strong reason to set out to do it better.
To make an obvious wishful thinking point – the focus of any performance assessments should be on the quality of research, rather than the speed and number of publications. Further to that, an increased ability to publish negative results would reduce some of the biases present. But working within the existing system, how can we do better science?
Well, for a start, I salute everybody doing systematic meta-studies. Given the problems outlined, I’m not particularly sure any one study really proves anything, but after a number of years and a number of studies, you can start getting an impression of what the conclusions are. Meta-studies are an excellent way of systematically incorporating the existing knowledge, and particularly if they also focus on the quality as well as the conclusions of the individual studies included – meta-studies are a really powerful tool. Sometimes just really powerful at telling us that our evidence is contradictory and we don’t actually know much of anything, but, you know what? That might be the real answer.
As for individual studies… I think it’s important to base your conclusions on multiple strands of evidence, and connect them with simple methods and good solid logic. I think high-throughput data is quite seductive, in that you can generate a lot of it quickly, and of course you can see patterns in it – you’ll see patterns in any large enough set of numbers. So, build up evidence from multiple sources, make sure that your sources of evidence follow best practice recommendations (controls, sufficient replicates, etc.), integrate the data (ooh, buzzword), be aware that noise and false positives are a problem at every stage of the process, and see what you can get out of it. It’s frustrating, and takes years, and you probably won’t get very flashy results out of it. Unfortunately, it’s how solid science is done. As a bioinformatician, I don’t aspire to use the most complicated models, or build the most sophisticated tools. What I aspire to do is to find simple, solid strategies to filter noise from meaning.
In conclusion – our field is messy, noisy, and filled with false positives. But it’s exciting and I love it anyway, and I think it’s crucial that we keep trying to make it better. I don’t believe in genomics, but I do believe in its future.