You are a rock star. You step on stage to face flashing lights, smoke from the fog machine and tens of thousands of screaming fans. You scan the audience. Your best friend said she’d come. You try to pick her out from the sea of faces in front of you. Is that her? Can you be sure?
Cosmologists aren’t (usually) rock stars, but finding clues about the first few moments of the Universe can be like searching for a single face in a vast crowd. Instead of looking through fog and flashing lights, you’re looking through galactic dust and starlight. Instead of scanning faces, you’re sifting through patterns in the leftover light from the Big Bang, trying to find new and creative ways to make the signal stand out. And in the same way many faces might look the same from a distance, you could be fooled by a signal that mimics the one you’re expecting.
This last stumbling block seems to have been the downfall of one of the most exciting potential discoveries in modern cosmology. In March last year, scientists working with the BICEP2 telescope at the South Pole saw a characteristic swirling pattern in the polarisation of the light left over from the Big Bang. This pattern, they said, was formed by primordial gravitational waves – ripples in space from the Universe’s birth. If confirmed it would be a huge boost to the theory that the Universe went through a period of rapid expansion in the first tiny fraction of a second. Press conferences were held, articles written, champagne corks popped.
But then came the doubts as researchers found hints that the BICEP2 team’s conclusions might not be reliable. The swirling pattern, instead of a tell-tale sign of the early Universe’s growth, could have been a by-product of our own galaxy’s dust – suspicions now confirmed by a joint analysis between the BICEP2 team and the Planck Telescope collaboration. The dust produces enough polarisation to account for everything BICEP2 saw. In physics and astronomy, seeing a signal is not enough. You have to be sure the signal is really there and not a fluke in your measurements. You also have to be sure it is caused by what you are trying to study. This means endless calibration and modelling to try to understand every aspect of your experiment and collecting enough data to eliminate chance.
The convention is to refrain from touting a “discovery” until the chance of a random signal is about one in 3.5 million. We call that a “five sigma result”, where “sigma” is a shorthand for the deviation from expectation. A one sigma result is fake one time in three – not even worth a mention. At two sigma, the level where it’s considered interesting in astronomy, the chance of the signal being spurious is about 5%. And a three sigma result, only about 0.3% likely to be false, is generally taken as “evidence”. The Higgs boson wasn’t announced as a discovery until two major experimental collaborations at the Large Hadron Collider had found a signal at five sigma.
But to calculate these probabilities you need to have a good understanding of what you’re actually measuring, and to be sure your signal isn’t mimicked by something else. This is where BICEP2 went awry. The original result quoted a detection of primordial gravitational waves at 5.9 sigma. But that relied on an assumption that the dust could not account for the signal – an assumption that seems to have been very wrong. It is as if you were sure your friend at the concert was going to be the only one wearing a red shirt. If you managed to spot anyone wearing red, you could say with confidence that it was her. Then you found out that, actually, hundreds of attendees were wearing red. Was the person in red your friend? Was she even there? Certainty is hard to come by in science. We generally have to aim for “beyond a reasonable doubt”, but sometimes what we consider a reasonable doubt changes with new information.
In the case of BICEP2, the “discovery” wasn’t proved wrong exactly. The signal might really be there – but if it is, it’s there along with an indistinguishable signal from a lot of galactic dust. We can’t say whether BICEP2 saw primordial gravitational waves or not, but it definitely didn’t meet the criteria for a discovery. If there’s anything we can learn from BICEP2’s non-discovery, it’s the immense importance of understanding and stating from the beginning all the reasons that we might not really be seeing what we think we’re seeing. When in doubt, put that doubt front and centre. Meanwhile, the cosmology community will continue doing good science: questioning, testing, double-checking and finding out where the edges of our knowledge really lie.
