On a cold January morning in 1986, the space shuttle Challenger lifted off its launch pad at the Kennedy Space Center in Florida. Morale was high, especially as the Challenger flight was to inaugurate the teacher-in-space program with astronaut/high school teacher Christa MacAuliffe in its crew. Alas, 73 seconds into the flight the shuttle disintegrated, destroying the spacecraft and killing all the astronauts on board. The cause of the accident was a leak of hot gas from one of the solid rocket boosters. The leak occured because of the failure of rubber “O-rings” which were supposed to seal the joints between rocket sections, and part of the reason they failed is that the temperature was so cold at the time of the launch — the O-ring material becomes more stiff at low temperature so it’s less likely to make a proper seal.
The ambient temperature at launch was near 31F (about -1C), the coldest shuttle launch yet attempted by a wide margin (the next-coldest launch on record was at 53F, more than 20F warmer). The low temperature was of great concern to engineers from Morton Thiokol, the contractor who built and maintained the shuttle’s solid rocket boosters. At a teleconference the night before the launch with engineers and managers from Thiokol and from NASA’s Kennedy and Marshall Space Flight Centers, several of the engineers — most notably Thiokol’s Roger Boisjoly — expressed concern about the effect of the temperature on the resilience of the O-rings.
Thiokol engineers argued that if the O-rings were colder than 53F (12C) they didn’t have enough data to determine whether the joint would seal properly. The O-rings were a “Criticality 1” component, meaning that their failure would destroy the Orbiter and its crew. As the Report of the Presidential Commission on the Space Shuttle Challenger Accident (the Rogers Report) makes clear, this alone made the engineers’ concerns sufficient reason to cancel the launch.
In fact there was enough data available to discern the connection between temperature and O-ring damage, but when the data were examined, those involved made a classic but very natural mistake. They looked only at data for flights on which O-ring damage had been observed, ignoring all data from flights with no observed O-ring damage. In volume 1 of the Rogers Report it states:
The record of the fateful series of NASA and Thiokol meetings, telephone conferences, notes, and facsimile transmissions on January 27th, the night before the launch of flight 51-L, shows that only limited consideration was given to the past history of O-ring damage in terms of temperature. The managers compared as a function of temperature the flights for which thermal distress of O-rings had been observed — not the frequency of occurrence based on all flights (Figure 6). In such a comparison, there is nothing irregular in the distribution of O-ring “distress” over the spectrum of joint temperatures at launch between 53 degrees Fahrenheit and 75 degrees Fahrenheit.
Their figure 6 shows the data they considered, and indeed there’s no obvious relationship between temperature and O-ring damage.
Unfortunately this graph omits all data for flights on which there was no O-ring damage. As the Rogers Report goes on to say:
When the entire history of flight experience is considered, including “normal” flights with no erosion or blow-by, the comparison is substantially different (Figure 7). This comparison of flight history indicates that only three incidents of O-ring thermal distress occurred out of twenty flights with O-ring temperatures at 66 degrees Fahrenheit or above, whereas, all four flights with O-ring temperatures at 63 degrees Fahrenheit or below experienced O-ring thermal distress.
Consideration of the entire launch temperature history indicates that the probability of O-ring distress is increased to almost a certainty if the temperature of the joint is less than 65.
Their figure 7 shows quite clearly that there is a relationship between temperature and O-ring damage on previous flights. The situation is well summarized in item 6 of the “conclusions” section of chapter 6 of the Rogers Report:
6. A careful analysis of the flight history of O-ring performance would have revealed the correlation of O-ring damage and low temperature. Neither NASA nor Thiokol carried out such an analysis; consequently, they were unprepared to properly evaluate the risks of launching the 51-L mission in conditions more extreme than they had encountered before.
Perhaps the saddest part of the story is that yes, the data did exist, and were on record and available for study, clearly to demonstrate the extreme danger of launching the shuttle at such cold temperature. They were simply not analyzed properly.
This failure wasn’t about advanced math. It had nothing to do with a lack of sophisticated, complex mathematical techniques to squeeze the last drop of information from the available numbers. It was a simple mistake of leaving out relevant data, which happened to constitute most of the data, including the part that made the important insight so obvious that you really don’t need fancy math to get it.
The science which deals with the analysis, interpretation, and presentation of data is called statistics. Done well, it can reveal keen insights and guide us to beneficial choices. Done poorly it can mislead us, contributing to unwise choices about everything from auto safety to the school budget to the best lineup for our baseball team, to approving the launch of a space shuttle.
It’s all too easy to mislead people, including yourself (especially yourself!) with badly-done statistics. It’s not so easy to do it right, so that the data can speak for themselves and make the important message plain. But it can be done, and it may not be easy but in many cases it’s not too difficult either. It really is well within the ability of most people to apply statistics correctly, in order to make far better use of the data they have available. You may never reach the heights of statistical sophistication — few do — but you can get the basics right, and that’s most of the battle. In fact that’s almost all of it. Most truly valuable insights are not far beneath the surface, you don’t need a jackhammer or a surgeon’s scalpel to extract them, all you need is a plain old shovel and the knowledge how to use it. And let’s face it: learning how to use a shovel is not that hard.
No amount of mathematical sophistication can protect you from overlooking something simple, making the kind of mistake that can lead to disaster. Fortunately, few such decisions are a matter of life and death.