In 1990, Parade magazine's Ask Marilyn column answered a question about the mathematics of a situation that arose in an American TV game show: in certain circumstances, should contestants stick with their initial choices, or could they improve their odds by switching?
Columnist Marilyn vos Savant's answer angered many mathematicians, who alleged that she had misunderstood the mathematics. Although it turned out that vos Savant was entirely right and the mathematicians were wrong, Pal Erdos, one of the greatest mathematicians of the last century, would not accept this even when presented with a mathematical proof. This confusing dilemma is now known as the Monty Hall Problem, after the game-show host, and is analysed in several recent popular maths books.
The Monty Hall problem is counter-intuitive: the reception that vos Savant received shows that it is not only the layman who can go wrong when asked to make judgments about probabilities. Mlodinow's timely book discusses the mistakes we all make. In the words of a statistician he quotes: "Our brains are just not wired to do probability problems."
The consequences can be serious. Mlodinow shows how the innocent Sally Clark spent three years of a life sentence in prison for murder because an eminent expert witness performed an incorrect calculation of the probabilities that two of her children had died of natural causes.
Many of us find ourselves having to make decisions - whether our children should be vaccinated, which cancer treatment to opt for - that rely on the interpretation of medical statistics, but many in the medical profession misunderstand the figures they quote.
The Drunkard's Walk does three things. It presents a host of examples of the common errors we make, showing just how fallible we are when it comes to assessing probabilities. It also offers a simple, non-technical introduction to the mathematical theory of probability, giving readers valuable tools for making their own estimates of likelihoods, and in the process tells the story of the development of this branch of mathematics over the past 2,000 years.
Mlodinow argues passionately that we do not comprehend the randomness of the world: our human habits lead us to find spurious explanations for essentially random events, and in the process we often make dangerously fallacious judgments about people.
The examples of the mistakes we all make are often entertaining and frequently frightening. Athletes are labelled cheats on the basis of tests that claim impressively low error rates, but Mlodinow shows that, on the basis of the evidence presented, there was a 15 per cent likelihood that Mary Decker Slaney, whose athletics career ended in 1996 after she failed a dope test, was innocent. He himself was told by his doctor in 1989, when he tested positive for HIV, that there was a 99.9 per cent chance that he would be dead within ten years: in fact the odds were 10 in 11 that he did not have the disease.
Simulated court cases have shown that juries come to different verdicts on the same facts depending on the apparently irrelevant incidental details that are presented to them, and many examples show how dangerously misleading the presentation of DNA evidence can be.
We learn how Michael Faraday's investigations into table-turning during seances found the phenomenon to be a consequence of participants' unconscious amplification of random fidgeting movements, and how our reluctance to accept randomness leads us to make false judgments about the abilities of sports teams and fund managers.
However, when the author concludes that an illegal lottery run by New York crime syndicates in the 1920s, cheekily using the last digits of the US Treasury balance as random numbers, was flawed, he seems to me to be misapplying Benford's law, which says only that the initial digits of numbers such as these are not uniformly distributed.
Particularly worrying for those of us who work in education is the book's analysis of the (in)validity of academic assessment: the dramatic and frequently demonstrated variability of the marks allocated by different markers to the same work, and the innate randomness of the assessment process, demonstrate the inadequacy of the classifications that employers and universities use to select between applicants.
Our tendency to use numerical marks, which have significant margins of error, demonstrably leads us astray, whether we are choosing between fine wines or candidates for a job.
Mlodinow's historical account of the development of the mathematics of probability takes in the Romans, via Girolamo Cardano in Renaissance Italy; the demographer John Graunt; William Petty, Cromwell's economist in 17th-century London; the English clergyman Thomas Bayes and his still controversial work on inference; the Belgian Adolphe Quetelet and his homme moyen; Francis Galton's analysis of regression to the mean; the work of Karl Pearson; Einstein's seminal paper of 1905 on statistical physics; and Edward Lorenz's discovery of the butterfly effect in chaos theory.
The historical story illuminates the exposition of the theory and gives valuable context, but at the inevitable cost of some over-simplification. It is fascinating to learn of the problems that motivated the development of probability theory. But, as the book's title - alluding to a standard mathematical model of randomness - indicates, the heart of Mlodinow's argument is that our human need to find explanations causes us to underestimate the role of randomness in our world.
All of us owe our lives to chance, and Mlodinow gives telling examples from his own family history. But we always seek explanations other than chance. The book describes Melvin Lerner's experiments, which show how, even when the outcomes are random, we still rate highly those who perform "well" and denigrate those who are less "successful".
Mlodinow is passionate about the unfairness of the judgments we make because of our innate inability to evaluate randomness. As he says, these judgments may not matter much when we overrate a lucky investment manager, but when we dismiss as failures those whom chance does not favour, the injustice can be very serious.
The author draws the valuable corollary that one should not be discouraged by rejection. The J. K. Rowlings of this world have been successful because they persisted despite disappointments. The view that talent and ability are immediately recognised is too simplistic in this random world.
Mlodinow's passionate, and sometimes emotional, arguments are entirely convincing. Any reader of this book who believes in the desirability of taking DNA samples from innocent people to create a national database for solving crimes, or who might choose a surgeon to perform a heart operation on the basis of previous success rates, or who assumes that a bestseller is necessarily a better book than one rejected by many publishers, will find reason to think again.
If I were ever to appear in court, if I were innocent I might be reassured if I thought that the members of the jury were familiar with Mlodinow's arguments: but, innocent or not, I would certainly want my defence lawyer to have read this important book.
Born in Chicago in 1954, Leonard Mlodinow has worked with luminaries from Stephen Hawking to Steven Spielberg.
Mlodinow developed his passion for physics after joining a kibbutz in Israel in 1973 and reading Richard Feynman's books. Returning to Brandeis University, he completed his degree in 1976 before gaining a PhD from the University of California, Berkeley. He taught at the California Institute of Technology, and then became a Fellow at the Max Planck Institute for Physics and Astrophysics in Munich.
In 1985, he headed for Hollywood with $6,000, dreaming of making it big as a script-writer. Six months and $5,890 later, he sold his first script and went on to write for MacGyver and Star Trek: The Next Generation, while pursuing physics as a hobby. In 1993, he moved to the gaming industry, working with Steven Spielberg, Robin Williams and the Walt Disney Company.
He returned to Caltech in 2005, where he teaches physics and still writes screenplays.
The Drunkard's Walk: How Randomness Rules our Lives
By Leonard Mlodinow
Allen Lane, 252pp, £20.00
Published 29 May 2008