The proofs are in the imagination

In 1945, the French mathematician Jacques Hadamard published The Psychology of Invention in the Mathematical Field . As well as finding evidence in the life and work of his predecessors, Hadamard sent a questionnaire to many scholars, including Albert Einstein, inquiring how ideas came to them.

This fine book did not create a stir in psychology: under the influence of behaviourism, researchers were reluctant to discuss the mind or the unconscious. The landscape has since changed, partly now that technology can watch the brain at work. As a sign of the times, Princeton University Press reissued the book in 1996 with the new title The Mathematician's Mind .

One might then think that David Ruelle's book from the same publisher with almost the same title takes the story further. It doesn't. Ruelle rambles through mathematics and mathematicians, ranging from the shabby treatment of Alexander Grothendieck by the French establishment to Taro Asano's proof of the Lee-Yang Circle Theorem. All we are told about how Grothendieck produced his massive work (uniting the discreteness of arithmetic and the continuity of geometry) is that he himself combined "exceptional working capacity" with "technical reliability and imagination".

When Ruelle reflects, it is as a philosopher rather than a psychologist. It is difficult to find a theme, but a distinction between the formal and structural aspects of mathematics is drawn, and we are steered between the extremes of Platonism and postmodernism.

Ruelle compares mathematics to rock-climbing. There is no Platonic route up the cliff, but there is a limited choice of routes that almost all climbers will find. However, a lizard, or an alien, will see a completely different landscape.

The book suffers from a number of misprints, poor typesetting of mathematics, some errors (Desargues' Theorem is attributed to Pappus), and the lack of an index.

In How Mathematicians Think , William Byers divides mathematics not into formal and structural, but into formal and creative (roughly the technical reliability and imagination that Ruelle ascribes to Grothendieck). His thesis is that mathematics is perhaps the most creative of all human endeavours, but that this aspect is downplayed in favour of its logical consistency. If mathematics were nothing but logic, it would be dead, and we could turn it over to the computers without regret. He intends to overturn the preconception in favour of logic.

Byers finds the source of this creativity in ambiguity, which he defines (following Arthur Koestler) as "a single idea perceived in two self-consistent but mutually incompatible frames of reference". A simple but important example concerns the square root of 2. The Greeks had no trouble with this as a geometric number: it is the ratio of the diagonal of a square to its side. The discovery of its irrationality showed that it could not be an arithmetic number in the sense they understood. This ambiguity led to a huge amount of mathematics, from the refined theory of proportion in Euclid down to the work of Grothendieck.

Desargues' Theorem is an example of this ambiguity. On the one hand, it is a theorem of Euclidean geometry that becomes "obvious" if the diagram is interpreted as being three-dimensional. On the other, it is crucial in the foundations of geometry: it tests whether our geometry can be co-ordinatised by a "field" satisfying the laws of arithmetic.

A common source of ambiguity is between object and process. Thus, the equation 1 = 0.999... makes students nervous because on the left we have an object, but on the right a process (adding up the terms of an infinite series). The ambiguity leads us to the notion of limit, which is basic to calculus and mathematical analysis.

The most important role of ambiguity in the creation of new mathematics is shown by conjectures and open problems. We do not know whether the result we are trying to understand is true or false; we have to keep both options open or risk wasting years on the wrong track. Imagine if, after Andrew Wiles had spent seven years in his attic wrestling with Fermat's Last Theorem, someone with a big computer had produced a counter-example.

Examples of ambiguity abound: from complementarity in quantum theory and duality in string theory, through the relationship between mathematics and physics, to the mind-body problem. All have driven creativity.

In summary, Byers argues, "Each deep concept... changed the mathematical and therefore the human landscape. Many appeared paradoxical, or, rather, did indeed have a paradoxical aspect to them. When they became part of normal mathematics, mathematicians learnt to take them for granted... It is only when... teachers see the difficulties that students experience in grasping these concepts that one becomes aware that these difficulties are real." The heart of the book is a discussion of many of these paradoxes: zero, limits, infinite sets, "monstrous moonshine", Sarkovskii's theorem on chaos, Imre Lakatos' account of Euler's polyhedron formula, Gödel's incompleteness theorem, and randomness.

Turning to philosophy, Byers attempts to steer between Platonism and postmodernism by invoking a paradoxical "objective subjectivity", which I found less convincing than Ruelle's clear metaphor.

Byers acknowledges that mathematicians understand and use ambiguity to drive their research. The book is written to persuade the lay person that the subject is not at all logically cut and dried. The result descends occasionally into a denigration of precision and logic. I would not like my students to gain the impression that correct proofs are unimportant.

The author is a Zen Buddhist, and his notion of the creative use of ambiguity resembles that of the Zen koan, driving the student beyond rational thought. But Buddhists also know that some truths are revealed only to advanced students because they would be misunderstood by beginners. The Zen student's satori must be preceded by years of arduous study - and so it is with mathematics.

Ruelle's book will take you on a tour. You will see strange things but will not be expected to understand them. From Byers's book, if you work at it, you will learn some mathematics and, more important, you may begin to see how mathematicians think.

Peter J. Cameron is professor of mathematics, Queen Mary, University of London.