Famous examples of "serendipity" in 20th century mathematics

Question

The term "serendipity" is commonly used in the literature to refer to the historical evidence that very often researchers make unexpected and beneficial discoveries by chance, while they are looking for something else or completely opposite. I wonder if there are any famous examples in 20th-century math where mathematicians made remarkable discoveries while searching for something completely different. As a topologist, the first example came to mind is the Whitehead manifold. In 1935, Whitehead discovered the first exotic contractible open manifold (not homeomorphic to $\mathbb R^3$) while he was trying to prove the Poincaré conjecture. Jones polynomial might be another one. I guess there might be a long anecdotal list of discoveries. But any references would be appreciated.

Edit A little background regarding the motivation of my question. The biochemist Yaqub recently identified serendipity into 4 basic types (Walpolian, Mertonian, Bushian, Stephanian). One reference would be Serendipity: Towards a taxonomy and a theory. The definitions are given in the comments to wlad's answer. But maybe the following table borrowed from Yaqub is more straightforward. Since Yaqub's research covers all fields in science, I was wondering if the same applies to 20th century math and which type(s) occur more frequently in math.

Answer 1

The discovery of chaotic, "unpredictable" dynamics, in a deterministic system by Edward Lorenz in 1963 was a serendipitous discovery.

Lorenz describes it as follows:

“At one point I decided to repeat some of the computations in order to examine what was happening in greater detail. I stopped the computer, typed in a line of numbers that it had printed out a while earlier, and set it running again. I went down the hall for a cup of coffee and returned after about an hour, during which time the computer had simulated about two months of weather. The numbers being printed were nothing like the old ones. I immediately suspected a weak vacuum tube or some other computer trouble, which was not uncommon, but before calling for service I decided to see just where the mistake had occurred, knowing that this could speed up the servicing process. Instead of a sudden break, I found that the new values at first repeated the old ones, but soon afterward differed by one and then several units in the last decimal place, and then began to differ in the next to the last place and then in the place before that. In fact, the differences more or less steadily doubled in size every four days or so, until all resemblance with the original output disappeared somewhere in the second month. This was enough to tell me what had happened: the numbers that I had typed in were not the exact original numbers, but were the rounded-off values that had appeared in the original printout. The initial round-off errors were the culprits; they were steadily amplifying until they dominated the solution.”

_{A close-up of Lorenz’s original printout shows two time series generated with the same equations but with slightly different initial conditions. The series diverge exponentially with time due to sensitive dependence on initial conditions.}

Source: Chaos at Fifty.

Answer 2

As I mentioned in the comments, one particularly striking example of serendipity is the connection between the theories of L-functions and random matrices.

The IAS hosted a conference recently, commemorating 50 years since the discovery of this connection, and I highly recommend the history talks given by Montgomery, Sarnak, and Keating.

The story, paraphrased a bit, goes as follows. In the early 70s, Hugh Montgomery (then a graduate student of Davenport), motivated by an approach to solving the Siegel zero problem, formulated what is now called Montgomery's pair correlation conjecture for the zeroes of $\zeta(s)$. By Montgomery's own telling, when you proved something "new" in analytic number theory at the time, the first thing you did was ask Atle Selberg whether he had already proved this and left it unpublished in a drawer somewhere. So, Montgomery visited Selberg at the IAS to talk to him about it.

It turned out that Selberg had not, in fact, already done what Montgomery showed him. After his conversation with Selberg, Sarvadaman Chowla took Montgomery to tea-time, and introduced him to Freeman Dyson (much to Montgomery's chagrin: Dyson was already a famous physicist by then, and Montgomery "didn't want to bother him").

Dyson asked Montgomery what he was working on, and Montgomery said something to the effect of "I think that the distribution function of spacings between zeroes of the Riemann zeta function is $1 - (\frac{\sin \pi u}{\pi u})^2$...", to which Dyson reportedly replied without missing a beat something to the effect of "... that's the pair correlation of the eigenvalues of random matrices from the GUE ensemble". Both these quotes are not precise -- I don't think anyone who was there actually remembers precisely what was said -- but it's not far off. I suggest this link and page 5 of this paper for some popular science accounts of the conversation. This article from the IAS, shared by Stopple in the comments is also very nice.

(In case you are wondering why Dyson knew this: random matrix theory started out as an approach to model energy states in nuclei of heavy atoms, and physicists -- including Dyson himself -- developed much of the early theory).

In terms of the mathematics, assume RH and define

$$F(\alpha) = \frac{1}{N(T)} \sum_{\gamma,\gamma'\leqslant T} T^{i\alpha (\gamma - \gamma')} w(\gamma - \gamma'),$$

where the sum runs over ordinates $\gamma,\gamma'$ of nontrivial zeroes of $\zeta(s)$ in the interval $[0,T]$, $w(t) = \frac{4}{4+t^2}$ is an explicit weight function that you should think of as damping the effects of large spacings $\gamma - \gamma'$, and $$N(T) = \sum_{\gamma \leqslant T} 1,$$ is the counting function of the zeroes. $F$ should be thought of as the normalized Fourier transform of the distribution of $\gamma - \gamma'$.

Montgomery proved that $$ F(\alpha) = |\alpha| + o_\epsilon(1),$$ uniformly for $\epsilon \leqslant |\alpha| \leqslant 1 - \epsilon$, (there's some lower-order terms I am neglecting for the sake of exposition), and conjectured based on heuristics involving prime $k$-tuples that $$F(\alpha) = 1 + o_\delta(1),$$ uniformly for $1 \leqslant \alpha \leqslant 1 + \delta$ and any $\delta > 0$.

If one assumes Montgomery's conjecture and applies Fourier inversion, then one obtains that $$ \frac{1}{N(T)} \sum_{\substack{\gamma,\gamma' \leqslant T\\0 < \frac{\gamma - \gamma'}{2\pi/\log T} \leqslant \beta}} 1 \sim \int_0^\beta\bigg(1 - (\tfrac{\sin \pi u}{\pi u})^2\bigg) du, $$ which hopefully clarifies the story above. (Note that one divides $\gamma - \gamma'$ by $\frac{2\pi}{\log T}$ as this is the mean spacing between zeroes for $\zeta$, which follows from $N(T) \sim \frac{1}{2\pi}T \log T$.)

If you replace the zeroes of $\zeta$ with eigenvalues of GUE matrices in the large dimension limit, then an analogue of the above was well-known. In fact, the eigenvalues of GUE matrices are an example a determinantal process, and their $n$-level correlations were all well-understood by physicists. Montgomery speculates on the basis of this that the distribution of zeroes on the critical line is the same as that of GUE matrices, which is usually called the "GUE hypothesis". This speculation has a lot of supporting numerical evidence, due to the work of Andrew Odlyzko, who used computers at Bell Labs for his computations; for this reason the GUE hypothesis is sometimes called the Montgomery--Odlyzko law.

Hejhal computed the triple correlation and Rudnick--Sarnak computed the $n$-level correlations (with as large Fourier support as current methods allow) for $\zeta(s)$ and showed they agree with the GUE hypothesis. Rudnick and Sarnak actually computed $n$-level correlations for all high zeroes of principal automorphic $L$-functions, and showed agreement with GUE.

This connection has proven to be fruitful in more ways than indicated above. The two most prominent examples are:

1. The Katz--Sarnak density conjectures. Motivated by their work on function field analogues, Katz and Sarnak conjectured that the low-lying distribution of zeroes (i.e., the zeroes with imaginary ordinate $\ll \frac{1}{\log \mathfrak{q}_f}$ where $\mathfrak{q}_f$ is the conductor of $L(s,f)$ and $f \in \mathcal{F}$ runs over some automorphic "family") of $L$-functions depends only on a symmetry type associated with $\mathcal{F}$, which can be one of unitary, symplectic, or (odd/even) orthogonal and matches the corresponding statistic for random matrix ensembles. One can view the above as a special case of these conjectures since $t \mapsto \zeta(1/2 + it)$ can be thought of a continuous family of $L$-functions parametrized by $t$, with unitary symmetry type. A seminal result on these conjectures is the work of Iwaniec--Luo--Sarnak, where these conjectures were proved for many families and bandlimited functions with small enough Fourier support.
2. The Keating--Snaith moment conjectures. Since the characteristic polynomial of a matrix vanishes at its eigenvalues, while $\zeta(s)$ vanishes (obviously...) at its zeroes, one might wonder whether $t \mapsto \zeta(\tfrac12 + it)$ might be modeled by characteristic polynomial of a random matrix. Keating and Snaith used this idea to generate conjectures for moments of $\zeta(s)$ by computing the moments of characteristic polynomials in the CUE ensemble [which is an analogue of GUE where the eigenvalues are on the complex unit circle instead of the real line], and comparing with $\zeta(s)$. This idea is robust and applies to moments in Katz--Sarnak families.

Answer 3

[Feel free to suggest or apply improvements to this answer; it's Community Wiki]

Machine Learning is arguably an example of serendipity in 20th century mathematics, but probably not what people had in mind. The fact that it can be done sometimes using only a computer can make it as self-contained as the rest of mathematics:

You take a linear classifier (SVM, logistic regression) or linear regression, which have low accuracy on most problems, and then you introduce a simple non-linearity, making accuracy on many difficult tasks jump from close to 0% to above 99%.

And then you try and learn the non-linearity using gradient descent, and you get Multi-Layer Perceptrons, and iterate this leading to the 21st century discovery of Deep Learning.

It's serendipitous because:

• One non-linearity has a dramatic positive effect.
• The learning algorithm is Stochastic Gradient Descent, which is often considered very crude.
• It only finds local minima for loss functions, but this is somehow not a problem.
• Increasing the number of free variables, or the number of "layers", actually improves accuracy. I think expectations were once that it would have the opposite effect.

Presently, SOTA algorithms for computer/board game AI, translation between languages, chatbots, automation of illustration jobs, etc use these approaches.