65

Lately, I have been learning about the replication crisis, see How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (good YouTube video) — by Michael Shermer and Stuart Ritchie. According to Wikipedia, the replication crisis (also known as the replicability crisis or reproducibility crisis) is

an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social sciences and medicine most severely.

Has the replication crisis impacted (pure) mathematics, or is mathematics unaffected? How should results in mathematics be reproduced? How can complicated proofs be replicated, given that so few people are able to understand them to begin with?

Ellie Kesselman
  • 109
John Tavers
  • 301
  • 4
  • 5
  • 7
    Not quite duplicates, but closely related questions https://mathoverflow.net/questions/35468/widely-accepted-mathematical-results-that-were-later-shown-to-be-wrong and https://mathoverflow.net/questions/27749/what-are-some-correct-results-discovered-with-incorrect-or-no-proofs . – JoshuaZ Sep 05 '20 at 01:13
  • 3
    Here's a quote from a MO question along the lines of what you're asking - "As the Bing topologists familiar with these arguments retire, the hopes of reproducing the details of the proof are fading, and with it, the insight that such a spectacular proof affords." - https://mathoverflow.net/questions/108631/fake-versus-exotic Though the concern might be misguided - https://mathoverflow.net/questions/87674/independent-evidence-for-the-classification-of-topological-4-manifolds – Sam Hopkins Sep 05 '20 at 01:41
  • 6
    Reading a proof and verifying all the details is the mathematical equivalent of replication. I would like to believe that most mathematicians do this for most of the results they cite. There are certainly results that are long and complicated and out of the reach for most people in a field to check in detail, but I would also like to believe that people avoid using them until experts in their field say they have checked the proof. As long as this replication is cheap and easy compared to physical experiments, our replication issues will be smaller. – Aaron Sep 05 '20 at 02:02
  • 46
    As opposed to Aaron I think a vast number of mathematicians do not read the proofs, in particular do not verify the details of (all of) the results they cite. It is not even clear what this would mean since these results themselves rely upon other results etc. I do think they ensure they understand the meaning of the statements and investigate particular interesting cases and the implications on their result. – Hercule Poirot Sep 05 '20 at 04:10
  • Just one ex...it is interesting that (depending on who you ask) Freedman's proof of the 4D Poincaré conjecture and Perelman's proof of the Thurston conjecture are incomplete/unclearly complete, as they are arguably the most widely renowned works in (resp.) geometric topology and geometric analysis. It seems that nobody has been able to reorganize the logic of the existing proofs in order to clarify their content, which seems to me to be the goal of an exposition. Also noteworthy that (depending on who you ask) the existing expositions lack clarity themselves or have their own errors, or both – Quarto Bendir Sep 05 '20 at 04:45
  • 2
    Abbas Bahri's article "Five gaps in mathematics" is an interesting paper which seems directly relevant to the question. (To my understanding, all of his five objections can be resolved, but I believe this is irrelevant to the question.) https://twma.files.wordpress.com/2016/12/five-gaps.pdf – Quarto Bendir Sep 05 '20 at 04:48
  • 6
    Whenever this topic arises, I always post this talk by Voevodsky. https://www.math.ias.edu/vladimir/sites/math.ias.edu.vladimir/files/2014_IAS.pdf – Deane Yang Sep 05 '20 at 05:50
  • 9
    Perhaps Kevin Buzzard's recent talk is also of interest here. https://www.andrew.cmu.edu/user/avigad/meetings/fomm2020/slides/fomm_buzzard.pdf – Balazs Sep 05 '20 at 06:55
  • 2
    See also https://mathoverflow.net/questions/338607/why-doesnt-mathematics-collapse-even-though-humans-quite-often-make-mistakes-in/338610#338610 – darij grinberg Sep 05 '20 at 07:41
  • 6
    See also Proofs shown to be wrong after formalization with proof assistant and Extent of “unscientific”, and of wrong, papers in research mathematics and Results that are widely accepted but no proof has appeared. – Timothy Chow Sep 05 '20 at 13:16
  • 14
    Also related: Which mathematician sampled published proofs and found one third of them to have errors?, Endless controversy about the correctness of significant papers, When have we lost a body of mathematics because errors were found?, What percentage of published mathematics papers are correct?, Nontrivially fillable gaps in published proofs of major theorems. – Timothy Chow Sep 05 '20 at 15:51
  • 5
    I believe the replication crisis in science generally refers to work which is empirical and experimental: basically, results from experimental studies can be impossible to replicate if you don't have that exact same experimental set-up. – Hollis Williams Sep 05 '20 at 21:35
  • 2
    I think this question does not make sense because mathematics does not have experiments. – Monroe Eskew Sep 06 '20 at 10:20
  • 12
    @MonroeEskew I would say reading something sketchy like Gromov's papers is pretty close to being an experiment (with the possible outcomes being either reconstructing or fail to recostruct the arguments). –  Sep 06 '20 at 10:31
  • 16
    One trend which I feel can be damaging to the subject is "announcing" big results, and then not publishing a proof for several years, or decades, or ever. This has a doubly damaging effect because preemptive announcements have the effect of stopping others working towards the results. Not offering a proof, even in preprint form, of course does not allow any checking by others. – Geoff Robinson Sep 06 '20 at 11:32
  • 5
    @JoeT Still, it is disanalogous. The the epistemological structure of mathematics does not involve an empirical data set that is subject increased credence via replication. I think this question is an attempt at buzzword-harpooning via analogy. This is not to contradict anything that is said above regarding doubt in mathematics, but only to question the idea of lumping it together with the “replication crisis.” – Monroe Eskew Sep 06 '20 at 22:10
  • 8
    @MonroeEskew : It sounds to me that you're getting hung up on the word "replication." Suppose we replace it with something like "validation" or "confirmation." It's certainly true that we gain increased confidence in the correctness of a proof every time someone independently checks the argument and confirms that it is correct. Erroneous results creep into the literature when people are careless and/or no additional person checks the result. This is highly analogous to errors creeping into the scientific literature because people are careless and/or no additional person checks the result. – Timothy Chow Sep 07 '20 at 01:19
  • 6
    Considering Blade Runner, just be thankful there's no Replicant crisis too. – Asaf Karagila Sep 07 '20 at 11:47
  • @DanielD. : Are you saying that MIP=RE does not* imply a negative answer to the Connes embedding problem? If so, this is news to me. Do you have any more information about this? – Timothy Chow Sep 09 '20 at 18:34
  • @Timothy Chow no I didn't meant that but I reflecting on it I think I know too little so I shouldn't have commented on the first place so I deleted it, will try to study more and comment less from now on =) – Dabed Sep 28 '20 at 02:53
  • 1
    Dear Ellie, you may find some elements of answer in my article A Replication Crisis in Mathematics?. – A. Bordg Mar 14 '21 at 10:41
  • +1 for @A.Bordg's article. Thank you! – Sam Nead Apr 13 '21 at 09:42

3 Answers3

73

Mathematics does have its own version of the replicability problem, but for various reasons it is not as severe as in some scientific literature.

A good example is the classification of finite simple groups - this was a monumental achievement (mostly) completed in the 1980's, spanning tens of thousands of pages written by dozens of authors. But over the past 20 years there has been significant ongoing effort undertaken by Gorenstein, Lyons, Solomon, and others to consolidate the proof in one place. This is partially to simplify and iron out kinks in the proof, but also out of a very real concern that the proof will be lost as experts retire and the field attracts fewer and fewer new researchers. This is one replicability issue in mathematics: some bodies of mathematical knowledge slide into folklore or arcana unless there is a concerted effort by the next generation to organize and preserve them.

Another example is the ongoing saga of Mochizuki's proposed proof of the abc conjecture. The proof involve thousands of pages of work that remains obscure to all but a few, and there remains serious disagreement over whether the argument is correct. There are numerous other examples where important results are called into question because few experts spend the time and energy necessary to carefully work through difficult foundational theory - symplectic geometry provides another recent example.

Why do I think these issues are not as big of a problem for mathematics as analogous issues in the sciences?

  1. Negative results: If you set out to solve an important mathematical problem but instead discover a disproof or counterexample, this is often just as highly valued as a proof. This provides a check against the perverse incentives which motivate some empirical researchers to stretch their evidence for the sake of getting a publication.
  2. Interconnectedness: Most mathematical research is part of an ecosystem of similar results about similar objects, and in an area with enough activity it is difficult for inconsistencies to develop and persist unnoticed.
  3. Generalization: Whenever there is a major mathematical breakthrough it is normally followed by a flurry of activity to extend it and solve other related problems. This entails not just replicating the breakthrough but clarifying it and probing its limits - a good example of this is all the work in the Langlands program which extends and clarifies Wiles' work on the modularity theorem.
  4. Purity: social science and psychology research is hard because the results of an experiment depend on norms and empirical circumstances which can change significantly over time - for instance, many studies about media consumption before the 90's were rendered almost irrelevant by the internet. The foundations of an area of mathematics can change, but the logical correctness of a mathematical argument can't (more or less).
Timothy Chow
  • 78,129
Paul Siegel
  • 28,772
  • It seems that some of your points #1-4 do indicate that most major theorems are correct, but it seems that none of them address the problem of proofs failing to be understandable or replicable to readers – Quarto Bendir Sep 05 '20 at 05:47
  • 8
    @QuartoBendir I think 2 and 3 provide some check against this. Modern algebraic geometry is hard to learn, but a modern student who has taken the time to learn it properly can come up with very simple proofs of results that would have been considered very hard a century ago. I think most areas of math gradually get easier over time as people discover new connections and generalizations that allow future readers to get more bang for their buck. – Paul Siegel Sep 05 '20 at 06:07
  • I agree, but I don't see how that's helpful for the problem of a present-day mathematician understanding present-day mathematics research, which is what I understand the replication crisis to be about. – Quarto Bendir Sep 05 '20 at 06:11
  • 10
    @QuartoBendir By my understanding the issues that have reached crisis levels in the sciences are twofold: first, that some classical, textbook studies (e.g. the Stanford prison experiment) turned out to be fraudulent or wrong; and second, that very few results are ever replicated at all, even by experts in the same field. I do think that the practice of experts applying and generalizing each other's results helps mathematics avoid these problems even in the shorter term, though admittedly it doesn't do much for mathematicians in other fields. – Paul Siegel Sep 05 '20 at 06:35
  • I agree that the standard textbook material in math must be among the most reliable in the sciences. And I think that what you say applies to certain results and certain methods in modern practice, but that a majority of research does not directly become part of the ecosystem in that way. In part it's like Voevoedksy said, "A technical argument by a trusted author, which is hard to check and looks similar to arguments known to be correct, is hardly ever checked in detail." I think such an instance is not naturally checked against by #2 and 3, although of course in certain cases it is. – Quarto Bendir Sep 05 '20 at 19:40
  • Another way a mathematical "result" can be negative is if it fails to solve a conjecture; i.e. an approach that neither proves nor disproves it. Information about this approach and why it failed can be useful to other mathematicians. – Christopher King Sep 06 '20 at 02:45
  • 12
    Why do you phrase it as if Mochizuki has a proof but only some experts disagree? Isn't it the case that most experts do not believe he has a proof, and only those close to Mochizuki believe that he does? In fact, the very fact that he can publish it in a journal that has himself as chief editor shows without doubt that mathematics will have a replication crisis (however small) if mathematicians do not maintain a list of reputable journals. – user21820 Sep 06 '20 at 02:58
  • 4
    @user21820 I deliberately phrased my remarks about the "serious disagreement" about the "proposed proof" to remain neutral about mathematics that I am not qualified to evaluate. I have not counted the number of experts who hold one opinion or the other on the matter, nor do I intend to - the existence of the disagreement itself is enough for the argument that I'm trying to make here. – Paul Siegel Sep 06 '20 at 05:14
  • 2
    @QuartoBendir I always found Voevodsky's concerns in this matter overblown... Yes, we do not check things as often as we should and there's been a few high profile peer review failures, but it's rare that a new important result is not studied in detail in numerous reading seminars across the world, and it does happen that people find mistakes that way (usually small and easily corrected, but still). The vast majority of published wrong results are small papers of limited interest, which tend to be partially sheltered from the effects of 2-3 in this answer. – Denis Nardin Sep 06 '20 at 08:07
  • 1
    @PaulSiegel: I do not agree that your phrasing was neutral. To describe something as "foundational" or some people as "most dedicated experts" is certainly not at all neutral. I have edited it into a truly neutral version. – user21820 Sep 06 '20 at 09:26
  • 7
    As far am I am aware the only high-profile person from the Anglo-European world who believes in Mochizuki is Fesenko. –  Sep 06 '20 at 09:41
  • 2
    @JoeT: That's not the only thing. Fesenko is not even some arbitrary high-profile person, but actually has vested interest in Mochizuki's work because he was and still is a top promoter of said work. If he ever publicly admits that he doubts the work, he would effectively be saying that he has wasted a large portion of the last 5 years or so, as well as all those mathematicians to whom he publicized said work to... Maybe they know there is a flaw but they are hoping it can be fixed before enough experts catch on. – user21820 Sep 06 '20 at 12:38
  • 2
    The fact that Mochizuki admits "it does not seem possible to get intermediate results in his proof of abc using IUT", as well as many other non-mathematical signals, suggests strongly that there is not much substance. I will personally accept a claimed proof of this level of obscurity only if it comes with a formal verification in some well-known formal system such as Mizar or Coq. Even the proof of Kepler's conjecture by Hales was actually quite understood and widely believed to be true though nobody could be 100% sure. Here the purported proof is described by experts as "impenetrable". – user21820 Sep 06 '20 at 12:44
  • 8
    @user21820 You are reading intent here where none is present. The phrase "dedicated experts" in the original answer refers to both the critics and advocates of the work, and the term "foundational" in context refers to the role of an idea in an argument, not its correctness. I'm going to leave your edits as a show of good faith (and I genuinely don't care), but in exchange I'd like you to go grind your ax somewhere else - none of this discussion is relevant to the question or the answer, and it's pretty clear that only one of us is actually neutral toward the irrelevant dispute. – Paul Siegel Sep 06 '20 at 13:52
  • 1
    @PaulSiegel: I accept your claim that your intent was to be neutral, but I wish to point out that to be neutral you cannot say "remains obscure to all but the most dedicated experts" because it directly implies "only the most dedicated experts understand it", which is not neutral because there is currently no evidence that any expert understands it (rather than merely claims to understand it). As for neutrality, I didn't say my personal view was neutral, but my edited version was definitely neutral (in standard English), so thanks for leaving it as it is now. – user21820 Sep 06 '20 at 14:00
  • 1
    @user21820 I agree with the assertion that "only the most dedicated experts understand [Mochizuki's work]" - as I understand it, the work departs significantly from the mainstream tools in its field. If a consensus emerges that the argument is incorrect then it will be thanks to the dedication of the experts who took the time to work though the argument and find a flaw. The underlying point in the second half of my answer is that mathematics is more robust against replicability issues because it naturally provides incentives for researchers to think critically about each others' work. – Paul Siegel Sep 06 '20 at 14:16
  • 2
    @user21820 But anyway, I think that underlying point still gets across, so if everyone is happy then we can leave the answer as it is now. – Paul Siegel Sep 06 '20 at 14:18
  • 2
    Just to make clear, I completely agree with everything in your answer now. =) – user21820 Sep 06 '20 at 14:30
  • 5
    +1 for "some bodies of mathematical knowledge slide into folklore or arcana unless there is a concerted effort by the next generation to organize and preserve them" I think this is an incredibly important point, perhaps nearly orthogonal to "replication". Some areas of research will naturally wane, perhaps as "interesting" results to prove become rare. But I wonder about cause and effect here: does a field die because it ceases to be interesting; or does it die because getting to the "coal-face" of research becomes too hard for incomers? – Matthew Daws Sep 07 '20 at 10:03
16

How can we expect that increasingly complicated proofs are replicated when so few people can understand them in the first place?

My answer to that is that we do not expect them to be replicated in the usual sense of this word (repeated and included into textbooks with just minor cosmetic and stylistic changes). Rather we expect them to be gradually simplified and streamlined either through changing the proofs themselves by finding a shortcut or replacing the whole argument with a completely different one, or by building a theory that is locally trivial but proceeds in the direction of making the proof understandable and verifiable much faster than the currently existing one. The latter is exactly what Mochizuki tried to do though his goal was rather to just reduce the difficulty from "totally impossible" to "barely feasible" and the prevailing opinion is that he failed in the case of the ABC conjecture though he has succeeded in several other problems.

The first approach is more common in analysis (broadly understood), the second is more common in algebra (also broadly understood), but you can try to play either game in either field. My own perception of what is proved and what is not borders on solipsism: I accept the fact as proven if I've read and understood the whole argument or figured it out myself. So most mathematics remains "unproved" to me and, apparently, will stay unproved for the rest of my life. Of course, it doesn't mean that I'm running around questioning the validity of the corresponding theorems. What it means is that I just never allow myself to rely in my own papers on anything that I haven't fully verified to my satisfaction, try to make my papers as self-contained as possible within practical limits, and that I consider the activity of simplifying the existing proofs as meaningful as solving open questions even in the case when the proofs are reasonably well-known and can already be classified as "accessible". But not everybody works this way. Many people are completely happy to drop a nuke any time they have an opportunity to do it and there is nothing formally wrong with that: the underlying point of view is that our time is short, we have to figure out as many things as possible, and the simplifications, etc. will come later. Probably, we need a mixture of both types to proceed as efficiently as we can.

So I would say that the mathematics is reasonably immune to this crisis in the sense that mathematicians are aware of the associated risks, take them willingly, and try to gradually build the safe ground of general accessibility under everything though the process of this building is always behind the process of the mathematical discovery itself. The same applies to physics and medicine though the gap between the "front line" and the "safe ground" there may be wider. In fact, it applies to any science that deserves to be called by that name. As to the so called "social sciences", they are often done at the level of alchemy and astrology today in my humble opinion (and not only mine: read the Richard Feinman critiques, for example) but we should not forget that those were the precursors to such well-respected sciences as chemistry and astronomy/cosmology, so I view the current crisis there as a part of the normal healthy process of transitioning from the prevailing general "blahblahblah" and weathervane behavior with respect to political winds to something more substantial.

Edit: Paul Siegel has convinced me that things have indeed changed since the time I took (obligatory) courses of Marxist philosophy and the history of communist party, though this change may be not easily visible to the general public because it mainly happens outside academia and is driven primarily by company business interests, so a huge part of it occurs behind closed doors (Paul, please correct me if I misinterpreted what you said in any way). So my statement that the current social sciences are not capable of something beyond general blahblahblah is no longer valid and I retract it. However I still maintain the opinion that it is blahblahblah rather than hard data analysis or other scientific approach that drives many public political and social discussions and decisions of today (I don't know what happens here behind the closed doors, of course, and it may be that, like in advertising, what we see is just what shepherds choose to show to their sheep to drive them in the direction they want, but I prefer to think that it is not exactly the case). If somebody can convincingly challenge that, I would be quite interested.

Apologies to everybody for switching this discussion to a sideline.

fedja
  • 59,730
  • 8
    I think the second half of this comment is typical of the low quality of commentary when it comes to the social sciences, which are a broad gamut of fields, all of which are different. Richard Feynman's thoughts on the topic 70 years ago are not the final word on the subject. There is a replication crisis in social psychology, which is not even all of psychology. Is it a general problem across all of social science? Who knows? But mention the word "social" and suddenly everyone's an expert. – arsmath Sep 06 '20 at 14:56
  • 3
    @arsmath Agreed. I haven't explained myself well enough preferring to curtail the subject to a few words and to refer to the opinion of a way more eloquent and knowledgeable person. If you want a more expanded version of what I think, I can provide it later. General psychology, by the way, is a part of medicine in my classification. We just understand the words "social sciences" slightly differently, that's mainly it. Upvoting your comment but changing nothing in the main text :-) – fedja Sep 06 '20 at 17:18
  • 5
    I suspect your impression of what modern social science looks like is a couple decades out of date. The level of scientific rigor found in digital advertising, market research segmentation, forecasting consumer sentiment, and so forth would meet whatever standard you care to throw at it - there are billions of dollars riding on tiny fractions of a percent accuracy. The problem used to be "How accurately can we model human behavior?" The question has become "Is it ethical to model human behavior as accurately as we can?" – Paul Siegel Sep 06 '20 at 23:27
  • 1
    @PaulSiegel Then why is the replication crisis hitting the social sciences so hard? I think the latest analysis shows that nearly 40% of the research in social psychology (in the top journals, in fact) cannot be replicated. – D.S. Lipham Sep 06 '20 at 23:37
  • 6
    @D.S.Lipham A lot of the best social science research isn't happening in universities anymore - this is because the underlying datasets are huge and proprietary. Think Amazon's consumer demand dataset, Google's clickstream data, Facebook's social graph, etc. Academic social science researchers generally work with different problems, smaller budgets, and less accountability, but a number of specific disciplines are thriving - computational linguistics, for instance, has made epoch-defining breakthroughs in the last year or two. – Paul Siegel Sep 07 '20 at 00:04
  • @PaulSiegel Thanks I understand your point now. – D.S. Lipham Sep 07 '20 at 00:54
  • @PaulSiegel I would rather call all things you are talking about "applied mathematics, computer science and statistics". Have nothing to say against either of them. When people in these fields know what they are doing, the results are most impressive. Even the AI research has finally lifted off despite all scepticism about it in the 90s. Unfortunately, what you say also shows that the current primary consumers of the scientific method (in the sense of the same Feynman) are people who want to make big money (like in 50's it was people who wanted to make big weapons). cont-d – fedja Sep 07 '20 at 01:21
  • 2
    @PaulSiegel I wish I could see that level of scientific rigor, let's say, in education. The budgets are also quite noticeable there but all I have seen coming out of it as far as academic setting is concerned are various "calculus reforms". Would you argue that universities are not primary places of teaching anymore? – fedja Sep 07 '20 at 01:26
  • @fedja If you call these ideas applied math, CS, and statistics, then: applied to what? I tend to define a discipline more by its goals than its methods, and the goals in this context are to model human behaviors and relationships. That's social science. – Paul Siegel Sep 07 '20 at 02:20
  • 1
    @fedja And I don't claim that all branches of the social sciences have been equally improved over the past 30-40 years, any more than I believe that all disciplines of physics, chemistry, or mathematics have been equally successful over that period. To provide comparable results in education, for instance, you would need a comparable dataset, and that would involve collecting minute-by-minute time series on student attention / behavior, mapped to empirical outcomes. I think China is trying this sort of experiment. As above, the question is not "Can we?" but "Should we?" – Paul Siegel Sep 07 '20 at 02:22
  • 1
    @PaulSiegel There is one more point I wanted to bring up. You are talking about market analysis, big databases and Google, Amazon, and Facebook. Now, whom do those companies normally hire to do that market analysis on those big data: social psychology majors (understood broadly) or math majors (also understood broadly), or both? My opinion of whether it should be classified as progress in social sciences or as a mere takeover of traditional social science subjects by natural science hangs on the answer to this simple query. So what would you say (if you know the answer yourself, of course)? – fedja Sep 07 '20 at 03:50
  • 1
    @fedja It's a mixture of both. I work for a much smaller company on a team that does this sort of work, and the backgrounds of the people involved include psychology, economics, political science, and linguistics as well as the more expected math and computer science, and I can tell from my professional network that you see similar diversity at big tech companies. I can also testify that I had to work hard to acquire domain expertise in linguistics, market research, sociology, etc. to be effective; the math / statistics isn't that hard, but good experimental design is. – Paul Siegel Sep 07 '20 at 04:08
  • @fedja Of course the engineering teams at these companies are much larger - maybe a 7 to 1 ratio where I work - and the domain experts are still expected to develop their skills in programming and math / stats. The work is very technical and I doubt it looks much like 1970's social science, but the overarching goals and the epistemology are pretty similar. – Paul Siegel Sep 07 '20 at 04:19
  • 3
    @PaulSiegel OK, then it is a mixture of the progress and the takeover in my opinion :-). That's good actually. Probably the best option of the three. I'll update my "outdated opinion" accordingly ;-) I hope that the ethical problems you are talking about will be also decided by a mixed team like yours, not by the people who are telling us what to think and how to behave today. Thanks for your contribution to this conversation! – fedja Sep 07 '20 at 04:29
  • 1
    @fedja Thanks for the opportunity to contribute! A lot of the work going on at my company is not public, but Gary King (https://gking.harvard.edu/) is on our board and has been a leader in modernizing social science methodology for quite some time; you might check out some of his work if you're curious. And we do make a strong effort to assert ourselves on ethical matters, though usually it feels like shouting at a hurricane... – Paul Siegel Sep 07 '20 at 04:41
  • @fedja Your view here is the kind of non-scientific view that led to the replication crisis in the first place. We have results from a single field (social psychology). This is like the result from a single small-sample experiment. From this, you generalize to all of social science, just like a single social psychology experiment applied to 30 students is generalized to all of humanity through time and space. How do we know that's not just social psychology? How do we know that it's actually not all of science, including physics and chemistry? We don't, until we put in the work. – arsmath Sep 07 '20 at 12:25
  • Social psychology is peculiar in that it's a route to become famous in a way that's available to few other academics. More people have probably heard of the Stanford Prison Experiment than have heard of the Michaelson-Morley experiment. You found an effect that is surprising and yet somehow confirms widespread biases, and you'll be in the New York Times in no time. The power pose research is an example. It was published in 2010, which led to a high profile TED talk in 2012, and being mentioned on Oprah. Most scientists have no route to being mentioned on Oprah. – arsmath Sep 07 '20 at 12:34
  • @PaulSiegel Just want to check with you that I summarized our conversation in my edit to the post correctly. If you find that something essential is distorted, dropped, or added, just let me know and I'll fix it. – fedja Sep 07 '20 at 12:54
  • 2
    @arsmath I wouldn't say "a single field" here: the corresponding Wikipedia article (https://en.wikipedia.org/wiki/Replication_crisis), for example believes it is way more widespread and some other sources corroborate that. But, of course, I'm inclined to agree with you that I have a general tendency to overgeneralize though I disagree that this tendency was the (or even main) root of the replication crisis. As to SPE, I have rather heard of the Milgram shock experiment, though not through Oprah. – fedja Sep 07 '20 at 13:11
  • @fedja Yes, that is a reasonable summary of my argument, and those who want more detail can find it in the comment section. Your remark about your own experience with "social science" as a student interesting - I bet there are some good stories there. – Paul Siegel Sep 07 '20 at 15:39
15

Has this crisis impacted (pure) mathematics, or do you believe that maths is mostly immune to it?

Immune to the replication problem, yes. But not immune to the attitudes which cause scientists to do unreplicable research in the first place. Some mathematicians will announce that a particular theorem has been proven, harvest the glory based on the fact that they have proved things in the past, and then never publish their results. Rota's Conjecture is one notorious example. Now we are in a situation where (a) nobody knows whether it is true and (b) nobody has worked on it for seven years, and probably (if it turns out that no proof actually exists) will not work on it for at least another decade.

How should results in mathematics be reproduced?

In science, it would be ideal if people dedicated research time to replicating published experimental results. This doesn't happen much because there is no glory to be gained by doing it.

The analogue in mathematics would be for people to publish new proofs of existing results, or expositions of existing proofs, which is happily much more common. I don't mean copying out well-known results in new language (Tom Leinster, The bijection between projective indecomposable and simple modules), I mean expository papers like this (Cao and Zhu, A complete proof of the Poincaré and geometrization conjectures, Asian J. Math. 10 (2006) pp. 165–492).

Even more noble are the people using proof assistant software to verify existing mathematics.

How can we expect that increasingly complicated proofs are replicated when so few people can understand them in the first place?

I think our best hope is proof assistant software. Perhaps by the end of this century, we will he living in a world where no mathematician can replicate any reasonably cutting-edge proof, yet research is still happily chugging along.

Timothy Chow
  • 78,129
Flounderer
  • 373