95

I have an idea for a website that could improve some well-known difficulties around peer review system and "hidden knowledge" in mathematics. It seems like a low hanging fruit that many people must've thought about before. My question is two-fold:

Has someone already tried this? If not, who in the mathematical community might be interested in creating and maintaining such a project or is working on similar projects?

Idea

A website dedicated to anonymous discussions of mathematical papers by experts.

Motivation 1: Hidden knowledge

Wilhelm Klingenberg's "Lectures on closed geodesics" can be found in every university's math library. One of the main theorems in the book is the following remarkable result, a culmination of decades of work on Morse theory of the loop space by many mathematicians: Every compact Riemannian manifold contains infinitely many prime closed geodesics.

Unfortunately, there is a mistake in the proof. 44 years after the book's publication the statement is still a widely open problem. The reason I know this is because when I was in grad school I mentioned the book to my adviser and my adviser told me about it. If I tried to look for this information online I wouldn't find it (in fact, I still haven't seen it written down anywhere).

This is one of many examples of "hidden knowledge", information that gets passed from adviser to student, but is inaccessible to an outsider. In principle, a new Ramanujan can open arxiv.org and get access to the cutting edge mathematical research. In reality, the hidden knowledge keeps many mathematical fields impenetrable to anyone who is not personally acquainted with one of a handful of experts.

Of course, there is much more to hidden knowledge than "this paper form 40 years ago actually contains a gap". But I feel that the experts' "oral tradition" on papers in the field is at the core of it. Making it common knowledge will be of great benefit to students, mathematicians from smaller universities, those outside of North America and Europe, people from adjacent fields, to the experts themselves and to the mathematical progress.

Motivation 2: Improving peer review

Consider the following situations:

  • You are refereeing a paper and get stuck on some minor issue. It will take the author 5 minutes to explain, but a few hours for you to figure it out on your own. But it doesn't quite feel worth initiating formal communication with the author through the editor over this and you don't want to break the veil of anonymity by contacting the author directly.
  • You are being asked to referee a paper, but don't have time to referee the whole paper. On the other hand, there is a part of it that is really interesting to you. Telling the editor "yes, but I will only referee Lemma 5.3" seems awkward.
  • You are refereeing a paper that is at the intersection of your field and a different field. You would like to discuss it with an expert in the other field to make sure you are not missing anything, but don't know a colleague in that area or feel hesitant revealing that you are a referee for this paper.

These are some of many situations where ability to anonymously discuss a paper with the author and other experts in a forum-like space would be helpful in the refereeing process. But also outside of peer review mathematicians constantly find small errors, fillable gaps and ways to make an old paper more understandable that they would be happy to share with the others. At the same time, they often don't have time to produce carefully polished notes that they would feel comfortable posting on arxiv, or if they post notes on their website they may not be easy to find for anyone else reading the paper. It would be helpful to have one place where such information is collected.

How will it work?

The hope is to continue the glorious tradition of collaborative anonymous mathematics. One implementation can work like this: Users of the website can create a page dedicated to a paper and post questions and comments about the paper on that page. To register on the website one needs to fill in a form asking for an email and two links to math arxiv papers that have the same email in them (this way registration does not require verification by moderators) and choose their fields of expertise. When a user makes a comment or question only their field/fields are displayed.

user2718
  • 137
  • 9
    Perhaps the very idea of a website that "exposes" (not in a malicious way) serious errata on well-known articles would be extremely useful. It also happened to me, that I was going to read some article and my advisor told me to be careful since there were errors in the methods used, that were fixed later (but absolutely no hints where these were fixed, and no mention of the errors nowhere). – efs May 30 '21 at 12:47
  • A first step for this is to have a comprehensive data of mathematical publications and books, say in the last 75-100 years. I'm not sure who has this, beyond MathSciNet (which draws financial benefits from this as well as of volunteer reviews) and ZBMath (which is now free). – YCor May 30 '21 at 12:50
  • 1
    This would need resources, first to code it up, and then (more seriously) to maintain it, to avoid it becoming a magnet for spammers. – J.J. Green May 30 '21 at 12:52
  • @J.J.Green The hope is that since only verified users can post on the website (e.g. those that have papers on arxiv) there will be less danger of spamming. – user2718 May 30 '21 at 12:58
  • @J.J.Green "magnet for spammers" ... yes – Gerald Edgar May 30 '21 at 14:17
  • 59
    Versions of this idea have been periodically proposed in the community. However, there is a very serious concern to any "official" review database of this sort, which is that there is the risk that such a database could be weaponised by anyone with a personal grudge or bias against another academic, school of mathematics, topic of study, or demographic. It could also worsen equity if certain groups of mathematicians attract disproportionate criticism while others have too much stature to have their work discussed in any negative fashion. These are nontrivial issues. – Terry Tao May 30 '21 at 15:02
  • 3
    @TerryTao Thank you! I agree that these are non-trivial issues. There is one aspect where anonymity is helpful: people are often reluctant to point out errors or gaps in works of mathematicians of high stature compared to those who are less established, and would be more willing to do it anonymously. One can try to address other issues by enforcing stricter moderation rules, e.g. allowing only factual statements: comments like "there appears to be a gap in Lemma 5.3 that can be filled using methods of \cite{XX}" would be allowed, but "this result is trivial" wouldn't. – user2718 May 30 '21 at 17:41
  • 8
    Another approach is to start with only older papers (e.g. that appeared 20+ years ago, where there's lower probability of drama and harm to someone's career) and if the project becomes successful and attracts sufficiently many moderators gradually expand it to more recent papers. – user2718 May 30 '21 at 17:41
  • 3
    @TerryTao do you have any examples of someone "getting in trouble" for pointing out an error in a mathematical paper, even if it's just bad will? I'm not sure I've ever heard of an example of that. – Christopher King May 30 '21 at 20:42
  • 2
    @user2718 I think now it is perfectly googlable, see page 3 of http://perso.ens-lyon.fr/marco.mazzucchelli/preprints/max_degree.pdf :) – kerzol May 30 '21 at 20:46
  • 1
    @kerzol great, thank you! it took only 40 years. – user2718 May 30 '21 at 20:53
  • 1
    @TerryTao I'd rather know about a serious error, whatever the motivation for pointing it out was, than not know about it at all. I frankly don't care what the motivation is, or see how it's relevant, and I suspect most mathematicians would say the same. – user76284 May 30 '21 at 22:06
  • 3
    @user2718 I'm not the specialist of the domain in question, but if Klingenberg's conjecture is important one, should one consider to mention it at https://en.wikipedia.org/wiki/Closed_geodesic ? – kerzol May 30 '21 at 22:06
  • 8
    @PyRulez I think it usually happens the other way: Person Alpha erroneously claims that Person X's result is wrong, or that their work is already contained in Person Alpha's work, and it is Person X that suffers. – Kimball May 30 '21 at 23:06
  • 5
    @user76284 One problem is there are often heated debates where the prover and the accuser disagree as to whether it is really a correct proof or not. One rather recent instance (I'm not sure how heated this was, but as far as I know there is still no resolution agreed upon by all parties) concerns Mochizuki's work related to the abc conjecture. – Kimball May 30 '21 at 23:10
  • @Kimball Even though it's painful for people involved, it's often good for the field. In contrast, I know a subfield where no one is willing to speak up about some important proofs of ambiguous status. The subfield is dying, as nobody (especially young people) wants to spend several years trying to clarify what's already considered proved, but also many people quietly stopped using the results or anything that relies on them. – user2718 May 31 '21 at 00:19
  • 11
    We already have anonymous websites for discussion and rating of our teaching, like the ratemyprofessors site. I am surprised that any researchers are eager to have anonymous websites for discussion and rating of our research too. –  May 31 '21 at 01:07
  • 5
    @user76284 Daniel Biss's political opponents were delighted at the news of errors in his mathematical papers. In this case, as far as I know, the alleged errors were bona fide errors. But it is not hard to imagine a political opponent making hay out of a false accusation of an error in a paper. There are some famous cases where there was a debate over whether an argument was "incomplete." These could have been uglier if the claim of incompleteness had been anonymous. – Timothy Chow May 31 '21 at 01:07
  • 5
    @user2718 It is indeed bad when nobody is willing to speak up about some important proofs of ambiguous status. But I am doubtful that anonymity will solve such problems. For one thing, an anonymous complaint is easier to ignore. And in many cases, the problem is that the work to clarify the situation is considerable. I think the promise of anonymity is unlikely to motivate someone to do the work if they're not sufficiently motivated already. – Timothy Chow May 31 '21 at 01:17
  • 2
    @TimothyChow I imagine the main application is clear errors and clarifications, not ambiguous issues of if the proof is clear enough. – Christopher King May 31 '21 at 01:56
  • 2
    info.inquire.pub (main site not yet launched) is a non-profit startup with a similar purpose. We aim to be a Q&A site with a reputation system and upvotes where each question is associated with a specific paper (we support ArXiv or DOI so far). We continually discuss the issues @TerryTao brings up above and are committed to designing a system which will be robust against trolling and weaponisation. The current peer review process is far from perfect and once articles are published they're rarely every edited again. If anyone would like to help us, my contact info is available at mathandy.ai – mathandy May 31 '21 at 04:03
  • 2
    The answers show that this idea has been already tried multiple times without much success. I would appreciate if OP accepted one of the answers, so that it is clear that this question is settled and it does not need any more input. (Because this is a factual question and not a thinly disguised "please discuss" one, right?) – Federico Poloni May 31 '21 at 08:57
  • @FedericoPoloni I greatly appreciate the answers and comments here. Learning about projects under construction in the answers below is also very useful. I will accept an answer tonight. – user2718 May 31 '21 at 11:45
  • 2
    The machine learning community sometimes uses OpenReview.net for conference submissions, which allows question answer discussion with the authors (by anyone). – user27182 May 31 '21 at 14:09
  • A concern would be, that given the small community likely to surround or focus on such specific papers, I doubt that anonymity will be possible. Writing styles, spelling, notation, social conduct, "mathematical beliefs/approaches/positions", will very quickly make clear who's who, to someone who knows the mathematicians active in a field. After all there aren't that many. – Stilez May 31 '21 at 16:48
  • 1
    @Stilez This is also true for the current peer review process; nonetheless, we often find the veil of anonymity helpful even if it's semi-transparent. For example, it makes it psychologically easier for a reviewer to ask a potentially stupid question, reveal their confusion. – user2718 May 31 '21 at 18:46
  • 3
    Besides the problem which Tao raises, I think there is a ''don't break it if it works'' aspect to all this. My experience is that although the current arrangement is not perfect, it works quite well most of the time and the outcome is usually fair. – Hollis Williams May 31 '21 at 19:12
  • 3
    Related: Are there any good websites for hosting discussions of mathematical papers?. Maybe to some extent also: Errata database? and What could be some potentially useful mathematical databases? – Martin Sleziak Jun 02 '21 at 10:04

6 Answers6

29

I'm the founder of https://papers-gamma.link, an Internet place to discuss scientific articles, mentioned by Matthieu Latapy. I have been supporting this site for 6 years now. I hope that one day it will become popular (in a good sense of the word) and useful for the entire scientific community. As you may imagine, I'm pretty convinced that the idea of public review and public comments is, potentially, a very promising one. My persuasion is less than 6 years ago though, and here's why.

Observing that Papers$^\gamma$ gains its popularity very slowly, I started to think more and more about the scientific review and publishing processes. I'll share with you my current understanding of this subjects, admitting that these issues are more likely to be in the field of sociology rather than mathematics.

The original goal of scientific journals was to inform about and discuss the research currents. But after, this had to make room for other things: archiving and bibliometrics. I'm OK with archiving. But, it seems that the optimization of bibliometric statistics negatively affects the discussion power, the original (!) goal of the scientific journals.

Let me try to illustrate exactly what I mean by "discussion power" of scientific journals. Consider Miller's one-page paper [2], which is an excellent example of conversational mathematics. The paper contains an alternative proof of Miles' results [1] about the characteristic equation of the $k$-th order Fibonacci sequence. Miller's paper can be considered as a response to Miles's paper. Compare this to the modern Internet forums like MathOverflow.

Does any journal, wiki, or another internet portal is attracted to archiving and bibliometrics as time goes forward? Maybe there is a natural or induced drift from "discussion" towards "judgment"? If it is the case, should we consider to create a new journals every, say, 10 or 20 years, to restart discussion processes? Or should we be extremely careful, trying to keep the discussion power of existing scientific journals?

Here are main arguments against the mass acceptance of public review and comments, that I can imagine:

  1. Existing systems works pretty well.

  2. Not all conversations should be made public. Private reviews and conversations have their advantages. People feel more free to commit errors, express misunderstanding, and criticize in private. Good reviews help the authors to polish and publish almost perfect articles.

  3. The more a resource is open, the more it is susceptible to spam.

  4. Anonymous public discussions may be used as a platform for attacking other scientists.

Here is a list of some online resources related to the idea of public reviews and collaborative science in general:

  • MathSciNet and Zentralblatt MATH, databases with post-publication reviews.

  • nLab wiki and The $n$-Category Café, collaborative works on Mathematics, Physics and Philosophy.

  • Polymath Project, a collaborative project that aims to solve important and difficult mathematical problems.

  • Machine Learning Paper Discussions subreddit.

  • Atmospheric Chemistry and Physics journal with interactive public peer review process.

  • CoScience, a service that aims to "recreate scientific communication as a virtuous, open, community-driven process".

  • F1000Research, a platform covering the life science publications.

  • PubPeer, a online platform for post-publication peer review. "The site has served as a whistleblowing platform, in that it highlighted shortcomings in several high-profile papers, in some cases leading to retractions and to accusations of scientific fraud", as Wikipedia says.

  • BibSonomy, a social bookmark service allowing comments.

  • Selected Papers Network was an open-source project for share and comment scientific articles.

I wonder if certain usenet groups in sci.* hierarchy are still active.

In any case, the source code of Papers$^\gamma$ is open under CC0 Public Domain Dedication. And you are welcome to send me a patch or fork it if you wish so.

To conclude, I think that "Hidden knowledge" will always be here, and the solution to this issue lies not in the technical but rather in the societal dimension. Someone just need to write it down in some searchable place. For instance, in enumerative combinatorics Sloane's The On-Line Encyclopedia of Integer Sequences helps a lot, but we need to watch out and constantly update it.

Conclusion update: for me, both OEIS and MathOverflow are popular because their main purpose is to allow people make a research together and not to judge each other.

Paywalled Biblio

[1] Miles, E. “Generalized Fibonacci numbers and associated matrices”. The American Mathematical Monthly, 67(8), 1960, 745–752

[2] Miller, M. D. "On generalized Fibonacci numbers". The American Mathematical Monthly, 78(10), 1971, 1108–1109.

kerzol
  • 335
  • 2
  • 11
  • Hi, about Miller's paper: I've tried to get free access to this paper online but I was unsuccessful. Do you know of any way I can get free access to his paper, or at least the (main) result in his paper? Thanks a lot. – A-Level Student Jun 01 '21 at 12:18
  • It basically has the following theorem proved by "nothing more than elementary theory of equations."

    THEOREM. Let $f(z) =z^k - z^{k-1} - \cdots - z^2 -z-1$, for $k> 1$. Then (a) $f$ has a real zero $z_0$ such that $1 < z_0 < 2$; (b) the remaining $k-1$ zeros of $f$ lie within the unit circle in the complex plane; (c) the zeros of $f$ are simple.

     – kerzol Jun 01 '21 at 14:46
  • Ok, thank you. How is that related to Fibonacci numbers or generalised ones? – A-Level Student Jun 01 '21 at 14:49
  • 1
    @A-LevelStudent When $k=2$, its largest root is the golden ratio, for $k=3$ it is the tribonacci constant, etc. $1/(1-x-x^2)$ is the generating function for Fibonacci numbers; $1/(1-x-x^2-x^3)$ is the generating function for tribonacci numbers, etc. Note that $1-x-x^2- \cdots - x^{k-1} - x^k$ equals the reciprocal of $x^k - x^{k-1} - \cdots - x^2 - x- 1$ – kerzol Jun 01 '21 at 19:40
  • ... and the knowledge about roots of this polynomial helps us study the asymptotic behavior of generalized Fibonacci numbers using the classical method described for instance in Flajolet and Sedgewick books (see https://aofa.cs.princeton.edu/40asymptotic/) – kerzol Jun 01 '21 at 22:01
  • Perhaps the site mentioned in this answer could also be posted as an answer to this older question: Are there any good websites for hosting discussions of mathematical papers? – Martin Sleziak Jun 02 '21 at 10:05
  • Yeah, it is a good idea I think. – kerzol Jun 02 '21 at 13:28
27
  • SelectedPapers was created "as a space where academics could read, share, and give feedback on articles and papers related to their field.". It is no longer in operation.
  • Scirate is an active discussion forum for papers on arXiv, mostly in the area of quantum information processing. It copies the arXiv subcategories in mathematics, so this could well function in that area as well.

Most contributors on these sites disclose their names, but that is not required. Some journals have implemented an open discussion functionality.

  • SciPost is one of these, it welcomes anonymous discussions --- both comments by experts invited as reviewers of a manuscript and also unsollicited comments.
    The physics branch of SciPost is well developed, the mathematics branch has just started with two editors, it now needs more editors who are willing to help set up this branch. (Disclosure, I'm a SciPost Physics editor, feel free to get in touch if this is something you might consider doing.)
  • Perhaps most relevant for MathOverflow, I mention that PhysicsOverflow has implemented a review section, where authors can ask the community to comment on their work. The discussion could be a critical appraisal or just a summary of the paper.
    I do note that, unlike MO, PhysicsOverflow is not part of the StackExchange ecosystem, I don't think such functionality could be implemented as part of SE. I also note that the PhysicsOverflow review functionality has not been a success: Less than 10 reviews per year are entered.
Community
  • 1
Carlo Beenakker
  • 177,695
18

Imho, mathematics has already a good culture and a relatively rich informational ecosystem to detect, document, and disseminate these issues (many options are mentioned above); what might be lacking is a kind of (semi)automated system of interlinked communication that helps spreading the information further.

Judging from the several examples we had to deal with at zbMATH Open (and the discussions involved), I would certainly agree that it seems less feasible to have a single seemingly authoritative source, but we need to employ and coordinate our existing procedures. A natural problem would be, e.g., the scope - there is a number of problematic (mostly, APC) journals around where one has doubts about the proper peer review - would you like to have a website that addresses all of them (including simple Fermat/Goldbach proofs etc.)? Likely, serious mathematicians lack the resources to do that.

But also with restricted scope, every single issue is a tedious work. To give a recent example, even in a rather clear-cut case

Ramm, Alexander G., Solution of the Navier-Stokes problem, Appl. Math. Lett. 87, 160-164 (2019). ZBL1410.35100.

it was impossible to convince the author of the reviewer's arguments (although we employed another expert as arbiter who went through the arguments and clearly ruled in favour of the reviewer). Every such issue is different and needs to be handled carefully.

On the technical side, e.g., if we detect such an issue with a widely accepted older article via zbMATH Open (documented usually as an additional "Editorial remark" in the original) which is lacking an erratum/corrigendum for this, I currently post this also manually as an answer at the standard question at MathOverflow

Widely accepted mathematical results that were later shown to be wrong?

which has the additional advantage that an automated backlink in the respective zbMATH entry is generated (e.g., see the entry

Saito, Masa-Hiko, On the infinitesimal Torelli problem of elliptic surfaces, J. Math. Kyoto Univ. 23, 441-460 (1983). ZBL0532.14019.)

Of course, this doesn't mean everyone is aware of this information, and there are likely various ways to improve/automatize things further. One could think about creating a specific MathOverflow entry/tagging for each issue/publication that needs discussion/updates which could trigger an activity on our and other sites. But the main effort will likely always be mathematically, not technically, and require mathematicians who are willing to explore things deeply. The platforms are there, we should just make sure that the documentation is appropriate, accessible, and ideally increasingly interoperable.

Olaf Teschke
  • 1,114
13

It seems to me that Papers$^\gamma$ is close to what you describe: people can upload a paper, inform its authors, and publicly (anonymously or not) discuss it.

I would also like to mention PCI (Peer Community In), "a free recommendation process of scientific preprints based on peer reviews" that published the discussion between authors, reviewers, and editors (or their equivalent); there is a nice video presentation.

Matthieu Latapy
  • 1,424
8

I'm one of the founders of info.inquire.pub (main site not yet available to the public). We're a non-profit startup with a similar purpose. We aim to be a Q&A site with a reputation system and upvotes, where each question is associated with a specific paper (we support ArXiv or DOI so far).

We are still deciding exactly how anonymous we want to be. We continually discuss potential issues like those @TerryTao brings up in a comment above and are committed to designing a system which will be robust against trolling and weaponisation.

In my opinion, academic research is one of societies greatest resources. We want to improve the state of research. We want to save researchers time and effort. We'd also like to provided a venue for cross-disciplinary review and for the general public to ask high quality objective questions about academic literature. We take possible negative impacts seriously.

The current peer review process is far from perfect and once articles are published they're rarely every edited again. If anyone would like to help us (including simply joining the conversation), my contact info is available at mathandy.ai.

mathandy
  • 279
  • 2
  • 5
  • 2
    I think there is great potential in your project. I'll be closely watching how it develops and I hope you succeed. – user2718 May 31 '21 at 23:45
  • 1
    I had trouble getting to GitHub page where I could post an issue etc from the link you gave on mobile so here is a link to the page where you can post an issue if you need an invite to slack etc https://github.com/community-review/community-review – Peter Gerdes Jun 01 '21 at 17:54
6

I'm 100% in agreement about the need for such a site. In fact I'm currently in the process of coding what I hope can grow into such a site.

The apps mentioned above like Papers$^\gamma$ are all nice but they all just kinda throw the user at a list of papers and say: go comment/review. That can work to collect some objections or point out issues but only if people are visiting the site enough to think that's where the comment goes and it's not that helpful to aggregate knowledge about what's a great paper.

The challenge here is that you need a website which is first and foremost useful and engaging enough for mathematicians to visit it on a regular basis. Something like the question system here (or at least regular professional gossip etc) is one obvious possibility but stack exchange already exists so another possibility is a good system for managing collaboration (so like a scratch pad where you can annotate notes someone else made, version controlled uploads of papers etc)

Furthermore, a good website for this purpose should also have

  1. The ability to rate or at least upvote articles without saying anything. Comments are nice enough for issues but do little to help other people find the best articles.

  2. Some way to feature particular articles that are either hand selected or are well rated by users.

    I'm thinking allowing users with enough karma/trust create a list of articles that they can add to regularly thus creating something like a mini-journal as well as allowing the creation of lists of good references for various subjects.

  3. The ability of authors to claim papers and respond to questions about them (both publicly and privately) as well as offer notes to be displayed alongside the paper/link (eg in Lemma 5 n must be at least k+1 not k as written)

I think it would also be desierable to have some way to verify participants as mathematicians (defined broadly to basically mean anyone who can understand a math paper..but it needs to exclude ppl looking for homework help and cranks) which could be easily done via the system arxiv uses of vouching someone else in or even do it on trust initially.

Anyway in some sense this isn't an answer yet but I think it's important to convey that the existing solutions either lack the features that would aggregate information (thinking academia.edu and researchgate) or lack the features which would enable a community to grow up around the site or both.

I'm going to give writing such a site myself a go because I've wanted it to exist for 15 years and so far all the attempts fail at the generating interest level. But I'm likely to fail down to just a fancy blog so I'm mentioning these aspects in the hope someone else will create a site with them.

Peter Gerdes
  • 2,613