I don't know, but I had a similar experience with chattGPT; after having heard from various sources that it is pretty decent at chess, I decided to challenge it to a friendly game. I was actually expecting to lose since I myself am not much of a chess player, and also having had conversations on topics where I am an expert, depending on your definition of that word, and being really impressed by the reasoning ability of the LLM in some of those conversations. Anyway, chattGPT insisted on playing as white. Then it said: "Ok, let's begin. You get the first move." So I had to explain to it that according to the rules of chess, it's white that moves first. So as usual it excused itself, and the game began. After each move, it printed an image of the table on the screen; I did not pay attention to that since I had my own physical board. (We were exchanging moves by stating the start square and the end square of each move, i.e. not the standard terminology for chess.) So a couple of moves in I noticed that the robot moved one of its Bishops right through one of its pawns. So I remarked that this is not a legal move. The robot agreed, and stated that it is not allowed to capture its own pawns. It made another move, it was my turn, I stated the moved but this time I also looked at the board that the robot displayed on the screen. I realized that something was very wrong, I had a knight where I should have a bishop. So I went back to trace where this error had started. But in the end it became to complicated and we decided to start a fresh game. This game also lasted a couple of moves before strange things started to happen. So we agreed to start fresh again, and now I insisted on playing as white (I was getting tired of having my physical board "upsidedown" compared to the board on screen). Anyway, the robot managed a couple of moves again, before it placed one of it's rooks outside of the board. Now I was tired so I just said "ok I give up, you're too good at this" (I don't want to offend it, maybe it will spare me in a human zoo when it takes over). But jokes asides, why is it so inconsistent? Maybe openAI have dumbed it down not to scare people; maybe it was investigating how I would react; or maybe it just had a bad day, and if I take the time and play it again it will be much better. What I mean by the last option is that all these games took place within the same chatt, so the LLM remembers things from earlier, so maybe, having had this bad start, and knowing that it is often the case that it runs the whole chatt up until the current point through the NN to get the next word (next "token" to be more precise), it might be that since less and less made any sense it went astray farther and farther from giving good moves.
So as for your search on Bing "Powered by chattGPT" maybe try to do the search again and see if you can get a better result. And also, maybe try searching using different words than last time, or trying to explain a little better, try pretending like you're talking to a human unless you did not do that last time.
Thanks for this post, and in particular for including Grothendieck as one of the examples to illustrate your ideas. I have thought that most people outside of mathematics, and even many that are studying math, are not familiar with him. So I like how you assume that such a reader will accept this and just start to read, in the first section, about a guy that is unknown to said theoretical reader. I think that shows respect for your audience.
There is one thing not related to your points but more to the work of Grothendieck that I would like to mention; your statement "It is his capacity to surface questions that set Grothendieck apart, more so than his capacity to answer them." (which I now took out of context, but still): I don't agree completely with this. For example, to a person not familiar with his work it sounds like as if he basically only came up with new questions. But equally important is that he was also able envision and to implement a solution to many of them. His SGA-seminar that ran for 9 years at IHES laid the foundation for what became modern algebraic geometry, and being a new theory it required a huge amount of ground work carried out during this seminar, large parts of which was done by Grothendieck himself. He then published the more foundational parts of this research, including the theory of schemes, in EGA, which is written mainly by him, with assistants from Dieudonné. Without this to me absurd amount of work on Grothendiecks part there would likely be nothing. At the time there was a need to generalize the concept of a variety to something defined over first any field and then also over any (commutative) ring, in particular the integers. Grothendieck's generalization is what is called a scheme, and in modern terminology a variety is a special type of scheme. So this being in the air at the time, someone else would have come up with some version of this theory. But the version that Grothendieck envisioned was also the one that became standard, and I doubt that that would have happened if Grothendieck himself did not actually put in the crazy amount of work that it required to go from this beautiful idea to a (more or less) finished theory.
Moreover, it is really fascinating now to be able to read an English translation of "Récoltes..." that you so kindly provided links to. (Back when I was into this subject I could not find any translation and so I had not read it before.) In there he talks about his motivation for developing algebraic geometry over the integers, namely the Weil conjectures (already proved for curves by André Weil himself); these conjectures relate geometric properties (topology) of the the complex points of a variety to arithmetic properties of the defining polynomial equations of the variety (the behavior of the integer solutions to these equations). A simpler to state example of another conjecture relating topology to arithmetic, proved later by Faltings using Grothendieck's framework, is that there can only be a finite number of rational points on an algebraic curve of genus > 1. Here the genus is an invariant of the curve that becomes visible when we view the curve over the complex numbers. So the simplest example is the projective line, which can be defined by the equation x^2 + y^2 = 1 together with a point at infinity; viewed over the complex numbers this looks like a sphere (another name for this object is the Riemann sphere); it has genus 0 since there are no "holes" in it. (I should say that it is a curve due to it's algebraic dimension being 1; and if you look at its points with real coordinates you will see a curve (a circle); now we look at the points where the coordinates are allowed to be complex numbers, and then it will look like a sphere, hence a surface, this can potentially be a little confusing.) Next, an elliptic curve looks like a torus when viewed over the complex numbers, and so has it has one hole, hence genus 1. And if we look at a given curve over the complex numbers and it turns out that it has even more holes, then its genus will be greater than 1. So the genus is a topological invariant of the curve that we can see if we view the curve over the complex numbers. Now, the equation above has infinitely many solutions that are rational numbers; the number of rational points on an elliptic curve can vary and there is a whole theory for describing their structure; and what Faltings proved is that if a curve has genus >1 it can only have a finite number of rational points. For example, the projective curve defined by x^4+y^4=1 has genus 3, so by Faltings theorem it can only have a finite number of rational points, or in other words, the equation can only have a finite number of solutions where both x and y are rational numbers. (But for this particular equation it was proved already by Fermat that it only have the trivial rational solutions, or stated differently, the equation x^4+y^4=z^4 has only the trivial integer solutions where at least one of the coordinates are 0. As I'm sure everybody knows, Fermat also claimed to have a proof that this was true for all exponents larger than 2, but that wasn't proved until the 1990:s by Andrew Wiles, again using the framework laid out by Grothendieck as a foundation.) In short: The Weil-conjectures indicated a link between the topology (form) of the points with complex coordinates on a variety, and the integer solutions of the equations that defines the variety (arithmetic). Grothendieck envisioned a method to make use of this link that would first require him to generalize the definition of a variety to something that could exist over the integers. That generalization is what is known as a scheme.
So before Grothendieck it was only possible to talk about varieties as something that is defined over an algebraically closed field like the complex numbers. That's why I said "arithmetic properties of the defining equations" earlier. But from developing the theory of schemes, which can be define even over the integers while at the same time has classical varieties as a special type of scheme, it becomes possible to replace "arithmetic properties of the defining equations" with "properties of the rational points on the variety", thereby opening up the field of Arithmetic geometry.
So back to "Récoltes...", Grothendieck talks about how he investigated this new undiscovered land; in this land, the Weil-conjectures was the capital, always visible at the horizon. But Grothendieck was exploring the whole country, down to the smallest cottage out in some distant province. I have to interpret this as that he was proving theorem after theorem that was needed for the foundation of his new theory but not directly related to the Weil-conjectures. In the end, Grothendieck and collaborators was able to prove 3 of the 4 Weil-conjectures using this newly developed theory. (The first of them had also been proved earlier using other methods). Then in 1970 Grothendieck left his position at IHES, to my understanding because they received funds from the military. Some years later, his student Deligné managed to prove the last of the Weil-conjectures, using the methods of Grothendieck, but not in the way that Grothendieck had envisioned (he wanted to do it by proving the so called "Standard conjectures", from which he already had showed that the last part of the Weil-conjectures would follow). He also left research mathematics for many years, but came back briefly in the 1980's with "Esquisse d'un Programme", including his "Dessins d'enfants" that can be used to describe Riemann surfaces, more precisely exactly those Riemann surfaces that arises as the complex points on an algebraic curve defined over the algebraic numbers. (Above we had two examples of such curves, the projective line and the Fermat-curve of genus 3). So in this program he actually surfaced the questions, whereas in the development of algebraic geometry there was already a need for a new framework, although your point that he was able to surface the right questions are valid also there. Anyway, this last research program is active to this day, but Grothendieck left it after a couple of years, during which he worked on the program alone. So in this case he was no longer able to put in the extreme work required to go from a question and idea to finished results. And I think this was my point (although it took a little more words than expected to convey it); that what really made Grothendieck so outstanding was not only ability to surface questions and his seminal ideas and visions, but equally much his ability to implement those ideas. (And to do that a second time was too much to ask; I assume that you are already familiar with what happened to Grothendieck later in his life, but if someone happens to read this and are not, I encourage you to read more about his life; in a way this is as close to the stereotypical mad genius that you can get, and it feels like a sad story, although that is not for me to judge but more up to Grothendieck himself when he was still alive.)