Loring W. Tu actually drew the same conclusion from Von Neumann feeling wanting (emphasis mine):

One of Raoul’s observations on life has played a crucial role in my mental equilibrium. When he was at the Institute for Advanced Study at Princeton in 1949–51, he once had a conversation with John von Neumann, a fellow Hungarian who was at the time a professor at the institute. Von Neumann told Raoul that he had known only one great mathematician, David Hilbert, and that having been a prodigy in his youth, he never felt that he had lived up to his promise. Raoul wrote in ([4], p. 270), “So you see, it is not difficult to be found wanting — one just needs an appropriate measuring rod.” If even von Neumann felt inadequate in his achievement in comparison with Hilbert’s, what chance for professional satisfaction do we ordinary mortals have? After Raoul recounted this incident to me, I resolved never to compare myself with anyone else, especially not with my friends and classmates who have achieved greatness.

from Remembering Raoul Bott (1923–2005)