Well, it was wartime, and it wasn't really a secret that physicists would be helping with the war effort (since basically everyone was expected to contribute). Many top scientists took a break from their work at this time to work on things like codebreaking and radar; many of these projects were also top secret at the time. And Feynman mentions (in Surely You're Joking, Mr. Feynman!) that they used obfuscation techniques like having him take a roundabout series of train trips to end up at Los Alamos, so probably it was very non-obvious that many famous scientists were congregating there in particular. They could plausibly have been thought to all be working on separate projects.
Overall my guess would be that people didn't find it all that unusual that top physicists were disappearing from public life, and that it would have been pretty hard to figure out they were all congregating in one place. This doesn't map easily to the AGI question since 1) we're not in the middle of WWII, and 2) modern technology would make it significantly harder to hide the development of such a large-scale project, especially if it took many prominent figures out of the public eye (this is basically what Dr_Manhattan said).