If I recall correctly, your education has taken place at particularly elite universities, which would tend to indicate that you've spent an atypically large amount of time interacting with people of very high intelligence. This may explain some of the disconnect between your impressions and those of other folks.

It's interesting to think about what this might mean, by thinking like a positivist.

Most people are stupid, relative to Old One, and most people are smart, relative to lichen. We need a standard reference point in order to assign a truth value.

But the only salient reference point for the speaker of that sentence is... the speaker of that sentence. Hence, the opinion reduces to:

"I am smarter than most people."

Now, that may be true about the speaker. But when everybody agrees about that... then we're living at Lake Wobegon, "where all the children are above average." Only now they're all below average (except for me, of course).

To jump off of what komponisto said, I had a similar reaction when I was at Yale also. Now that I'm a grad student at a less prestigious university I see a lot of dumb undergrads as well as dumb people in some other departments. It seems that the smartest kids at BU are as smart as the smartest Yalies. But there are a lot more dumb kids and the dumbest undergrads are much dumber than the dumbest Yalies.

Scott Adams, on being an idiot:

I proudly include myself in the idiot category. Idiocy in the modern age isn’t an all-encompassing, twenty-four-hour situation for most people. It’s a condition that everybody slips into many times a day. Life is just too complicated to be smart all the time.

The other day I brought my pager to the repair center because it wouldn’t work after I changed the battery. The repairman took the pager out of my hand, flipped open the battery door, turned the battery around, and handed the now functional pager back to me in one well-practiced motion. This took much of the joy out of my righteous indignation over the quality of their product. But the repairman seemed quite amused. And so did every other customer in the lobby.

On that day, in that situation, I was a complete idiot. Yet somehow I managed to operate a motor vehicle to the repair shop and back. It is a wondrous human characteristic to be able to slip into and out of idiocy many times a day without noticing the change or accidentally killing innocent bystanders in the process.

Ruling out a large class of predictable answers is part of it, but it's also arguably more a case of "impressions don't point anywhere" than "impressions point the other way". If challenged on that point I'd have to think about whether I could make it more precisely, but I was kind of hoping people would intuitively agree about the spirit of the question (which is why I didn't include the second paragraph at first).

I think you should add this rule to the post itself, in case the post receives many comments.