Spencer Greenberg (@spencerg) & Belen Cobeta at ClearerThinking.org have a more thorough and well-researched discussion at: Study Report: Is the Dunning-Kruger Effect real? (Also, their slightly-shorter blog post summary.)

This OP would mostly correspond to what ClearerThinking calls “noisy test of skill”. But ClearerThinking also goes through various other statistical artifacts impacting Dunning-Kruger studies, plus some of their own data analysis. Here’s (part of) their upshot:

The simulations above are remarkable because they show that when researchers are careful to avoid "fake" Dunning-Kruger effects, the real patterns that emerge in Dunning-Kruger studies, can typically be reproduced with just two assumptions:

1. Closer-To-The-Average Effect: people predict their skill levels to be closer to the mean skill level than they really are. This could be rational (when people simply have limited evidence about their true skill level), or irrational (if people still do this strongly when they have lots of evidence about their skill, then they are not adjusting their predictions enough based on that evidence).
2. Better-Than-Average Effect: on average, people tend to irrationally predict they are above average at skills. While this does not happen on every skill, it is known to happen for a wide range of skills. This bias is not the same thing as the Dunning-Kruger effect, but it shows up in Dunning-Kruger plots.

Critique about essay structure and communication style:

Walking through statistical issues with papers on the D-K effect is an interesting topic. However, most readers (me included) aren't intimately familiar with the details of this literature. Because of this the format of this essay, which is being tricky over and over again, makes it really hard for me to parse or learn about the D-K effect. This is an important issue, because you're writing a blog post to critique peer reviewed literature, so you have to do extra work to build credibility by convincingly walking us through the flawed reasoning. Because of the "tricky" style of the essay, it simultaneously provokes me to want to understand the issue better and yet makes me not want to use this essay as a resource for building that understanding.

It seems like the essay is possibly meant less to be an explainer and more to string together links and reactions to the literature on this topic. However, it's not obvious that that's the case, and the essay contains a lot of very bold, definitive judgments on what's right and wrong in the literature. These judgments are easy to grasp and remember, but the oblique reasoning process is not. Over time, I've come to really put my guard up against essays like this, because I don't want my brain to get polluted by an aura of confident judgmental rhetoric with no true understanding of the underlying issue.

There's no particular reason I'm flagging the issue on this essay in particular, BTW, but this is my primary reaction to it.

“some people are too dumb to know they’re dumb, and end up thinking they’re smarter than smart people” is just obviously true.

I interpret imprecise colloquial statements like these not as weak claims about existence of such people but as bolder claim that the effect is of practical importance in the challenges faced by many people.

Really interisting post, I learned a lot. I always assumed D-K came from a comparison of top experts and the general population on a super long test.

I agree with your demonstration for the first studies. They were small questionaires and could be explained by chance (not going to calculate the odds). After doing the third test (20 questions on grammar), the bottom quartile predicted they would get 12.9 questions correct and got 9.2. The top quartile predicted 16.9 and got 16.4. This increases the likelihood that it's more than chance, but it still could be that the bottom quartile just realized they didn't know the answers, and that in another sample they would get a better result. Maybe the top quartile included guesses in their estimate and in another sample they would trade places with people on the third quartile who predicted a similar result but had worse guesses.

However, I believe the results of the second phase of study 3 can be extrapolated even if the first phase result is by chance, and it has more important consequences. When grading the tests of 5 representative peers' the bottom quartile did a worse job, consistent with having a worse answer key (their own answers). And not being able to correctly evaluate their peers, they increased the estimate of their own score percentile from p60.5 to p65.4, when it was actually p10.1, even after seeing other people's answers. The top quartile did a better evaluation of their peers' answers and increased self evaluation from p69.5 to p79.7 (actual was p88.7). 

So, I wouldn't say the limitations of the test renders it useless. It appears to overestimate the error in assessment of absolute performance or performance in the topic in general, as you showed. But it independently demonstrated how using only their flawed answers people with wrong conceptions overestimate their relative performance, even after seeing the performance of others. 

An abstraction of the experiment is to give people random answers to 20 questions and different confidence values assigned to each answer. Then give another 5 random test answer sheets (without confidence values in their answers) and ask people to rank the tests. Without knowing a priori which tests are the best performing, I assume it's not possible to rank them with a bad first test and that the results will be similar to the paper's, given a similar distribution of answers. If true, this is a natural consequence of the state, bound to always happen.

Mandatory comment when talking about overconfidence: This seems true to me. Please show my mistakes so I can improve.

1. ^{^}

   "Unskilled and Unaware of it" (Kruger and Dunning, 1999)

Can someone tell me what the actual takeaway is here 

I don't mind a fake out but there were so many fake outs I lost the plot

I suspect there is a strong pro DK bias that is unlikely to change with additional information. The term is commonly assumed as fact and misused to explain individual or subgroup level behavior. Proponents of DK usually assume that peer-ratings (usually by themselves) are better than self-ratings. I find it ironic that people who signal they have knowledge about cognitive biases don’t recognize high ability people can disagree with them. If you look at Google Trends, the term peaked in 2020 during the COVID era. The fake graph continues to dominate the marketplace of image and video content even today, possibly because it's the collectivist version of the word "midwit".

Content-wise, I’d suggest adding residual plots to show what a simulated DK effect would look like. 

I didn't know about the Dunning-Kruger effect. It's interessing. 

It may be considered a bias, but in some sense it is not strictly irrational to be overconfident when you lack knowledge and have no way to measure the immensity of your ignorance. You have no frame of reference, you don't realize that the space of possibilities is vast. I can imagine a Cro-magnon very confident about it's understanding of the world. He listened to the Old Man when he was young, he knows his classics.

It's only when you accumulate knowledge that you begin to realize how ignorant you were before without even noticing, like most people. This experience provides strong evidence that you should update your beliefs in favor of greater caution and humility. This is the essence of Socrates' and Montaigne's famous teachings.