General Anthony Charles Zinni, Former Commander-in-Chief, U.S. Central Command (CINCENT):
80% of what I needed to know as CINCENT I got from open sources rather than classified reporting. And within the remaining 20%, if I knew what to look for, I found another 16%. At the end of it all, classified intelligence provided me, at best, with 4% of my command knowledge
This is an important and underappreciated point, but notice that he's not saying "I could do my job 96% as well -- or even 80% as well -- with nothing but public knowledge".
I strongly disagree with your interpretation. The person quoting him is saying that. Maybe the person quoting him is trying to trick the reader, but I haven't been able to track down the context.
He's definitely NOT describing how well he could do his job.
Imagine trying to play checkers when you couldn't see 13 squares (20.3% of the board), or solving a chess quiz when the contents of 2 squares (3.125%) (chosen by a malicious intelligence) are unknown.
The thing about non-public information is that it is selected for value- but it is selected for value to the opponents of the people keeping it secret. If you aren't the enemy of the people who are concealing information, there's a good chance that the information they are concealing isn't interesting to you.
I think it's likely that insider information is a shortcut to aggregating information from public sources. Ie, you can go to a bunch of effort to find all the public information about what a company is doing or planning to do and get a pretty good answer, but someone working in that company is likely to have that knowledge sorted and organized and known already, simply by virtue of wanting to know it for their job.
Yes. A tremendous amount of what is classified is classified not because no one else can learn these facts or these facts are not out there in an unclassified form somewhere, but rather because it would be highly meaningful if it was known that the CIA or the NSA or Seal Team 6 or whomever was currently batting these particular facts around,.
After all, all unclassified information is ultimately generated or gathered from not intrinsically classified sources.
GJP seems to be proving that, with actual quantified data; excerpting from a recent email I received as a participant:
Phil Tetlock has been recording a “subject matter expert” stream of forecasts, based on his review of all of the data we collect from you PLUS the input of subject-matter-experts whom he consults. [...] The “expert” forecast stream has fared far worse this season, with a current overall Brier score of .31 [...] The Good Judgment Team as a whole is still outperforming the benchmark forecasts against which IARPA judges us. The current overall Brier score for our best aggregate forecast stream is .2554. By comparison, the official “par” (IARPA’s benchmark) has risen to .48. We’re beating “par” by 61%. [...] Several of you have achieved amazing accuracy this season. (Any individual or team whose current Brier score is lower than the .31 for the “expert” stream should feel very proud.)
My current score is .33, fairly close to the "expert" stream score, so with some basic notions and Google it's possible, apparently, to do nearly as well as "experts". (The team I'm on has a so-so aggregate score, only .41 - I assume that the best teams are made up of reasonably good forecasters that are also good at pooling their work.)
Good Judgment Project, see also my recent two-part post.
The study is ongoing. It's an effort to have "normal" people make predictions on world events, assessing individual-, team- and project-level performance as Brier scores.
Tetlock makes this same point with actual data: access to classified information does not improve forecasting.
Does it factor in forecasting that was itself classified?
ETA: The forecast of someone in the Manhattan Project about nuclear bombs would've been much more accurate than almost anyone else's but those forecasts would've stayed entirely in the organization.
A claim that the valuable information just about entirely found by intelligently filtering public sources, in particular about the field of international relations.