Questions around Making Reliable Evaluations

Most existing forecasting platform questions are for very clearly verifiable questions:

• "Who will win the next election"
• "How many cars will Tesla sell in 2030?"

But many of the questions we care about are much less verifiable:

• "How much value has this organization created?"
• "What is the relative effectiveness of AI safety research vs. bio risk research?"

One solution attempt would be to have an "expert panel" assess these questions, but this opens up a bunch of issues. How could we know how much we could trust this group to be accurate, precise, and understandable?

The topic of, "How can we trust that a person or group can give reasonable answers to abstract questions" is quite generic and abstract, but it's a start.

I've decided to investigate this as part of my overall project on forecasting infrastructure. I've recently been working with Elizabeth on some high-level research.

I believe that this general strand of work could be useful both for forecasting systems and also for the more broad-reaching evaluations that are important in our communities.

Early concrete questions in evaluation quality

One concrete topic that's easily stud

Experimental predictability and generalizability are correlated

A criticism to having people attempt to predict the results of experiments is that this will be near impossible. The idea is that experiments are highly sensitive to parameters and these would need to be deeply understood in order for predictors to have a chance at being more accurate than an uninformed prior. For example, in a psychological survey, it would be important that the predictors knew the specific questions being asked, details about the population being sampled, many details about the experimenters, et cetera.

One counter-argument may not be to say that prediction will be easy in many cases, but rather that if these experiments cannot be predicted in a useful fashion without very substantial amounts of time, then these experiments aren’t probably going to be very useful anyway.

Good scientific experiments produce results are generalizable. For instance, a study on the effectiveness of Malaria on a population should give us useful information (probably for use with forecasting) about the effectiveness on Malaria on other populations. If it doesn’t, then value would be limited. It would really be more of a hist

I was recently pointed to the Youtube channel Psychology in Seattle. I think it's one of my favorites in a while.

I'm personally more interested in workspace psychology than relationship psychology, but my impression is that they share a lot of similarities.

Emotional intelligence gets a bit of a bad rap due to the fuzzy nature, but I'm convinced it's one of the top few things for most people to get better at. I know lots of great researchers and engineers who repeat a bunch of repeated failure modes, and this causes severe organizational and personal problems as a result.

Emotional intelligence books and training typically seem quite poor to me. The alternative format here of "let's just show you dozens of hours of people interacting with each other, and point out all the fixes they could make" seems much better than most books or lectures I've seen.

This Youtube series does an interesting job at that. There's a whole bunch of "let's watch this reality TV show, then give our take on it." I'd be pretty excited about there being more things like this posted online, especially in other contexts.

Related, I think the potential of reality TV is fairly underrated in intellectual circles, but that's a different story.

https://www.youtube.com/user/PsychologyInSeattle?fbclid=IwAR3Ux63X0aBK0CEwc8yPyjsFJ2EKQ2aSMs1XOjUOgaFqlguwz6Fxul2ExJw

Namespace pollution and name collision are two great concepts in computer programming. They way they are handled in many academic environments seems quite naive to me.

Programs can get quite large and thus naming things well is surprisingly important. Many of my code reviews are primarily about coming up with good names for things. In a large codebase, every time symbolicGenerator() is mentioned, it refers to the same exact thing. If after one part of the codebase has been using symbolicGenerator for a reasonable set of functions, and later another part comes up, and it's programmer realizes that symbolicGenerator is also the best name for that piece, they have to make a tough decision. Either they could refactor the codebase to change all previous mentions of symbolicGenerator to use an alternative name, or they have to come up with an alternative name. They can't have it both ways.

Therefore, naming becomes a political process. Names touch many programmers who have different intuitions and preferences. A large refactor of naming in a section of the codebase that others use would often be taken quite hesitantly by that group.

This makes it all the more important that good names are u

I think one idea I'm excited about is the idea that predictions can be made of prediction accuracy. This seems pretty useful to me.

Example

Say there's a forecaster Sophia who's making a bunch of predictions for pay. She uses her predictions to make a meta-prediction of her total prediction-score on a log-loss scoring function (on all predictions except her meta-predictions). She says that she's 90% sure that her total loss score will be between -5 and -12.

The problem is that you probably don't think you can trust Sophia unless she has a lot of experience making similar forecasts.

This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia's seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there's perhaps a simpler solution. If Sophia's provided loss distribution is correct, that would mean that she's calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the directi

Communication should be judged for expected value, not intention (by consequentialists)

TLDR: When trying to understand the value of information, understanding the public interpretations of that information could matter more than understanding the author's intent. When trying to understand the information for other purposes (like, reading a math paper to understand math), this does not apply.

If I were to scream "FIRE!" in a crowded theater, it could cause a lot of damage, even if my intention were completely unrelated. Perhaps I was responding to a devious friend who asked, "Would you like more popcorn? If yes, should 'FIRE!'".

Not all speech is protected by the First Amendment, in part because speech can be used for expected harm.

One common defense of incorrect predictions is to claim that their interpretations weren't their intentions. "When I said that the US would fall if X were elected, I didn't mean it would literally end. I meant more that..." These kinds of statements were discussed at length in Expert Political Judgement.

But this defense rests on the idea that communicators should be judged on intention, rather than expected outcomes. In those cases, it was often clear that

I feel like I've long underappreciated the importance of introspectability in information & prediction systems.

Say you have a system that produces interesting probabilities $p_n$ for various statements. The value that an agent gets from them is not directly correlating to the accuracy of these probabilities, but rather to the expected utility gain they get after using information of these probabilities in corresponding Bayesian-approximating updates. Perhaps more directly, something related to the difference between one's prior and posterior after updated on $p_n$.

Assuming that prediction systems produce varying levels of quality results, agents will need to know more about these predictions to really optimally update accordingly.

A very simple example would be something like a bunch of coin flips. Say there were 5 coins flipped, I see 3 of them, and I want to estimate the number that were heads. A predictor tells me that their prediction has a mean probability of 40% heads. This is useful, but what would be much more useful is a list of which specific coins the predictor saw and what their values were. Then I could get a much more confident answer; possibly a perfect answer.

Financial

The 4th Estate heavily relies on externalities, and that's precarious.

There's a fair bit of discussion of how much of journalism has died with local newspapers, and separately how the proliferation of news past 3 channels has been harmful for discourse.

In both of these cases, the argument seems to be that a particular type of business transaction resulted in tremendous positive national externalities.

It seems to me very precarious to expect that society at large to only work because of a handful of accidental and temporary externalities.

In the longer term

The term for the "fear of truth" is alethophobia. I'm not familiar of many other great terms in this area (curious to hear suggestions).

Apparently "Epistemophobia" is a thing, but that seems quite different; Epistemophobia is more the fear of learning, rather than the fear of facing the truth.

One given definition of alethophobia is, 
"The inability to accept unflattering facts about your nation, religion, culture, ethnic group, or yourself"

This seems like a incredibly common issue, one that is especially talked about as of recent, but without much spec

I keep seeing posts about all the terrible news stories in the news recently. 2020 is a pretty bad year so far.

But the news I've seen people posting typically leaves out most of what's been going on in India, Pakistan, much of the Middle East as of recent, most of Africa, most of South America, and many, many other places as well.

The world is far more complicated than any of us have time to adequately comprehend. One of our greatest challenges is to find ways to handle all this complexity.

The simple solution is to try to spend more time reading the usual n

Do people have precise understandings of the words they use?

On the phrase "How are you?", traditions, mimesis, Chesterton's fence, and their relationships to the definitions of words.

Epistemic status
Boggling. I’m sure this is better explained somewhere in the philosophy of language but I can’t yet find it. Also, this post went in a direction I didn’t originally expect, and I decided it wasn’t worthwhile to polish and post on LessWrong main yet. If you recommend I clean this up and make it an official post, let me know.

One recurrent joke is that one per

I've been reading through some of TVTropes.org and find it pretty interesting. Part of me wishes that Wikipedia were less deletionist, and wonders if there could be a lot more stuff similar to TV Tropes on it.

TVTropes basically has an extensive ontology to categorize most of the important features of games, movies, and sometimes real life. Because games & movies are inspired by real life, even those portions are applicable.

Here are some phrases I think are kind of nice; each that has a bunch of examples in the real world. These are often military relat

It seems inelegant to me that utility functions are created for specific situations, while these clearly aren't the same as that of the agent in total among all of their decisions. For instance, a model may estimate an agent's expected utility from the result of a specific intervention, but this clearly isn't quite right; the agent has a much more complicated utility function outside this intervention. According to a specific model, "Not having an intervention" could set "Utility = 0"; but for any real agent, it's quite likely their life wouldn't actually

Say Tim states, “There is a 20% probability that X will occur”. It’s not obvious to me what that means for Bayesians.

It could mean:

• Tim’s prior is that there’s a 20% chance. (Or his posterior in the context of evidence)
• Tim believes that when the listeners update on him saying there’s a 20% chance (perhaps with him providing insight in his thinking), their posterior will converge to there being a 20% chance.
• Tim believes that the posterior of listeners may not immediately converge to 20%, but the posterior of the enlightened versions of these listeners wo

Perhaps resolving forecasts with expert probabilities can be better than resolving them with the actual events.

The default in literature on prediction markets and decision markets is to expect that resolutions should be real world events instead of probabilistic estimates by experts. For instance, people would predict "What will the GDP of the US in 2025 be?”, and that would be scored using the future “GDP of the US.” Let’s call these empirical resolutions.

These resolutions have a few nice properties:

1. We can expect expect them to be roughly calibrated. (

He's an in-progress hierarchy of what's needed for information to be most useful to an organization or other multi-agent system. I'm sure there must be other very similar hierarchies out there, but don't currently know of any quite like this.

Say you've come up with some cool feature that Apple could include in it's next phone. You think this is a great idea and they should add it in the future.

You're outside of Apple, so the only way you have of interacting with them is by sending information through various channels. The question is: what things should yo

Would anyone here disagree with the statement:

Utilitarians should generally be willing to accept losses of knowledge / epistemics for other resources, conditional on the expected value of the trade being positive.

There's a lot of arguing, of course, on if humans are rational, but this often mixes up two things: there's the "Von Neumann-Morgenstern utility function maximization" definition of "rational", and there's a hypothetical "rational" that a human could fulfill with constraints much more complicated than the classical approach, more in the direction of prospect theory, or Predictive Coding.

I think I regard the second definition as sufficiently not understood or defined that it isn't yet worth using in most conversation. It seems challenging, to say the least,

Instillation, Proliferation, Amplification

Paul Christiano and Ought use the terminology of Distillation and Amplification to describe a high-level algorithm of one type of AI reasoning.

I’ve wanted to come up with an analogy to forecasting systems. I previously named a related concept Prediction-Augmented Evaluation Systems, one somewhat renamed to “Amplification” by Jacobjacob in this post.

I think one thing that’s going on is that “distillation” doesn’t have an exact equivalent with forecasting setups. The term “distillation” comes with the assumptions:

1. T

Agent-based modeling seems like one obvious step forward to me for much of social-science related academic progress. OpenAI's Hide and Seek experiment was one that I am excited about, but it is very simple and I imagine similar work could be greatly extended for other fields. The combination of simulation, possible ML distillation on simulation (to make it run much faster), and effective learning algorithms for agents, seems very powerful.

However, agent-based modeling still seems quite infrequently used within Academia. My impression is that agent-based so

Named Footnotes: A (likely) mediocre proposal

Epistemic status: This is probably a bad idea, because it's quite obvious yet not done; i.e. Chesterson's fence.

One bad practice in programming is to have a lot of unnamed parameters. For instance,

  createPost(author, post, comment, name, id, privacyOption, ...)

Instead it's generally better to used Named Parameters, like,

  createPost({author, post, comment, name, id, privacyOption})

Footnotes/endnotes seem similar. They are ordered by number, but this can be quite messy. It's particularly annoying for autho

It seems really hard to deceive a Bayesian agent who thinks you may be deceiving them, especially in a repeated game. I would guess there could be interesting theorems about Bayesian agents that are attempting to deceive one another; as in, in many cases their ability to deceive the other would be highly bounded or zero, especially if they were in a flexible setting with possible pre-commitment devices.

To give a simple example, agent A may tell agent B that they believe $\omega =0.8$, even though they internally believe $\omega =0.5$. However, if this were somewhat repeat