# ozziegooen's Shortform

New Comment
Some comments are truncated due to high volume. Change truncation settings

# Questions around Making Reliable Evaluations

Most existing forecasting platform questions are for very clearly verifiable questions:

• "Who will win the next election"
• "How many cars will Tesla sell in 2030?"

But many of the questions we care about are much less verifiable:

• "How much value has this organization created?"
• "What is the relative effectiveness of AI safety research vs. bio risk research?"

One solution attempt would be to have an "expert panel" assess these questions, but this opens up a bunch of issues. How could we know how much we could trust this group to be accurate, precise, and understandable?

The topic of, "How can we trust that a person or group can give reasonable answers to abstract questions" is quite generic and abstract, but it's a start.

I've decided to investigate this as part of my overall project on forecasting infrastructure. I've recently been working with Elizabeth on some high-level research.

I believe that this general strand of work could be useful both for forecasting systems and also for the more broad-reaching evaluations that are important in our communities.

## Early concrete questions in evaluation quality

One concrete topic that's easily stud

2romeostevensit1y> "How much value has this organization created?" can insights from prediction markets work for helping us select better proxies and decision criteria or do we expect people to be too poorly entangled with the truth of these matters for that to work? Do orgs always require someone who is managing the ontology and incentives to be super competent at that to do well? De facto improvements here are worth billions (project management tools, slack, email add ons for assisting managing etc.)
8ozziegooen1yI think that prediction markets can help us select better proxies, but the initial set up (at least) will require people pretty clever with ontologies. For example, say a group comes up with 20 proposals for specific ways of answering the question, "How much value has this organization created?". A prediction market could predict the outcome of the effectiveness of each proposal. I'd hope that over time people would put together lists of "best" techniques to formalize questions like this, so doing it for many new situations would be quite straightforward.
6mr-hire1yAnother related idea we played around with, but which didn't make it into the final whitepaper: What if we just assumed that Brier score was also predictive of good judgement. Then, people, could create a distribution over several measures of "how good will this organization do" and we could use standard probability theory and aggregation tools to create an aggregated final measure.
4mr-hire1yThe way we handled this with Verity was to pick a series of values, like "good judgement", "integrity," "consistency" etc. Then the community would select exemplars who they thought represented those values the best. As people voted on which proposals they liked best, we would weight their votes by: 1. How much other people (weighted by their own score on that value) thought they had that value. 2. How similarly they voted to the examplars. This sort of "value judgement" allows for fuzzy representation of high level judgement, and is a great supplement to more objective metrics like Brier score which can only measure well defined questions. Eigentrust++ is a great algorithm that has the properties needed for this judgement-based reputation. The Verity Whitepaper [http://verity.site/Verity-Whitepaper-Final.pdf] goes more into depth as to how this would be used in practice.
2romeostevensit1yDeference networks seem underrated.
2jimrandomh1yOne way to look at this is, where is the variance coming from? Any particular forecasting question has implied sub-questions, which the predictor needs to divide their attention between. For example, given the question "How much value has this organization created?", a predictor might spend their time comparing the organization to others in its reference class, or they might spend time modeling the judges and whether they tend to give numbers that are higher or lower. Evaluation consistency is a way of reducing the amount of resources that you need to spend modeling the judges, by providing a standard that you can calibrate against. But there are other ways of achieving the same effect. For example, if you have people predict the ratio of value produced between two organizations, then if the judges consistently predict high or predict low, this no longer matters since it affects both equally.
2ozziegooen1yYep, good points. Ideally one could do a proper or even estimated error analysis [https://www.webassign.net/question_assets/unccolphysmechl1/measurements/manual.html] of some kind. Having good units (like, ratios) seems pretty important.
-2zulupineapple1yIf you had a precise definition of "effectiveness" this shouldn't be a problem. E.g. if you had predictions for "will humans go extinct in the next 100 years?" and "will we go extinct in the next 100 years, if we invest 1M into AI risk research?" and "will we go extinct, if we invest 1M in bio risk research?", then you should be able to make decisions with that. And these questions should work fine in existing forecasting platforms. Their long term and conditional nature are problems, of course, but I don't think that can be helped. That's not a forecast. But if you asked "How much value will this organization create next year?" along with a clear measure of "value", then again, I don't see much of a problem. And, although clearly defining value can be tedious (and prone to errors), I don't think that problem can be avoided. Different people value different things, that can't be helped. Why would you do that? What's wrong with the usual prediction markets? Of course, they're expensive (require many participants), but I don't think a group of experts can be made to work well without a market-like mechanism. Is your project about making such markets more efficient?
4ozziegooen1yComing up with a precise definition is difficult, especially if you want multiple groups to agree. Those specific questions are relatively low-level; I think we should ask a bunch of questions like that, but think we may also want some more vague things as well. For example, say I wanted to know how good/enjoyable a specific movie would be. Predicting the ratings according to movie reviewers (evaluators) is an approach I'd regard as reasonable. I'm not sure what a precise definition for movie quality would look like (though I would be interested in proposals), but am generally happy enough with movie reviews for what I'm looking for. Agreed that that itself isn't a forecast, I meant in the more general case, for questions like, "How much value will this organization create next year" (as you pointed out). I probably should have used that more specific example, apologies. Can you be more explicit about your definition of "clearly"? I'd imagine that almost any proposal at a value function would have some vagueness. Certificates of Impact get around this by just leaving that for the review of some eventual judges, kind of similar to what I'm proposing. The goal for this research isn't fixing something with prediction markets, but just finding more useful things for them to predict. If we had expert panels that agreed to evaluate things in the future (for instance, they are responsible for deciding on the "value organization X has created" in 2025), then prediction markets and similar could predict what they would say.
1zulupineapple1yMy point is that "goodness" is not a thing in the territory. At best it is a label for a set of specific measures (ratings, revenue, awards, etc). In that case, why not just work with those specific measures? Vague questions have the benefit of being short and easy to remember, but beyond that I see only problems. Motivated agents will do their best to interpret the vagueness in a way that suits them. Is your goal to find a method to generate specific interpretations and procedures of measurement for vague properties like this one? Like a Shelling point for formalizing language? Why do you feel that can be done in a useful way? I'm asking for an intuition pump. Certainly there is some vagueness, but it seems that we manage to live with it. I'm not proposing anything that prediction markets aren't already doing.
2ozziegooen1yHm... At this point I don't feel like I have a good intuition for what you find intuitive. I could give more examples, but don't expect they would convince you much right now if the others haven't helped. I plan to eventually write more about this, and eventually hopefully we should have working examples up (where people are predicting things). Hopefully things should make more sense to you then. Short comments back<>forth are a pretty messy communication medium for such work.
1Tetraspace Grouping1yThere's something of a problem with sensitivity; if the x-risk from AI is ~0.1, and the difference in x-risk from some grant is ~10^-6, then any difference in the forecasts is going to be completely swamped by noise. (while people in the market could fix any inconsistency between the predictions, they would only be able to look forward to 0.001% returns over the next century)
2ozziegooen1yYea, in cases like these, having intermediate metrics seems pretty essential.
1zulupineapple1yMaking long term predictions is hard. That's a fundamental problem. Having proxies can be convenient, but it's not going to tell you anything you don't already know.

Namespace pollution and name collision are two great concepts in computer programming. They way they are handled in many academic environments seems quite naive to me.

Programs can get quite large and thus naming things well is surprisingly important. Many of my code reviews are primarily about coming up with good names for things. In a large codebase, every time symbolicGenerator() is mentioned, it refers to the same exact thing. If after one part of the codebase has been using symbolicGenerator for a reasonable set of functions, and later another part comes up, and it's programmer realizes that symbolicGenerator is also the best name for that piece, they have to make a tough decision. Either they could refactor the codebase to change all previous mentions of symbolicGenerator to use an alternative name, or they have to come up with an alternative name. They can't have it both ways.

Therefore, naming becomes a political process. Names touch many programmers who have different intuitions and preferences. A large refactor of naming in a section of the codebase that others use would often be taken quite hesitantly by that group.

This makes it all the more important that good names are u

4G Gordon Worley III1yI'm not sure if it's good or bad, but I find the way species get named interesting. The general rule is "first published name wins", and this is true even if the first published name is "wrong" in some way, like implies a relationship that doesn't exist, since that implication is not officially semantically meaningful. But there are ways to get around this, like if a name was based on a disproved phylogeny, in which case a new name can be taken up that fits the new phylogenic relationship. This means existing names get to stick, at least up until the time that they are proven so wrong that they must be replaced. Alas, there's no official registry of these things, so it's up to working researchers to do literature reviews and get the names right, and sometimes people get it wrong by accident and sometimes on purpose because they think an earlier naming is "invalid" for one reason or another and so only recognize a later naming. The result is pretty confusing and requires knowing a lot or doing a lot of research to realize that, for example, two species names might refer to the same species in different papers.
2ozziegooen1yThanks, I didn't know. That matches what I expect from similar fields, though it is a bit disheartening. There's an entire field of library science [https://en.wikipedia.org/wiki/Library_science] and taxonomy, but they seem rather isolated to specific things.
4ozziegooen1yAnother quick note on the LessWrong wiki: I'm skeptical of single definitions without disclaimers. I think it's misleading (to some) that "Truth is the correspondence between and one's beliefs about reality and reality. "[1]. Rather, it's fair to say that this is one specific definition of truth that has been used in many cases; I'm sure that others, including others on LessWrong, have used it differently. Most dictionaries have multiple definitions for words. This seems more like what we should aim for. In fairness, when I searched for "Rationality", the result [https://wiki.lesswrong.com/wiki/Rationality] states, "Rationality is something of fundamental importance to LessWrong that is defined in many ways", which I of course agree with. [1] https://wiki.lesswrong.com/wiki/Truth [https://wiki.lesswrong.com/wiki/Truth]
3Pattern1yAt the meta-level it isn't clear what value other definitions might offer (in this case). ("Truth" seems like a basic concept that is understood prior to reading that article - it's easier to imagine such an argument for other concepts without such wide understanding.) Perhaps more definitions should be brought in (as necessary), with the same level of attention to detail - when they are used (extensively). It's possible that relevant posts have already been made, they just haven't been integrated into the wiki. Is the wiki up to date as of 2012, but not after that?
3Pattern1yFootnote not found. The refactoring sounds like a good idea, though the main difficulty would be propagating the new names.
3ozziegooen1yThanks for point that out! I forgot the specific note, removed the [1]. I definitely would agree that refactoring would be difficult, especially if we haven't figured out a great refactoring process.
2mr-hire1yOne of the issues with this in both an academic and LW context is that changing the name of something in a single source of truth codebase is much cheaper than changing the name of something in a community. The more popular an idea, the more cost goes up to change the name. Similarly, when you're working with a single organization, creating a process that everyone follows is relatively cheap compared to a loosely tied together community with various blogs, individuals, and organizations coining their own terms.
3ozziegooen1yYep, I'd definitely agree that it's harder. That said, this doesn't mean that it's not high-ev to improve on. One outcome could be that we should be more careful introducing names, as it is difficult to change them. Another would be to work to attempt to have formal ways of changing them after, even though it is difficult (It would be worthwhile in some cases, I assume).
6mr-hire1yIn a recent thread about changing the name of Solstice to Solstice Advent, Oliver Habryka estimated it would cost at least $100,000 to make that happen. This seems like a reasonable estimate to me, and a good lower bound for how much value you could get from a name change to make it worth it The idea of lowering this cost is quite appealing, but I'm not sure how to make a significant difference there. I think it's also worth thinking about the counterfactual cost of discouraging naming things. As an example, here's a post with an important concept that hasn't really spread because it doesn't have a snappy name: https://www.lesswrong.com/posts/K4eDzqS2rbcBDsCLZ/unrolling-social-metacognition-three-levels-of-meta-are-not [https://www.lesswrong.com/posts/K4eDzqS2rbcBDsCLZ/unrolling-social-metacognition-three-levels-of-meta-are-not] I was recently pointed to the Youtube channel Psychology in Seattle. I think it's one of my favorites in a while. I'm personally more interested in workspace psychology than relationship psychology, but my impression is that they share a lot of similarities. Emotional intelligence gets a bit of a bad rap due to the fuzzy nature, but I'm convinced it's one of the top few things for most people to get better at. I know lots of great researchers and engineers who repeat a bunch of repeated failure modes, and this causes severe organizational and personal problems as a result. Emotional intelligence books and training typically seem quite poor to me. The alternative format here of "let's just show you dozens of hours of people interacting with each other, and point out all the fixes they could make" seems much better than most books or lectures I've seen. This Youtube series does an interesting job at that. There's a whole bunch of "let's watch this reality TV show, then give our take on it." I'd be pretty excited about there being more things like this posted online, especially in other contexts. Related, I think the potential of reality TV is fairly underrated in intellectual circles, but that's a different story. https://www.youtube.com/user/PsychologyInSeattle?fbclid=IwAR3Ux63X0aBK0CEwc8yPyjsFJ2EKQ2aSMs1XOjUOgaFqlguwz6Fxul2ExJw 6G Gordon Worley III5moOne of the things I love about entertainment is that much of it offers evidence about how humans behave in a wide variety of scenarios. This has gotten truer over time, at least within Anglophone media, with its trend towards realism and away from archetypes and morality plays. Yes, it's not the best possible or most reliable evidence about how real humans behave in real situations and it's a meme around here that you should be careful not to generalize from fictional evidence [https://www.lesswrong.com/posts/rHBdcHGLJ7KvLJQPk/the-logical-fallacy-of-generalization-from-fictional] , but I also think it's better than nothing (I don't think reality TV is especially less fictional than other forms of entertainment with regards to how human behave, given its heavy use of editing and loose scripting to punch up situations for entertainment value). 1Rudi C5moNothing is a low bar though. :) 4mr-hire5moYou might also enjoy the channel "charisma on command" which has a similar format of finding youtube videos of charismatic and non-charismatic people, and seeing what they do and don't do well. 2ozziegooen5moThanks! I'll check it out. 4romeostevensit5moNovel and obviously some good ideas/directions. Thanks. ## Experimental predictability and generalizability are correlated A criticism to having people attempt to predict the results of experiments is that this will be near impossible. The idea is that experiments are highly sensitive to parameters and these would need to be deeply understood in order for predictors to have a chance at being more accurate than an uninformed prior. For example, in a psychological survey, it would be important that the predictors knew the specific questions being asked, details about the population being sampled, many details about the experimenters, et cetera. One counter-argument may not be to say that prediction will be easy in many cases, but rather that if these experiments cannot be predicted in a useful fashion without very substantial amounts of time, then these experiments aren’t probably going to be very useful anyway. Good scientific experiments produce results are generalizable. For instance, a study on the effectiveness of Malaria on a population should give us useful information (probably for use with forecasting) about the effectiveness on Malaria on other populations. If it doesn’t, then value would be limited. It would really be more of a hist ... (read more) 1NunoSempere1yIn that paragraph, did you mean to say "findings_i is correct"? *** Neat idea. I'm also not sure whether the idea is valuable because it could be implementable, or from "this is interesting because it gets us better models". In the first case, I'm not sure whether the correlation is strong enough to change any decisions. That is, I'm having trouble thinking of decisions for which I need to know the generalizability of something, and my best shot is measuring its predictability. For example, in small foretold/metaculus communities, I'd imagine that miscellaneous factors like "is this question interesting enough to the top 10% of forecasters" will just make the path predictability -> differential entropy -> generalizability difficult to detect. 2ozziegooen1yThe main point I was getting at is that the phrases: 1. Experiments are important to perform. 2. Predictors cannot decently predict the results of experiments unless they have gigantic amounts of time. Are a bit contradictory. You can choose either, but probably not both. Likewise, I'd expect that experiments that are easier to predict are ones that are more useful, which is more convenient than the other alternative. I think generally we will want to estimate importance/generality of experiments separate from their predictability. I think one idea I'm excited about is the idea that predictions can be made of prediction accuracy. This seems pretty useful to me. ## Example Say there's a forecaster Sophia who's making a bunch of predictions for pay. She uses her predictions to make a meta-prediction of her total prediction-score on a log-loss scoring function (on all predictions except her meta-predictions). She says that she's 90% sure that her total loss score will be between -5 and -12. The problem is that you probably don't think you can trust Sophia unless she has a lot of experience making similar forecasts. This is somewhat solved if you have a forecaster that you trust that can make a prediction based on Sophia's seeming ability and honesty. The naive thing would be for that forecaster to predict their own distribution of the log-loss of Sophia, but there's perhaps a simpler solution. If Sophia's provided loss distribution is correct, that would mean that she's calibrated in this dimension (basically, this is very similar to general forecast calibration). The trusted forecaster could forecast the adjustment made to her term, instead of forecasting the same distribution. Generally this would be in the directi ... (read more) 2jacobjacob1yI think this is equivalent to applying a non-linear transformation to your proper scoring rule. When things settle, you get paid S(p) both based on the outcome of your object-level prediction p, and your meta prediction q(S(p)). Hence: S(p)+B(q(S(p))) where B is the "betting scoring function". This means getting the scoring rules to work while preserving properness will be tricky (though not necessarily impossible). One mechanism that might help is that if each player makes one object prediction p and one meta prediction q, but for resolution you randomly sample one and only one of the two to actually pay out. 4ozziegooen1yInteresting, thanks! Yea, agreed it's not proper. Coming up with interesting payment / betting structures for "package-of-forecast" combinations seems pretty great to me. 8jacobjacob1yI think this paper might be relevant: https://users.cs.duke.edu/~conitzer/predictionWINE09.pdf [https://users.cs.duke.edu/~conitzer/predictionWINE09.pdf] 1NunoSempere1yIs it actually true that forecasters would find it easier to forecast the adjustment? 2ozziegooen1yOne nice thing about adjustments is that they can be applied to many forecasts. Like, I can estimate the adjustment for someone's [list of 500 forecasts] without having to look at each one. Over time, I assume that there would be heuristics for adjustments, like, "Oh, people of this reference class typically get a +20% adjustment", similar to margins of error in engineering. That said, these are my assumptions, I'm not sure what forecasters will find to be the best in practice. I feel like I've long underappreciated the importance of introspectability in information & prediction systems. Say you have a system that produces interesting probabilities for various statements. The value that an agent gets from them is not directly correlating to the accuracy of these probabilities, but rather to the expected utility gain they get after using information of these probabilities in corresponding Bayesian-approximating updates. Perhaps more directly, something related to the difference between one's prior and posterior after updated on . Assuming that prediction systems produce varying levels of quality results, agents will need to know more about these predictions to really optimally update accordingly. A very simple example would be something like a bunch of coin flips. Say there were 5 coins flipped, I see 3 of them, and I want to estimate the number that were heads. A predictor tells me that their prediction has a mean probability of 40% heads. This is useful, but what would be much more useful is a list of which specific coins the predictor saw and what their values were. Then I could get a much more confident answer; possibly a perfect answer. Financial ... (read more) 5jacobjacob1yIn some sense, markets have a particular built-in interpretability: for any trade, someone made that trade, and so there is at least one person who can explain it. And any larger market move is just a combination of such smaller trades. This is different from things like the huge recommender algorithms running YouTube, where it is not the case that for each recommendation, there is someone who understands that recommendation. However, the above argument fails in more nuanced cases: * Just because for every trade there's someone who can explain it, doesn't mean that there is a particular single person who can explain all trades * Some trades might be made by black-box algorithms * There can be weird "beauty contest" dynamics where two people do something only because the other person did it 7ozziegooen1yGood point, though I think the "more nuanced cases" are very common cases. The 2010 flash crash [https://en.wikipedia.org/wiki/2010_Flash_Crash] seems relevant; it seems like it was caused by chaotic feedback loops with algorithmic components, that as a whole, are very difficult to understand. While that example was particularly algorithmic-induced, other examples also could come from very complex combinations of trades between many players, and when one agent attempts to debug what happened, most of the traders won't even be available or willing to explain their parts. The 2007-2008 crisis [https://en.wikipedia.org/wiki/Financial_crisis_of_2007%E2%80%9308] may have been simpler, but even that has 14 listed causes on Wikipedia and still seems hotly debated. In comparison, YouTube I think algorithms may be even simpler, though they are still quite messy. Charity investigators could be time-effective by optimizing non-cause-neutral donations. There are a lot more non-EA donors than EA donors. It may also be the case that EA donation research is somewhat saturated. Say you think that$1 donated to the best climate change intervention is worth 1/10th that of $1 for the best AI-safety intervention. But you also think that your work could increase the efficiency of$10mil of AI donations by 0.5%, but it could instead increase the efficiency of \$50mil of climate change donations by 10%. Then, for you to maximize expected value, your time is best spent optimizing the climate change interventions.

The weird thing here may be in explaining this to the donors. "Yea, I'm spending my career researching climate change interventions, but my guess is that all these funders are 10x less effective than they would be by donating to other things." While this may feel strange, both sides would benefit; the funders and the analysts would both be maximizing their goals.

Separately, there's a second plus that teaching funders to be effectiveness-focused; it's possible that this will eventually lead some of them to optimize further.

I think this may be the c

## More Narrow Models of Credences

Epistemic Rigor
I'm sure this has been discussed elsewhere, including on LessWrong. I haven't spent much time investigating other thoughts on these specific lines. Links appreciated!

The current model of a classically rational agent assume logical omniscience and precomputed credences over all possible statements.

This is really, really bizarre upon inspection.

First, "logical omniscience" is very difficult, as has been discussed (The Logical Induction paper goes into this).

Second, all possible statements include statements of all complexity classes that we know of (from my understanding of complexity theory). "Credences over all possible statements" would easily include uncountable infinities of credences. One could clarify that even arbitrarily large amounts of computation would not be able to hold all of these credences.

Precomputation for things like this is typically a poor strategy, for this reason. The often-better strategy is to compute things on-demand.

A nicer definition could be something like:

A credence is the result of an [arbitrarily large] amount of computation being performed using a reasonable inference engine.

It should be quite clear

Communication should be judged for expected value, not intention (by consequentialists)

TLDR: When trying to understand the value of information, understanding the public interpretations of that information could matter more than understanding the author's intent. When trying to understand the information for other purposes (like, reading a math paper to understand math), this does not apply.

If I were to scream "FIRE!" in a crowded theater, it could cause a lot of damage, even if my intention were completely unrelated. Perhaps I was responding to a devious friend who asked, "Would you like more popcorn? If yes, should 'FIRE!'".

Not all speech is protected by the First Amendment, in part because speech can be used for expected harm.

One common defense of incorrect predictions is to claim that their interpretations weren't their intentions. "When I said that the US would fall if X were elected, I didn't mean it would literally end. I meant more that..." These kinds of statements were discussed at length in Expert Political Judgement.

But this defense rests on the idea that communicators should be judged on intention, rather than expected outcomes. In those cases, it was often clear that

It seems like there are a few distinct kinds of questions here.

1. You are trying to estimate the EV of a document.
Here you want to understand the expected and actual interpretation of the document. The intention only matters to how it effects the interpretations.

2. You are trying to understand the document.
Example: You're reading a book on probability to understand probability.
Here the main thing to understand is probably the author intent. Understanding the interpretations and misinterpretations of others is mainly useful so that you can understand the intent better.

3. You are trying to decide if you (or someone else) should read the work of an author.
Here you would ideally understand the correctness of the interpretations of the document, rather than that of the intention. Why? Because you will also be interpreting it, and are likely somewhere in the range of people who have interpreted it. For example, if you are told, "This book is apparently pretty interesting, but every single person who has attempted to read it, besides one, apparently couldn't get anywhere with it after spending many months trying", or worse, "This author is actually quite clever, but the vast majority of people who read their work misunderstand it in profound ways", you should probably not make an attempt; unless you are highly confident that you are much better than the mentioned readers.

2ozziegooen10moOne nice thing about cases where the interpretations matter, is that the interpretations are often easier to measure than intent (at least for public figures). Authors can hide or lie about their intent or just never choose to reveal it. Interpretations can be measured using surveys.
4Dagon10moSeems reasonable. It also seems reasonable to predict others' future actions based on BOTH someone's intentions and their ability to understand consequences. You may not be able to separate these - after the third time someone yells "FIRE" and runs away, you don't really know or care if they're trying to cause trouble or if they're just mistaken about the results.
4ozziegooen10moRelated, there seems to be a decent deal of academic literature on intention vs. interpretation in Art, though maybe less in news and media. https://www.iep.utm.edu/artinter/#H1 [https://www.iep.utm.edu/artinter/#H1] https://en.wikipedia.org/wiki/Authorial_intent [https://en.wikipedia.org/wiki/Authorial_intent] Some other semi-related links: https://foundational-research.org/should-we-base-moral-judgments-on-intentions-or-outcomes/ [https://foundational-research.org/should-we-base-moral-judgments-on-intentions-or-outcomes/] https://en.wikipedia.org/wiki/Intention_(criminal_law) [https://en.wikipedia.org/wiki/Intention_(criminal_law)] https://en.wikipedia.org/wiki/Negligence [https://en.wikipedia.org/wiki/Negligence] https://en.wikipedia.org/wiki/Recklessness_(law) [https://en.wikipedia.org/wiki/Recklessness_(law)]

I think the thing I find the most surprising about Expert Systems is that people expected them to work so early on, and apparently they did work in some circumstances. Some issues:

1. The user interfaces, from what I can tell, were often exceedingly mediocre. User interfaces are difficult to do well and difficult to specify, so are hard to guarantee quality in large and expensive projects. It was also significantly harder to make good UIs back when expert systems were more popular, than it is today.
2. From what I can tell, many didn't even have notions of unce

## The 4th Estate heavily relies on externalities, and that's precarious.

There's a fair bit of discussion of how much of journalism has died with local newspapers, and separately how the proliferation of news past 3 channels has been harmful for discourse.

In both of these cases, the argument seems to be that a particular type of business transaction resulted in tremendous positive national externalities.

It seems to me very precarious to expect that society at large to only work because of a handful of accidental and temporary externalities.

In the longer term

3Dagon5moIt seems to me very arrogant and naive to expect that society at large could possibly work without the myriad of evolved and evolving externalities we call "culture". Only a tiny part of human interaction is legible, and only a fraction of THAT is actually legislated.
4ozziegooen5moFair point. I imagine when we are planning for where to aim things though, we can expect to get better at quantifying these things (over the next few hundred years), and also aim for strategies that would broadly work without assuming precarious externalities.
4Dagon5moIndeed. Additionally, we can hope to get better over the coming centuries (presuming we survive) at scaling our empathy, and the externalities can be internalized by actually caring about the impact, rather than (better: in addition to) imposition of mechanisms by force.
2Pattern5moThis seems accurate - but just observation itself is valuable.

## Do people have precise understandings of the words they use?

On the phrase "How are you?", traditions, mimesis, Chesterton's fence, and their relationships to the definitions of words.

Epistemic status
Boggling. I’m sure this is better explained somewhere in the philosophy of language but I can’t yet find it. Also, this post went in a direction I didn’t originally expect, and I decided it wasn’t worthwhile to polish and post on LessWrong main yet. If you recommend I clean this up and make it an official post, let me know.

One recurrent joke is that one per

2ozziegooen10moUpdate: After I wrote this shortform, I did more investigation in Pragmatics and realized most of this was better expressed there.
3Raemon10moWhat's Pragmatics in this case?
2ozziegooen3moAh, sorry for not responding earlier. By Pragmatics I meant Pragmatics in linguistics. It studies what people mean when they say words. https://plato.stanford.edu/entries/pragmatics/

It seems inelegant to me that utility functions are created for specific situations, while these clearly aren't the same as that of the agent in total among all of their decisions. For instance, a model may estimate an agent's expected utility from the result of a specific intervention, but this clearly isn't quite right; the agent has a much more complicated utility function outside this intervention. According to a specific model, "Not having an intervention" could set "Utility = 0"; but for any real agent, it's quite likely their life wouldn't actually

5ozziegooen1yRelated to this, one common argument against utility maximization is that "we still cannot precisely measure utility". But here, it's perhaps more clear that we don't need to. What's important for decision making is that we have models that we can expect will help us maximize our true utility functions, even if we really don't know much about what they really are.
4Stuart_Armstrong1yI delve into that here: https://www.lesswrong.com/posts/Lb3xCRW9usoXJy9M2/platonic-rewards-reward-features-and-rewards-as-information#Extending_the_problem [https://www.lesswrong.com/posts/Lb3xCRW9usoXJy9M2/platonic-rewards-reward-features-and-rewards-as-information#Extending_the_problem]
4ozziegooen1yOh fantastic, thanks for the reference!
3Pattern1y^U and ^U look to be the same.
5ozziegooen1yThanks! Fixed. I'm sure the bottom notation could be improved, but am not sure the best way. In general I'm trying to get better at this kind of mathematics.
2Pattern1yYou got the basic idea across, which is a big deal. Though whether it's A or B isn't clear: A) "this isn't all of the utility function, but its everything that's relevant to making decisions about this right now". ^U doesn't have to be U, or even a good approximation in every situation - just (good enough) in the situations we use it. Building a building? A desire for things to not fall on people's heads becomes relevant (and knowledge of how to do that). Writing a program that writes programs? It'd be nice if it didn't produce malware. Both desires usually exist - and usually aren't relevant. Models of utility for most situations won't include them. B) The cost of computing the utility function more exactly in the case exceeds the (expected) gains. isn't clear.
3ozziegooen1yI think I agree with you. There's a lot of messiness with using ^U and often I'm sure that this approximation leads to decision errors in many real cases. I'd also agree that better approximations of ^U would be costly and are often not worth the effort. Similar to how there's a term for "Expected value of perfect information", there could be an equivalent for the expected value of a utility function, even outside of uncertainty of parameterized that were thought to be included. Really, there could be calculations for "expected benefit from improvements to a model", though of course this would be difficult to parameterize (how would you declare that a model has been changed a lot vs. a little? If I introduce 2 new parameters, but these parameters aren't that important, then how big of a deal should this be considered in expectation?)
3Pattern1yThe model has changed when the decisions it is used to make change. If the model 'reverses' and suggests doing the opposite/something different in every case from what it previously recommended, then it has 'completely changed'. (This might be roughly the McNamara fallacy, of declaring that things that 'can't be measured' aren't important.) EDIT: Also, if there's a set of information consisting of a bunch of pieces, A, B, and C, and incorporating all but one of them doesn't have a big impact on the model, but the last piece does, whichever piece that is, 'this metric' could lead to overestimating the importance of whichever piece happened to be last, when it's A, B, and C together that made an impact. It 'has this issue' because the metric by itself is meant to notice 'changes in the model over time', not figure out why/solve attribution.

I've been trying to scurry academic fields for discussions of how agents optimally reduce their expected error for various estimands (parameters to estimate). This seems like a really natural thing to me (the main reason why we choose some ways of predictions over others), but the literature seems kind of thin from what I can tell.

The main areas I've found have been Statistical Learning Theory and Bayesian Decision / Estimation Theory. However, Statistical Learning Theory seems to be pretty tied to Machine Learning, and Bayesian Decision / Estimation Theor

The term for the "fear of truth" is alethophobia. I'm not familiar of many other great terms in this area (curious to hear suggestions).

Apparently "Epistemophobia" is a thing, but that seems quite different; Epistemophobia is more the fear of learning, rather than the fear of facing the truth.

One given definition of alethophobia is,
"The inability to accept unflattering facts about your nation, religion, culture, ethnic group, or yourself"

This seems like a incredibly common issue, one that is especially talked about as of recent, but without much spec

6Dagon5moWithout looking, I'll predict it's a neologism - invented in the last ~25 years to apply to someone the inventor disagrees with. Yup, google n-grams shows 0 uses in any indexed book. A little more searching shows almost no actual uses anywhere - mostly automated dictionary sites that steal from each other, presumably with some that accept user submissions. I did find one claim to invention, in 2017: http://www.danielpipes.org/comments/236053 [http://www.danielpipes.org/comments/236053] . Oh, and earlier (2008), there's a book with that title: https://www.amazon.com/Alethophobia-Manoucher-Parvin/dp/1588140474 [https://www.amazon.com/Alethophobia-Manoucher-Parvin/dp/1588140474] . I still submit that this is a word in search of a need, which mostly exists as a schoolyard insult dressed up in Latin.
2ozziegooen5moYea, I also found the claim, as well as a few results from old books before the claim. The name come straight from the Latin though, so isn't that original or surprising. Just because it hasn't been used much before doesn't mean we can't start to use it and adjust the definition as we see fit. I want to see more and better terminology in this area.

> The name comes straight from the Latin though

From the Greek as it happens. Also, alethephobia would be a double negative, with a-letheia meaning a state of not being hidden; a more natural neologism would avoid that double negative. Also, the greek concept of truth has some differences to our own conceptualization. Bad neologism.

3ozziegooen4moAh, good to know. Do you have recommendations for other words?
4ChristianKl4moThe trend of calling things that aren't fears "-phobia" seems to me a trend that's harmful for clear communication. Adjusting the definition only leads to more confusion.
2Dagon5moI think I want less terminology in this area, and generally more words and longer descriptions for things that need nuance and understanding. Dressing up insults as diagnoses doesn't help any goals I understand, and jargon should only be introduced as part of much longer analyses where it illuminates rather than obscures.
2mr-hire5moCan you give examples of things you think would fit under this? It seems that there are lots of instances of being resistant to the truth, but I can think of very few that I would categorize as fear of truth. It's often fear of something else (e.g. fear of changing your identity) or biases (e.g. the halo effect or consistency bias) that cause people to resist. I can think of very few cases where people have a general fear of truth.

I keep seeing posts about all the terrible news stories in the news recently. 2020 is a pretty bad year so far.

But the news I've seen people posting typically leaves out most of what's been going on in India, Pakistan, much of the Middle East as of recent, most of Africa, most of South America, and many, many other places as well.

The world is far more complicated than any of us have time to adequately comprehend. One of our greatest challenges is to find ways to handle all this complexity.

The simple solution is to try to spend more time reading the usual n

7Dagon5moThere's a parallel development that makes any strategy difficult - it's financially rewarding to misdirect and mis-aggregate the big-picture publications and comparisons. So you can't spend less time on details, as you can't trust the aggregates without checking. See also Gell-Mann Amnesia. Without an infrastructure for divvying up the work to trusted cells that can understand (parts of) the details and act as a check on the aggregators and each other, the only answer is to spot-check details yourself, and accept ignorance of things you didn't verify.
3Rudi C4moI feel most people, including myself, don't even use the aggregators already available. For example, there are lots of indices and statistics (ideally there should be much more, but anyways), but I rarely go out of my way to consume them. Some examples I just thought of: * https://rsf.org/en/ranking [https://rsf.org/en/ranking] * https://www.globalhungerindex.org/results.html [https://www.globalhungerindex.org/results.html] * Keeping a watch on new entries to and exits from Fortune 500 * Looking at the stocks of the top 20 companies every quarter * ... There are several popular books that throw surprising statistics around, like Factfulness; This suggests a lot of us are disconnected from basic statistics, that we presumably could easily get by just googling.

Intervention dominance arguments for consequentialists

## Global Health

There's a fair bit of resistance to long-term interventions from people focused on global poverty, but there are a few distinct things going on here. One is that there could be a disagreement on the use of discount rates for moral reasoning, a second is that the long-term interventions are much more strange.

No matter which is chosen, however, I think that the idea of "donate as much as you can per year to global health interventions" seems unlikely to be ideal upon clever thinking.

For the

I've been reading through some of TVTropes.org and find it pretty interesting. Part of me wishes that Wikipedia were less deletionist, and wonders if there could be a lot more stuff similar to TV Tropes on it.

TVTropes basically has an extensive ontology to categorize most of the important features of games, movies, and sometimes real life. Because games & movies are inspired by real life, even those portions are applicable.

Here are some phrases I think are kind of nice; each that has a bunch of examples in the real world. These are often military relat

6mr-hire1yA think I want: A recommendation engine that works based on listing the tropes you enjoy.
3Pattern1yThis article expresses the same sentiment, and may include links to what that looked like, and where it went: https://www.gwern.net/In-Defense-Of-Inclusionism [https://www.gwern.net/In-Defense-Of-Inclusionism]

There's a lot of arguing, of course, on if humans are rational, but this often mixes up two things: there's the "Von Neumann-Morgenstern utility function maximization" definition of "rational", and there's a hypothetical "rational" that a human could fulfill with constraints much more complicated than the classical approach, more in the direction of prospect theory, or Predictive Coding.

I think I regard the second definition as sufficiently not understood or defined that it isn't yet worth using in most conversation. It seems challenging, to say the least,

3Pattern1yOr it could be an intuitive usage and mean "(more) optimal". "Why don't more people do [thing that will improve their health]?"
2ozziegooen1yI like that question. I think that if people were to try to define optimal in a specific way, they would find that it requires a model of human behavior; the common one that academics would fall back to is that of Von Neumann-Morgenstern utility function maximization. I think it's quite possible that when we have better models of human behavior, we'll better recognize that in cases where people seem to be doing silly things to improve their health, they're actually being somewhat optimal given a large sets of physical and mental constraints.

Say Tim states, “There is a 20% probability that X will occur”. It’s not obvious to me what that means for Bayesians.

It could mean:

• Tim’s prior is that there’s a 20% chance. (Or his posterior in the context of evidence)
• Tim believes that when the listeners update on him saying there’s a 20% chance (perhaps with him providing insight in his thinking), their posterior will converge to there being a 20% chance.
• Tim believes that the posterior of listeners may not immediately converge to 20%, but the posterior of the enlightened versions of these listeners wo
5crabman1yIt's definitely the first. The second is bizarre. The third can be steelmanned as "Given my evidence, an ideal thinker would estimate the probability to be 20%, and we all here have approximately the same evidence, so we all should have 20% probabilities", which is almost the same as the first.
4ozziegooen1yI don't think it's only the first. It seems weird to me imagine telling to a group that "There's a 20% probability that X will occur" if I really have little idea and would guess many of them would have a better sense than me. I would only personally feel comfortable doing this if I was quite sure my information was quite a bit better than theirs. Else, I'd say something like, "I personally think there's a 20% chance, but I really don't have much information."
5ChristianKl1yI take it to mean that if Tim is acting optimally and has to take a bet on the outcome 1:4 would be the point where both sides of the bad are equally profitable to him while if the odds deviate from 1:4 one side of the bet would be preferable to him.
2ozziegooen1yOne thing this wouldn't take into account is strength or weight of evidence. If Tim knew that all of the listeners had far more information than him, and thus probably could produce better estimates of X, then it seems strange for Tim to tell them that the chances are 20%. I guess my claim that saying “There is a 20% probability that X will occur” is more similar to: "I'm quite confident that the chances are 20%, and you should generally be too" than it is to, "I personally believe that the chances are 20%, but have no idea o how much that should update the rest of you."
4ozziegooen1yI think my current best guess to this is something like: When humans say thing X, they don't mean the literal translation of X, but rather are pointing to X', which is a specific symbol that other humans generally understand. For instance, "How are you" is a greeting, not typically a literal question. [How Are You] can be thought of as a symbol that's very different than the sum of it's parts. That said, I find it quite interesting that the basics of human use of language seem to be relatively poorly understood; in the sense that I'd expect many people to disagree on what they think “There is a 20% probability that X will occur” means, even after using it with each other in a setting that assumes some amount of understanding.
2Tetraspace Grouping1yOther things that Tim might mean when he says 20%: * Tim is being dishonest, and believes that the listeners will update away from the radical and low-status figure of 20% to avoid being associated with the lowly Tim. * Tim believes that other listeners will be encouraged to make their own probability estimates with explicit reasoning in response, which will make their expertise more legible to Tim and other listeners. * Tim wants to show cultural allegiance with the Superforecasting tribe.

## Perhaps resolving forecasts with expert probabilities can be better than resolving them with the actual events.

The default in literature on prediction markets and decision markets is to expect that resolutions should be real world events instead of probabilistic estimates by experts. For instance, people would predict "What will the GDP of the US in 2025 be?”, and that would be scored using the future “GDP of the US.” Let’s call these empirical resolutions.

These resolutions have a few nice properties:

1. We can expect expect them to be roughly calibrated. (
5NunoSempere1yHere is another point by @jacobjacob, which I'm copying here in order for it not to be lost in the mists of time:
4ChristianKl1yYou can train experts to be calibrated in different ways. If you train experts to be calibrated to pick the right probability on GPOpen where probability is done in steps on 1, I don't think those experts will be automatically calibrated to distinguish a p=0.00004 event from a p=0.00008. Experts would actually need to be calibrated on getting probabilities inside the tail right. I don't think we know how to do calibration training for that tail.
2ozziegooen1yI think this could be a good example for what I'm getting at. I think there are definitely some people in some situations who can distinguish a p=0.00004 event from a p=0.00008 event. How? By making a Fermi model or similar. A trivial example would be a lottery with calculable odds of success. Just because the odds are low doesn't mean they can't be precisely estimated. I expect that the kinds of problems that GPOpen would consider asking AND are incredibly unlikely, would be difficult to estimate within 1 order of magnitude. But may still be able to do a decent job, especially in cases where you can make neat Fermi models. However, of course, it seems very silly to use the incentive mechanism "you'll get paid once we know for sure if the event happened" on such an event. Instead, if resolutions are done with evaluators, then there is much more of a signal.
5mr-hire1yI'm fairly skeptical of this. From a conceptual perspective, we expect the tails to be dominated by unknown unknowns and black swans. Fermi estimates and other modelling tools are much better at estimating scenarios that we expect. Whereas, if we find ourselves in the extreme tails, its often because of events or factors that we failed to model.
2ozziegooen1yI'm not sure. The reasons things happen at the tails typically fall into categories that could be organized to be a small set. For instance: * The question wasn't understood correctly. * A significant exogenous event happened. But, as we do a bunch of estimates, we could get empirical data about these possibilities, and estimate the potentials for future tails. This is a bit different to what I was mentioning, which was more about known but small risks. For instance, the "amount of time I spend on my report next week" may be an outlier if I die. But the chance of serious accident or death can be estimated decently well enough. These are often repeated known knowns.
2ChristianKl1yYou might have people who can distinguish those, but I think it's a mistake to speak of calibration in that sense as the word usually refers to people who actually trained to be calibrated via feedback.
3NunoSempere1yAnother point in favor of such a set-up would be that aspiring superforecasters get much, much more information when they see ~[the prediction of a superforecaster would have made having their information]; a point vs a distribution. I'd expect that this means that market participants would get better, faster.
2ozziegooen1yYep, this way would basically be much more information-dense, with all the benefits that comes from that.
3Pattern1ySo you don't want predictions*, you want models**. Robust/fully fleshed out models. *predictions of events **predictions of which model is correct
3ozziegooen1yI'm not sure I'd say that in the context of this post, but more generally, models are really useful. Predictions that come with useful models are a lot more useful than raw predictions. I wrote this other [https://www.lesswrong.com/posts/Df2uFGKtLWR7jDr5w/ozziegooen-s-shortform?commentId=3JoFTh6My4eX3E37p] post about a similar topic. For this specific post, I think what we're trying to get is the best prediction we could have had using data pre-event.
2mr-hire1yI'm really interested in this type of scheme because it would also solve a big problem in futarchy and futarchy-like setups that use prediction polling, namely, the inability to score conditional counterfactuals (which is most of the forecasting you'll be doing in Futarchy-like setup). One thing you could do instead of scoring people against expert assesments is also potentially score people against the final aggregate and extremized distribution. One issue with any framework like this is that general calibration may be very different than calibration at the tails. Whatever scoring rule you're using to determine calibration of experts or aggregate scoring has the same issue that long tail events rarely happen. Another solution to this problem (although it doesn't solve the counterfactual conditional problem) is to create tailored scoring rules that provide extra rewards for events at the tails. If an event at the tails is a million times less likely to happen, but you care about it equally to events at the center, then provide a million times reward for accuracy near the tail in the event it happens. Prior work on tailored scoring rules for different utility functions here: https://www.evernote.com/l/AAhVczys0ddF3qbfGk_s4KLweJm0kUloG7k/ [https://www.evernote.com/l/AAhVczys0ddF3qbfGk_s4KLweJm0kUloG7k/]
2ozziegooen1yGood points! Also, thanks for the link, that's pretty neat. I think that an efficient use of expert assessments would be for them to see the aggregate, and then basically adjust that as is necessary, but to try to not do much original research. I just wrote a more recent shortform post about this. I think that we can get calibration to be as good as experts can figure out, and that could be enough to be really useful.

He's an in-progress hierarchy of what's needed for information to be most useful to an organization or other multi-agent system. I'm sure there must be other very similar hierarchies out there, but don't currently know of any quite like this.

Say you've come up with some cool feature that Apple could include in it's next phone. You think this is a great idea and they should add it in the future.

You're outside of Apple, so the only way you have of interacting with them is by sending information through various channels. The question is: what things should yo

4ozziegooen1yAnother note to this; there are cases where a system is both broken and fixable at the step-3 level. In some of these cases, it could be worth it to fix the system there instead, especially if you may want to make similar changes in the future. For instance, you may have an obvious improvement for your city to make. You may then realize that the current setups to suggest feedback are really difficult to use, but that it's actually quite feasible to make sure some changes happen that will make all kinds of useful feedback easier for the city to incorporate.

I think brain-in-jar or head-in-jar are pretty underrated. By this I mean separating the head from the body and keeping it alive with other tooling. Maybe we could have a few large blood processing plants for many heads, and the heads could be connected to nerve I/O that would be more efficient than finger -> keyboard IO. This seems fairly easier than uploading, and possibly doable in 30-50 years.

I can't find much about how difficult it is. It's obviously quite hard and will require significant medical advances, but it's not clear just how many are need... (read more)

4avturchin12dYes. But the head also ages and could have terminal diseases: cancer, stroke, ALZ. Given the steep nature of the Gompertz law, the life expectancy of even a perfect head in jar (of an old man) will be less than 10 years (I guess). So it is not immortality, but a good way to wait until better life extension technologies.
2ozziegooen11dI was thinking of it less for life extension, and more for a quality of life and cost improvement.

Western culture is known for being individualistic instead of collectivist. It's often assumed (with evidence) that individualistic cultures tend to be more truth seeking than collectivist ones, and that this is a major advantage.

But theoretically, there could be highly truth seeking collectivist cultures. One could argue that Bridgewater is a good example here.

In terms of collective welfare, I'm not sure if there are many advantages to individualism besides the truth seeking. A truth seeking collectivist culture seems pretty great to me, in theory.

2MakoYass3moI suspect there's a limit on how good at truthseeking individualism can make people. Good information is a commons, its sum value is greater the more it is shared, it is not funded in proportion to its potential, under economies of atomized decisions. We need a political theory of due deference to expertise. Wherever experts fail, or the wrong experts are appointed, or where a layperson on the ground stops believing that experts are even identifiable, there is work to be done.

Would anyone here disagree with the statement:

Utilitarians should generally be willing to accept losses of knowledge / epistemics for other resources, conditional on the expected value of the trade being positive.

4Dagon1y[ not a utilitarian; discount my opinion appropriately ] This hits one of the thorniest problems with Utilitarianism: different value-over-time expectations depending on timescales and assumptions. If one is thinking truly long-term, it's hard to imagine what resource is more valuable than knowledge and epistemics. I guess tradeoffs in WHICH knowledge to gain/lose have to be made, but that's an in-category comparison, not a cross-category one. Oh, and trading it away to prevent total annihilation of all thinking/feeling beings is probably right.
2ozziegooen1yI think my thinking is that for utilitarians, these are generally instrumental, not terminal values. Often they're pretty important instrumental values, but this still would mean that they could be traded off in respect to the terminal values. Of course, if they are "highly important" instrumental values, then something very large would have to be offered for a trade to be worth it. (total annihilation being one example)
2Dagon1yI think we're agreed that resources, including knowledge, are instrumental (though as a human, I don't always distinguish very closely). My point was that for very-long-term terminal values, knowledge and accuracy of evaluation (epistemics) are far more important than almost anything else. It may be that there's a declining marginal value for knowledge, as there is for most resources, and once you know enough to confidently make the tradeoffs, you should do so. But if you're uncertain, go for the knowledge.
3edoarad10moNon-Bayesian Utilitarian that are ambiguity averse [https://www.lesswrong.com/posts/iDYaCJ3o3Q7ypriTF/knightian-uncertainty-and-ambiguity-aversion-motivation] sometimes need to sacrifice "expected utility" to gain more certainty (in quotes because that need not be well defined).
3AprilSR1yDoesn't being willing to accept a trade *directly follow* from the expected value of the trade being positive? Isn't that like, the *definition* of when you should be willing to accept a trade? The only disagreement would be how likely it is that losses of knowledge / epistemics are involved in positive value trades. (My guess is it does happen rarely.)
2ozziegooen1yI'd generally say that, but wouldn't be surprised if there were some who disagreed; who's argument would be something like what-to-me would sound like a modification of utilitarianism, [utilitarianism+epistemic-terminal-values].
1AprilSR1yIf you have epistemic terminal values then it would not be a positive expected value trade, would it? Unless "expected value" is referring to the expected value of something other than your utility function, in which case it should've been specified.
2ozziegooen10moYep, I would generally think so. I was doing what may be a poor steelman of my assumptions of how others would disagree; I don't have a great sense of what people who would disagree would say at this point.
1Pattern1yHappiness + Knowledge. (A related question is, do people with these values drink?)
2[anonymous]1yOnly if the trade is voluntary. If the trade is forced (e.g. in healthcare) then you may have two bad options, and the option you do want is not on the table.
3Isnasene1yIn general, I would agree with the above statement (and technically speaking, I have made such trade-offs). But I do want to point out that it's important to consider what the loss of knowledge/epistemics entails. This is because certain epistemic sacrifices have minimal costs (I'm very confident that giving up FDT for CDT for the next 24 hours won't affect me at all) and some have unbounded costs (if giving up materialism causes me to abandon cryonics, it's hard to quantify how large of a blunder that would be). This is especially true of epistemics that allow to you be unboundedly exploited by an adversarial agent. As a result, even when the absolute value looks positive to me, I'll still try to avoid this kinds of trade-offs because certain black swans (ie bumping into an adversarial agent that exploits your lack of knowledge about something) make such bets very high risk.
2ozziegooen1yThis sounds pretty reasonable to me; it sounds like you're basically trying to maximize expected value, but don't always trust your initial intuitions, which seems quite reasonable.
3Pattern1y[What "utilitarian" means could use some resolving, so I just treated this as "people".] I would disagree. I tried to find the relevant post in the sequences and found this [https://www.lesswrong.com/posts/PkDn96XQZDHPLcreY/aella-on-rationality-and-the-void] along with it: Would I accept that processes that take into account resource constraints might be more effective? Certainly, thought I think of that as 'starting the journey in a reasonable fashion' rather than 'going backwards' as your statement brings to mind.
3George1yHow would you define loss of knowledge ?
2ozziegooen1yBasically, information that can be handled in "value of information" style calculations. So, if I learn information such that my accuracy of understanding the world increases, my knowledge is increased. For instance, if I learn the names of everyone in my extended family.
1George1yOk, but in this case do you mean "loss of knowledge" as in "loss of knowledge harbored within the brain" or "loss of knowledge no matter where it's stored, be it a book, brain, text file... etc" ? Further more, does losing copies of a certain piece of knowledge count as loss of knowledge ? What about translations of said knowledge (in another language or another philosophical/mathematical framework) that doesn't add any new information, just makes it accessible to a larger demographic ?
2ozziegooen1yI was thinking the former, but I guess the latter could also be relevant/count. It seems like there's no strict cut-off. I'd expect a utilitarian to accept trade-offs against all these kinds of knowledge, conditional on the total expected value being positive.
1George1yWell, the problem with the former (knowledge harbored within the brain) is that it's very vague and hard to define. If I have, say, a method to improve the efficacy of VX (an easily weaponizable nerve toxin). As a utilitarian I conclude this information is going to be harmful, I can purge it of my hard-drive, I can burn the papers I used to come up with this... etc. But I can't wipe my head clean of the information, at best I can resign to never talk about it to anyone and to not accord it much import, such that I may forget it. But that's not destruction per-say, it's closer to lying, not sharing the information with anyone (even if asked specifically), or to biasing your brain towards transmitting and remembering certain pieces of information (which we do all the time). However I don't see anything contentious with this case, nor with any other case of information-destruction, as long as it is for the greater utility. I think in general people don't advocate for destroying/forgetting information because: a) It's hard to do b) As a general rule of thumb the accumulation of information seems to be a good thing, even if the utility of a specific piece of information is not obvious But this is more of a heuristic, an exact principle.
2ozziegooen1yI'd agree that the first one is generally pretty separated from common reality, but think it's a useful thought experiment. I was originally thinking of this more in terms of "removing useful information" than "removing expected-harmful information", but good point; the latter could be interesting too.
2George1yWell,I think the "removing useful information" bit contradicts with utility to being with. As in, if you are a utilitarian, useful information == helps maximize utility. Thus the trade-off is not possible. I can think of some contrived examples where the trade-off is possible (e.g. where the information is harmful now but will be useful later), but in that case it's so easy to "hide" information in the modern age, instead of destroying it entirely, that the problem seem too theoretical to me. But at the end of the day, assuming you reached a contrived enough situation where the information must either be destroyed (or where hiding it devoid other people of the ability to discover further useful information), I think the utilitarian perspective has nothing fundamental against destroying it. However, no matter how hard I try, I can't really think of a very relevant example where this could be the case.
3ozziegooen1yOne extreme case would be committing suicide because your secret is that important. A less extreme case may be being OK with forgetting information; you're losing value, but the cost to maintain it wouldn't be worth it. (In this case the information is positive though)
2ozziegooen1yThere's some related academic work around this here: https://www.princeton.edu/~tkelly/papers/epistemicasinstrumental.pdf [https://www.princeton.edu/~tkelly/papers/epistemicasinstrumental.pdf] https://core.ac.uk/download/pdf/33752524.pdf [https://core.ac.uk/download/pdf/33752524.pdf] They don't specifically focus on utilitarians, but the arguments are still relevant. Also, this post is relevant: https://www.lesswrong.com/posts/dMzALgLJk4JiPjSBg/epistemic-vs-instrumental-rationality-approximations [https://www.lesswrong.com/posts/dMzALgLJk4JiPjSBg/epistemic-vs-instrumental-rationality-approximations]

Agent-based modeling seems like one obvious step forward to me for much of social-science related academic progress. OpenAI's Hide and Seek experiment was one that I am excited about, but it is very simple and I imagine similar work could be greatly extended for other fields. The combination of simulation, possible ML distillation on simulation (to make it run much faster), and effective learning algorithms for agents, seems very powerful.

However, agent-based modeling still seems quite infrequently used within Academia. My impression is that agent-based so

5johnswentworth1yCould you give a few specific examples where you imagine agent-based models would help?
8ozziegooen1ySure, 1. Humans as agents / psychology / economics. Instead of making mathematical models of rational agents, have people write code that predicts the behaviors of rational agents or humans. Test the "human bots" against empirical experimental results of humans in different situations, to demonstrate that the code accurately models human behavior. 2. Mechanism design. Show that according to different incentive structures, humans will behave differently, and use this to optimize the incentive structures accordingly. 3. Most social science. Make agent-based models to generally help explain how groups of humans interact with each other and what collective behaviors emerge. I guess when I said, "Much of academic progress"; I should have specified, "Academic fields that deal with modeling humans to some degree"; perhaps most of social science.
2ozziegooen1yI thought Probabilistic Models of Cognition [https://probmods.org/] was quite great (it seems criminally underappreciated); that seems like a good step in this direction. Perhaps in the future, one could prove that "This environment with these actors will fail in these ways" by empirically showing that reinforcement agents optimizing in those setups lead to specific outcomes.

One question around the "Long Reflection" or around "What will AGI do?" is something like, "How bottlenecked will be by scientific advances that we'll need to then spend significant resources on?"

I think some assumptions that this model typically holds are:

1. There will be decision-relevant unknowns.
2. Many decision-relevant unkowns will be EV-positive to work on.
3. Of the decision-relevant unknowns that are EV-positive to work on, these will take between 1% to 99% of our time.

(3) seems quite uncertain to me in the steady state. I believe it makes an intuitiv

I feel like a decent alternative to a spiritual journey would be an epistemic journey.

An epistemic journey would basically involve something like reading a fair bit of philosophy and other thought, thinking, and becoming less wrong about the world.

Instillation, Proliferation, Amplification

Paul Christiano and Ought use the terminology of Distillation and Amplification to describe a high-level algorithm of one type of AI reasoning.

I’ve wanted to come up with an analogy to forecasting systems. I previously named a related concept Prediction-Augmented Evaluation Systems, one somewhat renamed to “Amplification” by Jacobjacob in this post.

I think one thing that’s going on is that “distillation” doesn’t have an exact equivalent with forecasting setups. The term “distillation” comes with the assumptions:

1. T
3bgold1yIs there not a distillation phase in forecasting? One model of the forecasting process is person A builds up there model, distills a complicated question into a high information/highly compressed datum, which can then be used by others. In my mind its: Model -> Distill - > "amplify" (not sure if that's actually the right word) I prefer the term scalable instead of proliferation for "can this group do it cost-effectively" as it's a similar concept to that in CS.
5ozziegooen1yDistillation vs. Instillation My main point here is that distillation is doing 2 things: transitioning knowledge (from training data to a learned representation), and then compressing that knowledge.[1] The fact that it's compressed in some ways arguably isn't always particularly important; the fact that it's transferred is the main element. If a team of forecasters basically learned a signal, but did so in a very uncompressed way (like, they wrote a bunch of books about said signal), but still were somewhat cost-effective, I think that would be fine. Around "Profileration" vs. "Scaling"; I'd be curious if there are better words out there. I definitely considered scaling, but it sounds less concrete and less specific. To "proliferate" means "to generate more of", but to "scale" could mean, "to make look bigger, even if nothing is really being done." I think my cynical guess is that "instillation/proliferation" won't catch on because they are too uncommon, but also that "distillation" won't catch on because it feels like a stretch from the ML use case. Could use more feedback here. [1] Interestingly, there seem to be two distinct stages in Deep Learning that map to these two different things, according to Naftali Tishby's claims [https://www.youtube.com/watch?v=XL07WEc2TRI&t=982s].

It seems really hard to deceive a Bayesian agent who thinks you may be deceiving them, especially in a repeated game. I would guess there could be interesting theorems about Bayesian agents that are attempting to deceive one another; as in, in many cases their ability to deceive the other would be highly bounded or zero, especially if they were in a flexible setting with possible pre-commitment devices.

To give a simple example, agent A may tell agent B that they believe , even though they internally believe . However, if this were somewhat repeat

8habryka1yBayesian agents are logically omniscient, and I think a large fraction of deceptive practices rely on asymmetries in computation time between two agents with access to slightly different information (like generating a lie and checking the consistencies between this new statement and all my previous statements) My sense is also that two-player games with bayesian agents are actually underspecified and give rise to all kinds of weird things due to the necessity for infinite regress (i.e. an agent modeling the other agent modeling themselves modeling the other agent, etc.), which doesn't actually reliably converge, though I am not confident. A lot of decision-theory seems to do weird things with bayesian agents. So overall, not sure how well you can prove theorems in this space, without having made a lot of progress in decision-theory, and I expect the resolution to a lot of our confusions in decision-theory to be resolved by moving away from bayesianism.
6ozziegooen1yHm... I like the idea of an agent deceiving another due to it's bounds on computational time, but could imagine many stable (though smaller) solutions that wouldn't. I'm curious if a good bayesian agent could do "almost perfect" on many questions given limited computation. For instance, a good bayesian would be using bayesianism to semi-optimally use any set of computation (assuming it has some sort of intuition, which I assume is necessary?) On being underspecified, it seems to me like in general our models of agent cognition forever have been pretty underspecified, so would definitely agree here. "Ideal" bayesian agents are somewhat ridiculously overpowered and unrealistic. I found the simulations around ProbMods to be interesting at modeling similar things; I think I'd like to see a lot more simulations for this kind of work. https://probmods.org/ [https://probmods.org/]

Named Footnotes: A (likely) mediocre proposal

Epistemic status: This is probably a bad idea, because it's quite obvious yet not done; i.e. Chesterson's fence.

One bad practice in programming is to have a lot of unnamed parameters. For instance,

  createPost(author, post, comment, name, id, privacyOption, ...)


Instead it's generally better to used Named Parameters, like,

  createPost({author, post, comment, name, id, privacyOption})


Footnotes/endnotes seem similar. They are ordered by number, but this can be quite messy. It's particularly annoying for autho

1Pattern1yThat's what arrays are for. What software does this well? 10 endnotes? Break the document up into sections, and the footnotes up into sections. Authors also could not re-order the footnotes. Or separate drafting and finished product: * indicates a footnote (to be replaced with a number later). At the end, searching * will find the first instance. (If the footnotes are being made at the same, then /* for the notes in the body, and \* for the footnotes at the end. Any uncommon symbols work - like qw.)
2ozziegooen1yArrays are useful for some kinds of things, for sure, but not when you have some very different parameters, especially if they are of different kinds. It would be weird to replace getUser({userId, databaseId, params}) with something like getUser([inputs]) where inputs is an array of [userId, databaseId, params]. Depends on your definition of "well", but things like Microsoft Word and to what I believe is a lesser extent Google Docs at least have ways of formally handling footnotes/endnotes, which is better than not using these features (like, in most internet comment editors). That could work in some cases. I haven't seen that done much on most online blog posts. Also, there's definitely controversy [https://libguides.usc.edu/writingguide/notes] if this is a good idea. Fair point, but this would be seen as lazy, and could be confusing. If your footnotes are numbers [8], [2], [3], [1], etc. that seems unpolished. That said, I wouldn't mind this much, and it could be worth the cost.
3Pattern1yThe page you linked was a great overview. It noted: With a physical document, the two parts (body+endnotes) can be separated for side by side reading. With a digital document, it helps to have two copies open. This seems like a problem with an easy solution. (The difficult solution is trying to make it easier for website makers to get more sophisticated editors in their comments section.) A brief search for browser extensions suggests it might be possible with an extension that offers an editor, or made easier with one that allows searching multiple things and highlighting them in different colors. Alternatively, a program for this might: Make sure every pair of []s is closed.* Find the strings contained in []s. Make sure they appear (at most) two times.** If they appear 3 times, increment the later one (in both places where it appears, if applicable). This requires that the footnotes below already be written. This requirement could be removed if the program looked for (or created) a "Footnotes" or "Endnotes" header (just that string and an end of line), and handled things differently based on that. Such a program could be on a website, though that's requires people bother switching to that, which, even if bookmarked, is only slightly easier than opening an editor. As a browser extension, it would have to figure out/be told 1. what part of the text it's supposed to work on, 2. when to be active, and 3. how to make the change. 1. could be done by having the message begin with start, and end with end, as long as the page doesn't include those words (with []s in between them). 2. This could be done automatically, or with a button. 3. Change the text automatically, or make a suggestion? This could simplify things - if people took things in that format and ran them through a program that fixed it/pointed it out (and maybe other small mistakes). *A programming IDE does this. **And this as well, with a bit more work, enabling named footnotes. The tric

Prediction evaluations may be best when minimally novel

Imagine a prediction pipeline is resolved with a human/judgemental evaluation. For instance, a group today starts predicting what a trusted judge 10 years from now will say for the question, "How much counterfactual GDP benefit did policy X make, from 2020-2030?"

So, there are two stages:

1. Prediction
2. Evaluation

One question for the organizer of such a system is how many resources to delegate to the prediction step vs. the evaluation step. It could be expensive to both pay for predictors and evaluators,