<a class="PostsPageTitle-link" href="/posts/Df2uFGKtLWR7jDr5w/ozziegooen-s-shortform">ozziegooen&#x27;s Shortform

The term for the "fear of truth" is alethophobia. I'm not familiar of many other great terms in this area (curious to hear suggestions).

Apparently "Epistemophobia" is a thing, but that seems quite different; Epistemophobia is more the fear of learning, rather than the fear of facing the truth.

One given definition of alethophobia is,
"The inability to accept unflattering facts about your nation, religion, culture, ethnic group, or yourself"

This seems like a incredibly common issue, one that is especially talked about as of recent, but without much spec

... (read more)

6Dagon3y

Without looking, I'll predict it's a neologism - invented in the last ~25 years to apply to someone the inventor disagrees with. Yup, google n-grams shows 0 uses in any indexed book. A little more searching shows almost no actual uses anywhere - mostly automated dictionary sites that steal from each other, presumably with some that accept user submissions. I did find one claim to invention, in 2017: http://www.danielpipes.org/comments/236053 [http://www.danielpipes.org/comments/236053] . Oh, and earlier (2008), there's a book with that title: https://www.amazon.com/Alethophobia-Manoucher-Parvin/dp/1588140474 [https://www.amazon.com/Alethophobia-Manoucher-Parvin/dp/1588140474] . I still submit that this is a word in search of a need, which mostly exists as a schoolyard insult dressed up in Latin.

2ozziegooen3y

Yea, I also found the claim, as well as a few results from old books before the claim. The name come straight from the Latin though, so isn't that original or surprising. Just because it hasn't been used much before doesn't mean we can't start to use it and adjust the definition as we see fit. I want to see more and better terminology in this area.

[-]NunoSempere3y 10

> The name comes straight from the Latin though

From the Greek as it happens. Also, alethephobia would be a double negative, with a-letheia meaning a state of not being hidden; a more natural neologism would avoid that double negative. Also, the greek concept of truth has some differences to our own conceptualization. Bad neologism.

3ozziegooen3y

Ah, good to know. Do you have recommendations for other words?

4ChristianKl3y

The trend of calling things that aren't fears "-phobia" seems to me a trend that's harmful for clear communication. Adjusting the definition only leads to more confusion.

2Dagon3y

I think I want less terminology in this area, and generally more words and longer descriptions for things that need nuance and understanding. Dressing up insults as diagnoses doesn't help any goals I understand, and jargon should only be introduced as part of much longer analyses where it illuminates rather than obscures.

2Matt Goldenberg3y

Can you give examples of things you think would fit under this? It seems that there are lots of instances of being resistant to the truth, but I can think of very few that I would categorize as fear of truth. It's often fear of something else (e.g. fear of changing your identity) or biases (e.g. the halo effect or consistency bias) that cause people to resist. I can think of very few cases where people have a general fear of truth.

I keep seeing posts about all the terrible news stories in the news recently. 2020 is a pretty bad year so far.

But the news I've seen people posting typically leaves out most of what's been going on in India, Pakistan, much of the Middle East as of recent, most of Africa, most of South America, and many, many other places as well.

The world is far more complicated than any of us have time to adequately comprehend. One of our greatest challenges is to find ways to handle all this complexity.

The simple solution is to try to spend more time reading the usual n

... (read more)

7Dagon3y

There's a parallel development that makes any strategy difficult - it's financially rewarding to misdirect and mis-aggregate the big-picture publications and comparisons. So you can't spend less time on details, as you can't trust the aggregates without checking. See also Gell-Mann Amnesia. Without an infrastructure for divvying up the work to trusted cells that can understand (parts of) the details and act as a check on the aggregators and each other, the only answer is to spot-check details yourself, and accept ignorance of things you didn't verify.

3Rudi C3y

I feel most people, including myself, don't even use the aggregators already available. For example, there are lots of indices and statistics (ideally there should be much more, but anyways), but I rarely go out of my way to consume them. Some examples I just thought of: * https://rsf.org/en/ranking [https://rsf.org/en/ranking] * https://www.globalhungerindex.org/results.html [https://www.globalhungerindex.org/results.html] * Keeping a watch on new entries to and exits from Fortune 500 * Looking at the stocks of the top 20 companies every quarter * ... There are several popular books that throw surprising statistics around, like Factfulness; This suggests a lot of us are disconnected from basic statistics, that we presumably could easily get by just googling.

Intervention dominance arguments for consequentialists

Global Health

There's a fair bit of resistance to long-term interventions from people focused on global poverty, but there are a few distinct things going on here. One is that there could be a disagreement on the use of discount rates for moral reasoning, a second is that the long-term interventions are much more strange.

No matter which is chosen, however, I think that the idea of "donate as much as you can per year to global health interventions" seems unlikely to be ideal upon clever thinking.

For the

... (read more)

Do people have precise understandings of the words they use?

On the phrase "How are you?", traditions, mimesis, Chesterton's fence, and their relationships to the definitions of words.

Epistemic status
Boggling. I’m sure this is better explained somewhere in the philosophy of language but I can’t yet find it. Also, this post went in a direction I didn’t originally expect, and I decided it wasn’t worthwhile to polish and post on LessWrong main yet. If you recommend I clean this up and make it an official post, let me know.

One recurrent joke is that one per

... (read more)

2ozziegooen3y

Update: After I wrote this shortform, I did more investigation in Pragmatics and realized most of this was better expressed there.

2Raemon3y

What's Pragmatics in this case?

2ozziegooen3y

Ah, sorry for not responding earlier. By Pragmatics I meant Pragmatics in linguistics. It studies what people mean when they say words. https://plato.stanford.edu/entries/pragmatics/

I've been reading through some of TVTropes.org and find it pretty interesting. Part of me wishes that Wikipedia were less deletionist, and wonders if there could be a lot more stuff similar to TV Tropes on it.

TVTropes basically has an extensive ontology to categorize most of the important features of games, movies, and sometimes real life. Because games & movies are inspired by real life, even those portions are applicable.

Here are some phrases I think are kind of nice; each that has a bunch of examples in the real world. These are often military relat

... (read more)

6Matt Goldenberg3y

A think I want: A recommendation engine that works based on listing the tropes you enjoy.

3Pattern3y

This article expresses the same sentiment, and may include links to what that looked like, and where it went: https://www.gwern.net/In-Defense-Of-Inclusionism [https://www.gwern.net/In-Defense-Of-Inclusionism]

I think the thing I find the most surprising about Expert Systems is that people expected them to work so early on, and apparently they did work in some circumstances. Some issues:

The user interfaces, from what I can tell, were often exceedingly mediocre. User interfaces are difficult to do well and difficult to specify, so are hard to guarantee quality in large and expensive projects. It was also significantly harder to make good UIs back when expert systems were more popular, than it is today.
From what I can tell, many didn't even have notions of unce

... (read more)

It seems inelegant to me that utility functions are created for specific situations, while these clearly aren't the same as that of the agent in total among all of their decisions. For instance, a model may estimate an agent's expected utility from the result of a specific intervention, but this clearly isn't quite right; the agent has a much more complicated utility function outside this intervention. According to a specific model, "Not having an intervention" could set "Utility = 0"; but for any real agent, it's quite likely their life wouldn't actually

... (read more)

4ozziegooen3y

Related to this, one common argument against utility maximization is that "we still cannot precisely measure utility". But here, it's perhaps more clear that we don't need to. What's important for decision making is that we have models that we can expect will help us maximize our true utility functions, even if we really don't know much about what they really are.

3Stuart_Armstrong3y

I delve into that here: https://www.lesswrong.com/posts/Lb3xCRW9usoXJy9M2/platonic-rewards-reward-features-and-rewards-as-information#Extending_the_problem [https://www.lesswrong.com/posts/Lb3xCRW9usoXJy9M2/platonic-rewards-reward-features-and-rewards-as-information#Extending_the_problem]

3ozziegooen3y

Oh fantastic, thanks for the reference!

2Pattern3y

^U and ^U look to be the same.

4ozziegooen3y

Thanks! Fixed. I'm sure the bottom notation could be improved, but am not sure the best way. In general I'm trying to get better at this kind of mathematics.

2Pattern3y

You got the basic idea across, which is a big deal. Though whether it's A or B isn't clear: A) "this isn't all of the utility function, but its everything that's relevant to making decisions about this right now". ^U doesn't have to be U, or even a good approximation in every situation - just (good enough) in the situations we use it. Building a building? A desire for things to not fall on people's heads becomes relevant (and knowledge of how to do that). Writing a program that writes programs? It'd be nice if it didn't produce malware. Both desires usually exist - and usually aren't relevant. Models of utility for most situations won't include them. B) The cost of computing the utility function more exactly in the case exceeds the (expected) gains. isn't clear.

2ozziegooen3y

I think I agree with you. There's a lot of messiness with using ^U and often I'm sure that this approximation leads to decision errors in many real cases. I'd also agree that better approximations of ^U would be costly and are often not worth the effort. Similar to how there's a term for "Expected value of perfect information", there could be an equivalent for the expected value of a utility function, even outside of uncertainty of parameterized that were thought to be included. Really, there could be calculations for "expected benefit from improvements to a model", though of course this would be difficult to parameterize (how would you declare that a model has been changed a lot vs. a little? If I introduce 2 new parameters, but these parameters aren't that important, then how big of a deal should this be considered in expectation?)

2Pattern3y

The model has changed when the decisions it is used to make change. If the model 'reverses' and suggests doing the opposite/something different in every case from what it previously recommended, then it has 'completely changed'. (This might be roughly the McNamara fallacy, of declaring that things that 'can't be measured' aren't important.) EDIT: Also, if there's a set of information consisting of a bunch of pieces, A, B, and C, and incorporating all but one of them doesn't have a big impact on the model, but the last piece does, whichever piece that is, 'this metric' could lead to overestimating the importance of whichever piece happened to be last, when it's A, B, and C together that made an impact. It 'has this issue' because the metric by itself is meant to notice 'changes in the model over time', not figure out why/solve attribution.

I've been trying to scurry academic fields for discussions of how agents optimally reduce their expected error for various estimands (parameters to estimate). This seems like a really natural thing to me (the main reason why we choose some ways of predictions over others), but the literature seems kind of thin from what I can tell.

The main areas I've found have been Statistical Learning Theory and Bayesian Decision / Estimation Theory. However, Statistical Learning Theory seems to be pretty tied to Machine Learning, and Bayesian Decision / Estimation Theor

... (read more)

[-]ozziegooen2y 6

I think brain-in-jar or head-in-jar are pretty underrated. By this I mean separating the head from the body and keeping it alive with other tooling. Maybe we could have a few large blood processing plants for many heads, and the heads could be connected to nerve I/O that would be more efficient than finger -> keyboard IO. This seems fairly easier than uploading, and possibly doable in 30-50 years.

I can't find much about how difficult it is. It's obviously quite hard and will require significant medical advances, but it's not clear just how many are need... (read more)

4avturchin2y

Yes. But the head also ages and could have terminal diseases: cancer, stroke, ALZ. Given the steep nature of the Gompertz law, the life expectancy of even a perfect head in jar (of an old man) will be less than 10 years (I guess). So it is not immortality, but a good way to wait until better life extension technologies.

2ozziegooen2y

I was thinking of it less for life extension, and more for a quality of life and cost improvement.

Western culture is known for being individualistic instead of collectivist. It's often assumed (with evidence) that individualistic cultures tend to be more truth seeking than collectivist ones, and that this is a major advantage.

But theoretically, there could be highly truth seeking collectivist cultures. One could argue that Bridgewater is a good example here.

In terms of collective welfare, I'm not sure if there are many advantages to individualism besides the truth seeking. A truth seeking collectivist culture seems pretty great to me, in theory.

2mako yass3y

I suspect there's a limit on how good at truthseeking individualism can make people. Good information is a commons, its sum value is greater the more it is shared, it is not funded in proportion to its potential, under economies of atomized decisions. We need a political theory of due deference to expertise. Wherever experts fail, or the wrong experts are appointed, or where a layperson on the ground stops believing that experts are even identifiable, there is work to be done.

Say Tim states, “There is a 20% probability that X will occur”. It’s not obvious to me what that means for Bayesians.

It could mean:

Tim’s prior is that there’s a 20% chance. (Or his posterior in the context of evidence)
Tim believes that when the listeners update on him saying there’s a 20% chance (perhaps with him providing insight in his thinking), their posterior will converge to there being a 20% chance.
Tim believes that the posterior of listeners may not immediately converge to 20%, but the posterior of the enlightened versions of these listeners wo

... (read more)

5philip_b3y

It's definitely the first. The second is bizarre. The third can be steelmanned as "Given my evidence, an ideal thinker would estimate the probability to be 20%, and we all here have approximately the same evidence, so we all should have 20% probabilities", which is almost the same as the first.

4ozziegooen3y

I don't think it's only the first. It seems weird to me imagine telling to a group that "There's a 20% probability that X will occur" if I really have little idea and would guess many of them would have a better sense than me. I would only personally feel comfortable doing this if I was quite sure my information was quite a bit better than theirs. Else, I'd say something like, "I personally think there's a 20% chance, but I really don't have much information."

4ozziegooen3y

I think my current best guess to this is something like: When humans say thing X, they don't mean the literal translation of X, but rather are pointing to X', which is a specific symbol that other humans generally understand. For instance, "How are you" is a greeting, not typically a literal question. [How Are You] can be thought of as a symbol that's very different than the sum of it's parts. That said, I find it quite interesting that the basics of human use of language seem to be relatively poorly understood; in the sense that I'd expect many people to disagree on what they think “There is a 20% probability that X will occur” means, even after using it with each other in a setting that assumes some amount of understanding.

4ChristianKl3y

I take it to mean that if Tim is acting optimally and has to take a bet on the outcome 1:4 would be the point where both sides of the bad are equally profitable to him while if the odds deviate from 1:4 one side of the bet would be preferable to him.

1ozziegooen3y

One thing this wouldn't take into account is strength or weight of evidence. If Tim knew that all of the listeners had far more information than him, and thus probably could produce better estimates of X, then it seems strange for Tim to tell them that the chances are 20%. I guess my claim that saying “There is a 20% probability that X will occur” is more similar to: "I'm quite confident that the chances are 20%, and you should generally be too" than it is to, "I personally believe that the chances are 20%, but have no idea o how much that should update the rest of you."

2Tetraspace3y

Other things that Tim might mean when he says 20%: * Tim is being dishonest, and believes that the listeners will update away from the radical and low-status figure of 20% to avoid being associated with the lowly Tim. * Tim believes that other listeners will be encouraged to make their own probability estimates with explicit reasoning in response, which will make their expertise more legible to Tim and other listeners. * Tim wants to show cultural allegiance with the Superforecasting tribe.

Perhaps resolving forecasts with expert probabilities can be better than resolving them with the actual events.

The default in literature on prediction markets and decision markets is to expect that resolutions should be real world events instead of probabilistic estimates by experts. For instance, people would predict "What will the GDP of the US in 2025 be?”, and that would be scored using the future “GDP of the US.” Let’s call these empirical resolutions.

These resolutions have a few nice properties:

We can expect expect them to be roughly calibrated. (

... (read more)

[-]NunoSempere3y 11

Here is another point by @jacobjacob, which I'm copying here in order for it not to be lost in the mists of time:

Though just realised this has some problems if you expected predictors to be better than the evaluators: e.g. they’re like “one the event happens everyone will see I was right, but up until then no one will believe me, so I’ll just lose points by predicting against the evaluators” (edited)

Maybe in that case you could eventually also score the evaluators based on the final outcome… or kind of re-compensate people who were wronged the first time…

8Matt Goldenberg3y

I'm really interested in this type of scheme because it would also solve a big problem in futarchy and futarchy-like setups that use prediction polling, namely, the inability to score conditional counterfactuals (which is most of the forecasting you'll be doing in Futarchy-like setup). One thing you could do instead of scoring people against expert assesments is also potentially score people against the final aggregate and extremized distribution. One issue with any framework like this is that general calibration may be very different than calibration at the tails. Whatever scoring rule you're using to determine calibration of experts or aggregate scoring has the same issue that long tail events rarely happen. Another solution to this problem (although it doesn't solve the counterfactual conditional problem) is to create tailored scoring rules that provide extra rewards for events at the tails. If an event at the tails is a million times less likely to happen, but you care about it equally to events at the center, then provide a million times reward for accuracy near the tail in the event it happens. Prior work on tailored scoring rules for different utility functions here: https://www.evernote.com/l/AAhVczys0ddF3qbfGk_s4KLweJm0kUloG7k/ [https://www.evernote.com/l/AAhVczys0ddF3qbfGk_s4KLweJm0kUloG7k/]

2ozziegooen3y

Good points! Also, thanks for the link, that's pretty neat. I think that an efficient use of expert assessments would be for them to see the aggregate, and then basically adjust that as is necessary, but to try to not do much original research. I just wrote a more recent shortform post about this. I think that we can get calibration to be as good as experts can figure out, and that could be enough to be really useful.

3NunoSempere3y

Another point in favor of such a set-up would be that aspiring superforecasters get much, much more information when they see ~[the prediction of a superforecaster would have made having their information]; a point vs a distribution. I'd expect that this means that market participants would get better, faster.

2ozziegooen3y

Yep, this way would basically be much more information-dense, with all the benefits that comes from that.

3ChristianKl3y

You can train experts to be calibrated in different ways. If you train experts to be calibrated to pick the right probability on GPOpen where probability is done in steps on 1, I don't think those experts will be automatically calibrated to distinguish a p=0.00004 event from a p=0.00008. Experts would actually need to be calibrated on getting probabilities inside the tail right. I don't think we know how to do calibration training for that tail.

1ozziegooen3y

I think this could be a good example for what I'm getting at. I think there are definitely some people in some situations who can distinguish a p=0.00004 event from a p=0.00008 event. How? By making a Fermi model or similar. A trivial example would be a lottery with calculable odds of success. Just because the odds are low doesn't mean they can't be precisely estimated. I expect that the kinds of problems that GPOpen would consider asking AND are incredibly unlikely, would be difficult to estimate within 1 order of magnitude. But may still be able to do a decent job, especially in cases where you can make neat Fermi models. However, of course, it seems very silly to use the incentive mechanism "you'll get paid once we know for sure if the event happened" on such an event. Instead, if resolutions are done with evaluators, then there is much more of a signal.

5Matt Goldenberg3y

I'm fairly skeptical of this. From a conceptual perspective, we expect the tails to be dominated by unknown unknowns and black swans. Fermi estimates and other modelling tools are much better at estimating scenarios that we expect. Whereas, if we find ourselves in the extreme tails, its often because of events or factors that we failed to model.

2ozziegooen3y

I'm not sure. The reasons things happen at the tails typically fall into categories that could be organized to be a small set. For instance: * The question wasn't understood correctly. * A significant exogenous event happened. But, as we do a bunch of estimates, we could get empirical data about these possibilities, and estimate the potentials for future tails. This is a bit different to what I was mentioning, which was more about known but small risks. For instance, the "amount of time I spend on my report next week" may be an outlier if I die. But the chance of serious accident or death can be estimated decently well enough. These are often repeated known knowns.

2ChristianKl3y

You might have people who can distinguish those, but I think it's a mistake to speak of calibration in that sense as the word usually refers to people who actually trained to be calibrated via feedback.

3Pattern3y

So you don't want predictions*, you want models**. Robust/fully fleshed out models. *predictions of events **predictions of which model is correct

2ozziegooen3y

I'm not sure I'd say that in the context of this post, but more generally, models are really useful. Predictions that come with useful models are a lot more useful than raw predictions. I wrote this other [https://www.lesswrong.com/posts/Df2uFGKtLWR7jDr5w/ozziegooen-s-shortform?commentId=3JoFTh6My4eX3E37p] post about a similar topic. For this specific post, I think what we're trying to get is the best prediction we could have had using data pre-event.

He's an in-progress hierarchy of what's needed for information to be most useful to an organization or other multi-agent system. I'm sure there must be other very similar hierarchies out there, but don't currently know of any quite like this.

Say you've come up with some cool feature that Apple could include in it's next phone. You think this is a great idea and they should add it in the future.

You're outside of Apple, so the only way you have of interacting with them is by sending information through various channels. The question is: what things should yo

... (read more)

4ozziegooen3y

Another note to this; there are cases where a system is both broken and fixable at the step-3 level. In some of these cases, it could be worth it to fix the system there instead, especially if you may want to make similar changes in the future. For instance, you may have an obvious improvement for your city to make. You may then realize that the current setups to suggest feedback are really difficult to use, but that it's actually quite feasible to make sure some changes happen that will make all kinds of useful feedback easier for the city to incorporate.

[-]ozziegooen2y 5

Real-world complexity is a lot like pollution and like a lot like taxes.

Pollution because it’s often an unintended negative externality of other decisions and agreements.

Whenever you write a new feature or create a new rule, that’s another thing you and others will need to maintain and keep track of. There are some processes that pollute a lot (messy bureaucratic systems producing ugly legislation) and processes that pollute a little (top programmers carefully adding to a codebase).

Taxes, because it introduces a steady cost to a whole bunch of interactions... (read more)

-1garbageactual2y

Almost like there's an Incompleteness Theorem somewhere in there or something?

[-]ozziegooen2y 5

On Berkeley coworking:

I've recently been looking through available Berkeley coworking places.

The main options seem to be WeWork, NextSpace, CoWorking with Wisdom, and The Office: Berkeley. The Office seems basically closed now, CoWorking with Wisdom seemed empty when I passed by, and also seems fairly expensive, but nice.

I took a tour of WeWork and Nextspace. They both provide 24/7 access for all members, both have a ~$300/m option for open coworking, a ~$375/m for fixed desks, and more for private/shared offices. (At least now, with the pandemic. WeWork i... (read more)

[-]ozziegooen3y 5

Would anyone here disagree with the statement:

Utilitarians should generally be willing to accept losses of knowledge / epistemics for other resources, conditional on the expected value of the trade being positive.

4Dagon3y

[ not a utilitarian; discount my opinion appropriately ] This hits one of the thorniest problems with Utilitarianism: different value-over-time expectations depending on timescales and assumptions. If one is thinking truly long-term, it's hard to imagine what resource is more valuable than knowledge and epistemics. I guess tradeoffs in WHICH knowledge to gain/lose have to be made, but that's an in-category comparison, not a cross-category one. Oh, and trading it away to prevent total annihilation of all thinking/feeling beings is probably right.

2ozziegooen3y

I think my thinking is that for utilitarians, these are generally instrumental, not terminal values. Often they're pretty important instrumental values, but this still would mean that they could be traded off in respect to the terminal values. Of course, if they are "highly important" instrumental values, then something very large would have to be offered for a trade to be worth it. (total annihilation being one example)

2Dagon3y

I think we're agreed that resources, including knowledge, are instrumental (though as a human, I don't always distinguish very closely). My point was that for very-long-term terminal values, knowledge and accuracy of evaluation (epistemics) are far more important than almost anything else. It may be that there's a declining marginal value for knowledge, as there is for most resources, and once you know enough to confidently make the tradeoffs, you should do so. But if you're uncertain, go for the knowledge.

3edoarad3y

Non-Bayesian Utilitarian that are ambiguity averse [https://www.lesswrong.com/posts/iDYaCJ3o3Q7ypriTF/knightian-uncertainty-and-ambiguity-aversion-motivation] sometimes need to sacrifice "expected utility" to gain more certainty (in quotes because that need not be well defined).

3AprilSR3y

Doesn't being willing to accept a trade *directly follow* from the expected value of the trade being positive? Isn't that like, the *definition* of when you should be willing to accept a trade? The only disagreement would be how likely it is that losses of knowledge / epistemics are involved in positive value trades. (My guess is it does happen rarely.)

2ozziegooen3y

I'd generally say that, but wouldn't be surprised if there were some who disagreed; who's argument would be something like what-to-me would sound like a modification of utilitarianism, [utilitarianism+epistemic-terminal-values].

1AprilSR3y

If you have epistemic terminal values then it would not be a positive expected value trade, would it? Unless "expected value" is referring to the expected value of something other than your utility function, in which case it should've been specified.

2ozziegooen3y

Yep, I would generally think so. I was doing what may be a poor steelman of my assumptions of how others would disagree; I don't have a great sense of what people who would disagree would say at this point.

1Pattern3y

Happiness + Knowledge. (A related question is, do people with these values drink?)

2[anonymous]3y

Only if the trade is voluntary. If the trade is forced (e.g. in healthcare) then you may have two bad options, and the option you do want is not on the table.

3Isnasene3y

In general, I would agree with the above statement (and technically speaking, I have made such trade-offs). But I do want to point out that it's important to consider what the loss of knowledge/epistemics entails. This is because certain epistemic sacrifices have minimal costs (I'm very confident that giving up FDT for CDT for the next 24 hours won't affect me at all) and some have unbounded costs (if giving up materialism causes me to abandon cryonics, it's hard to quantify how large of a blunder that would be). This is especially true of epistemics that allow to you be unboundedly exploited by an adversarial agent. As a result, even when the absolute value looks positive to me, I'll still try to avoid this kinds of trade-offs because certain black swans (ie bumping into an adversarial agent that exploits your lack of knowledge about something) make such bets very high risk.

2ozziegooen3y

This sounds pretty reasonable to me; it sounds like you're basically trying to maximize expected value, but don't always trust your initial intuitions, which seems quite reasonable.

3Pattern3y

[What "utilitarian" means could use some resolving, so I just treated this as "people".] I would disagree. I tried to find the relevant post in the sequences and found this [https://www.lesswrong.com/posts/PkDn96XQZDHPLcreY/aella-on-rationality-and-the-void] along with it: Would I accept that processes that take into account resource constraints might be more effective? Certainly, thought I think of that as 'starting the journey in a reasonable fashion' rather than 'going backwards' as your statement brings to mind.

3George3d63y

How would you define loss of knowledge ?

2ozziegooen3y

Basically, information that can be handled in "value of information" style calculations. So, if I learn information such that my accuracy of understanding the world increases, my knowledge is increased. For instance, if I learn the names of everyone in my extended family.

1George3d63y

Ok, but in this case do you mean "loss of knowledge" as in "loss of knowledge harbored within the brain" or "loss of knowledge no matter where it's stored, be it a book, brain, text file... etc" ? Further more, does losing copies of a certain piece of knowledge count as loss of knowledge ? What about translations of said knowledge (in another language or another philosophical/mathematical framework) that doesn't add any new information, just makes it accessible to a larger demographic ?

2ozziegooen3y

I was thinking the former, but I guess the latter could also be relevant/count. It seems like there's no strict cut-off. I'd expect a utilitarian to accept trade-offs against all these kinds of knowledge, conditional on the total expected value being positive.

1George3d63y

Well, the problem with the former (knowledge harbored within the brain) is that it's very vague and hard to define. If I have, say, a method to improve the efficacy of VX (an easily weaponizable nerve toxin). As a utilitarian I conclude this information is going to be harmful, I can purge it of my hard-drive, I can burn the papers I used to come up with this... etc. But I can't wipe my head clean of the information, at best I can resign to never talk about it to anyone and to not accord it much import, such that I may forget it. But that's not destruction per-say, it's closer to lying, not sharing the information with anyone (even if asked specifically), or to biasing your brain towards transmitting and remembering certain pieces of information (which we do all the time). However I don't see anything contentious with this case, nor with any other case of information-destruction, as long as it is for the greater utility. I think in general people don't advocate for destroying/forgetting information because: a) It's hard to do b) As a general rule of thumb the accumulation of information seems to be a good thing, even if the utility of a specific piece of information is not obvious But this is more of a heuristic, an exact principle.

2ozziegooen3y

I'd agree that the first one is generally pretty separated from common reality, but think it's a useful thought experiment. I was originally thinking of this more in terms of "removing useful information" than "removing expected-harmful information", but good point; the latter could be interesting too.

2George3d63y

Well,I think the "removing useful information" bit contradicts with utility to being with. As in, if you are a utilitarian, useful information == helps maximize utility. Thus the trade-off is not possible. I can think of some contrived examples where the trade-off is possible (e.g. where the information is harmful now but will be useful later), but in that case it's so easy to "hide" information in the modern age, instead of destroying it entirely, that the problem seem too theoretical to me. But at the end of the day, assuming you reached a contrived enough situation where the information must either be destroyed (or where hiding it devoid other people of the ability to discover further useful information), I think the utilitarian perspective has nothing fundamental against destroying it. However, no matter how hard I try, I can't really think of a very relevant example where this could be the case.

3ozziegooen3y

One extreme case would be committing suicide because your secret is that important. A less extreme case may be being OK with forgetting information; you're losing value, but the cost to maintain it wouldn't be worth it. (In this case the information is positive though)

2ozziegooen3y

There's some related academic work around this here: https://www.princeton.edu/~tkelly/papers/epistemicasinstrumental.pdf [https://www.princeton.edu/~tkelly/papers/epistemicasinstrumental.pdf] https://core.ac.uk/download/pdf/33752524.pdf [https://core.ac.uk/download/pdf/33752524.pdf] They don't specifically focus on utilitarians, but the arguments are still relevant. Also, this post is relevant: https://www.lesswrong.com/posts/dMzALgLJk4JiPjSBg/epistemic-vs-instrumental-rationality-approximations [https://www.lesswrong.com/posts/dMzALgLJk4JiPjSBg/epistemic-vs-instrumental-rationality-approximations]

[-]ozziegooen3y 5

There's a lot of arguing, of course, on if humans are rational, but this often mixes up two things: there's the "Von Neumann-Morgenstern utility function maximization" definition of "rational", and there's a hypothetical "rational" that a human could fulfill with constraints much more complicated than the classical approach, more in the direction of prospect theory, or Predictive Coding.

I think I regard the second definition as sufficiently not understood or defined that it isn't yet worth using in most conversation. It seems challenging, to say the least,

... (read more)

3Pattern3y

Or it could be an intuitive usage and mean "(more) optimal". "Why don't more people do [thing that will improve their health]?"

2ozziegooen3y

I like that question. I think that if people were to try to define optimal in a specific way, they would find that it requires a model of human behavior; the common one that academics would fall back to is that of Von Neumann-Morgenstern utility function maximization. I think it's quite possible that when we have better models of human behavior, we'll better recognize that in cases where people seem to be doing silly things to improve their health, they're actually being somewhat optimal given a large sets of physical and mental constraints.

[-]ozziegooen2y 4

I write a lot of these snippets to my Facebook wall, almost all just to my friends there. I just posted a batch of recent ones, might post similar in the future in batches. In theory it should be easy to post to both places, but in practice it seems a bit like a pain. Maybe in the future I'll use some solution to use the API to make a Slack -> (Facebook + LessWrong short form) setup.

That said, posting just to Facebook is nice as a first pass, so if people get too upset with it, I don't need to make totally public.

[-]ozziegooen2y 4

It’s a shame that our culture promotes casual conversation, but you’re generally not allowed to use it for much of the interesting stuff.

(Meets person with a small dog) “Oh, you have a dog, that’s so interesting. Before I get into specifics, can I ask for your age/gender/big 5/enneagram/IQ/education/health/personal wealth/family upbringing/nationality? How much well-being does the dog give you? Can you divide that up to include the social, reputational, self-motivational benefits? If it died tomorrow, and you mostly forgot about it, what percentage of your... (read more)

One question around the "Long Reflection" or around "What will AGI do?" is something like, "How bottlenecked will be by scientific advances that we'll need to then spend significant resources on?"

I think some assumptions that this model typically holds are:

There will be decision-relevant unknowns.
Many decision-relevant unkowns will be EV-positive to work on.
Of the decision-relevant unknowns that are EV-positive to work on, these will take between 1% to 99% of our time.

(3) seems quite uncertain to me in the steady state. I believe it makes an intuitiv

... (read more)

I feel like a decent alternative to a spiritual journey would be an epistemic journey.

An epistemic journey would basically involve something like reading a fair bit of philosophy and other thought, thinking, and becoming less wrong about the world.

Instillation, Proliferation, Amplification

Paul Christiano and Ought use the terminology of Distillation and Amplification to describe a high-level algorithm of one type of AI reasoning.

I’ve wanted to come up with an analogy to forecasting systems. I previously named a related concept Prediction-Augmented Evaluation Systems, one somewhat renamed to “Amplification” by Jacobjacob in this post.

I think one thing that’s going on is that “distillation” doesn’t have an exact equivalent with forecasting setups. The term “distillation” comes with the assumptions:

... (read more)

3bgold3y

Is there not a distillation phase in forecasting? One model of the forecasting process is person A builds up there model, distills a complicated question into a high information/highly compressed datum, which can then be used by others. In my mind its: Model -> Distill - > "amplify" (not sure if that's actually the right word) I prefer the term scalable instead of proliferation for "can this group do it cost-effectively" as it's a similar concept to that in CS.

5ozziegooen3y

Distillation vs. Instillation My main point here is that distillation is doing 2 things: transitioning knowledge (from training data to a learned representation), and then compressing that knowledge.[1] The fact that it's compressed in some ways arguably isn't always particularly important; the fact that it's transferred is the main element. If a team of forecasters basically learned a signal, but did so in a very uncompressed way (like, they wrote a bunch of books about said signal), but still were somewhat cost-effective, I think that would be fine. Around "Profileration" vs. "Scaling"; I'd be curious if there are better words out there. I definitely considered scaling, but it sounds less concrete and less specific. To "proliferate" means "to generate more of", but to "scale" could mean, "to make look bigger, even if nothing is really being done." I think my cynical guess is that "instillation/proliferation" won't catch on because they are too uncommon, but also that "distillation" won't catch on because it feels like a stretch from the ML use case. Could use more feedback here. [1] Interestingly, there seem to be two distinct stages in Deep Learning that map to these two different things, according to Naftali Tishby's claims [https://www.youtube.com/watch?v=XL07WEc2TRI&t=982s].

Agent-based modeling seems like one obvious step forward to me for much of social-science related academic progress. OpenAI's Hide and Seek experiment was one that I am excited about, but it is very simple and I imagine similar work could be greatly extended for other fields. The combination of simulation, possible ML distillation on simulation (to make it run much faster), and effective learning algorithms for agents, seems very powerful.

However, agent-based modeling still seems quite infrequently used within Academia. My impression is that agent-based so

... (read more)

4johnswentworth3y

Could you give a few specific examples where you imagine agent-based models would help?

7ozziegooen3y

Sure, 1. Humans as agents / psychology / economics. Instead of making mathematical models of rational agents, have people write code that predicts the behaviors of rational agents or humans. Test the "human bots" against empirical experimental results of humans in different situations, to demonstrate that the code accurately models human behavior. 2. Mechanism design. Show that according to different incentive structures, humans will behave differently, and use this to optimize the incentive structures accordingly. 3. Most social science. Make agent-based models to generally help explain how groups of humans interact with each other and what collective behaviors emerge. I guess when I said, "Much of academic progress"; I should have specified, "Academic fields that deal with modeling humans to some degree"; perhaps most of social science.

1ozziegooen3y

I thought Probabilistic Models of Cognition [https://probmods.org/] was quite great (it seems criminally underappreciated); that seems like a good step in this direction. Perhaps in the future, one could prove that "This environment with these actors will fail in these ways" by empirically showing that reinforcement agents optimizing in those setups lead to specific outcomes.

[-]ozziegooen3y 3

Named Footnotes: A (likely) mediocre proposal

Epistemic status: This is probably a bad idea, because it's quite obvious yet not done; i.e. Chesterson's fence.

One bad practice in programming is to have a lot of unnamed parameters. For instance,

  createPost(author, post, comment, name, id, privacyOption, ...)

Instead it's generally better to used Named Parameters, like,

  createPost({author, post, comment, name, id, privacyOption})

Footnotes/endnotes seem similar. They are ordered by number, but this can be quite messy. It's particularly annoying for autho

... (read more)

1Pattern3y

That's what arrays are for. What software does this well? 10 endnotes? Break the document up into sections, and the footnotes up into sections. Authors also could not re-order the footnotes. Or separate drafting and finished product: * indicates a footnote (to be replaced with a number later). At the end, searching * will find the first instance. (If the footnotes are being made at the same, then /* for the notes in the body, and \* for the footnotes at the end. Any uncommon symbols work - like qw.)

2ozziegooen3y

Arrays are useful for some kinds of things, for sure, but not when you have some very different parameters, especially if they are of different kinds. It would be weird to replace getUser({userId, databaseId, params}) with something like getUser([inputs]) where inputs is an array of [userId, databaseId, params]. Depends on your definition of "well", but things like Microsoft Word and to what I believe is a lesser extent Google Docs at least have ways of formally handling footnotes/endnotes, which is better than not using these features (like, in most internet comment editors). That could work in some cases. I haven't seen that done much on most online blog posts. Also, there's definitely controversy [https://libguides.usc.edu/writingguide/notes] if this is a good idea. Fair point, but this would be seen as lazy, and could be confusing. If your footnotes are numbers [8], [2], [3], [1], etc. that seems unpolished. That said, I wouldn't mind this much, and it could be worth the cost.

3Pattern3y

The page you linked was a great overview. It noted: With a physical document, the two parts (body+endnotes) can be separated for side by side reading. With a digital document, it helps to have two copies open. This seems like a problem with an easy solution. (The difficult solution is trying to make it easier for website makers to get more sophisticated editors in their comments section.) A brief search for browser extensions suggests it might be possible with an extension that offers an editor, or made easier with one that allows searching multiple things and highlighting them in different colors. Alternatively, a program for this might: Make sure every pair of []s is closed.* Find the strings contained in []s. Make sure they appear (at most) two times.** If they appear 3 times, increment the later one (in both places where it appears, if applicable). This requires that the footnotes below already be written. This requirement could be removed if the program looked for (or created) a "Footnotes" or "Endnotes" header (just that string and an end of line), and handled things differently based on that. Such a program could be on a website, though that's requires people bother switching to that, which, even if bookmarked, is only slightly easier than opening an editor. As a browser extension, it would have to figure out/be told 1. what part of the text it's supposed to work on, 2. when to be active, and 3. how to make the change. 1. could be done by having the message begin with start, and end with end, as long as the page doesn't include those words (with []s in between them). 2. This could be done automatically, or with a button. 3. Change the text automatically, or make a suggestion? This could simplify things - if people took things in that format and ran them through a program that fixed it/pointed it out (and maybe other small mistakes). *A programming IDE does this. **And this as well, with a bit more work, enabling named footnotes. The trick

[-]ozziegooen3y 3

It seems really hard to deceive a Bayesian agent who thinks you may be deceiving them, especially in a repeated game. I would guess there could be interesting theorems about Bayesian agents that are attempting to deceive one another; as in, in many cases their ability to deceive the other would be highly bounded or zero, especially if they were in a flexible setting with possible pre-commitment devices.

To give a simple example, agent A may tell agent B that they believe $ω = 0.8$ , even though they internally believe $ω = 0.5$ . However, if this were somewhat repeat

... (read more)

7habryka3y

Bayesian agents are logically omniscient, and I think a large fraction of deceptive practices rely on asymmetries in computation time between two agents with access to slightly different information (like generating a lie and checking the consistencies between this new statement and all my previous statements) My sense is also that two-player games with bayesian agents are actually underspecified and give rise to all kinds of weird things due to the necessity for infinite regress (i.e. an agent modeling the other agent modeling themselves modeling the other agent, etc.), which doesn't actually reliably converge, though I am not confident. A lot of decision-theory seems to do weird things with bayesian agents. So overall, not sure how well you can prove theorems in this space, without having made a lot of progress in decision-theory, and I expect the resolution to a lot of our confusions in decision-theory to be resolved by moving away from bayesianism.

5ozziegooen3y

Hm... I like the idea of an agent deceiving another due to it's bounds on computational time, but could imagine many stable (though smaller) solutions that wouldn't. I'm curious if a good bayesian agent could do "almost perfect" on many questions given limited computation. For instance, a good bayesian would be using bayesianism to semi-optimally use any set of computation (assuming it has some sort of intuition, which I assume is necessary?) On being underspecified, it seems to me like in general our models of agent cognition forever have been pretty underspecified, so would definitely agree here. "Ideal" bayesian agents are somewhat ridiculously overpowered and unrealistic. I found the simulations around ProbMods to be interesting at modeling similar things; I think I'd like to see a lot more simulations for this kind of work. https://probmods.org/ [https://probmods.org/]

If you think it’s important that people defer to “experts”, then it should also make sense that people should decide which people are “experts” by deferring to “expert experts”.

There are many groups that claim to be the “experts”, and ask that the public only listens to them on broad areas they claim expertise over. But groups like this also have a long history of underperforming other clever groups out there.

The US government has a long history of claiming “good reasons based on classified intel” for military interventions, where later this turns out to b... (read more)

People are used to high-precision statements given by statistics (the income in 2016 was $24.4 Million), and are used to low-precision statements given by human intuitions (From my 10 minute analysis, I think our organization will do very well next year). But there’s a really weird cultural norm against high-precision, intuitive statements. (From my 10 minute analysis, I think this company will make $28.5 Million in 2027).

Perhaps in part because of this norm, I think that there are a whole lot of gains to be made in this latter cluster. It’s not trivial to do this well, but it’s possible, and the potential value is really high.

I find SEM models to be incredibly practical. They might often over-reach a bit, but at least they present a great deal of precise information about a certain belief in a readable format.

I really wish there would be more attempts at making more diagrams like these in cases where there isn't statistical data. For examples, to explain phenomena like:

What caused the fall of Rome?
Why has scientific progress fallen over time in the US?
Why did person X get upset, after event Y?
Why did I make that last Facebook post?

In all of these cases, breath and depth... (read more)

There’s a big stigma now against platforms to give evaluations or ratings on individuals or organizations along various dimensions. See the rating episode of Black Mirror, or the discussion on the Chinese credit system.

I feel like this could be a bit of a missed opportunity. This sort of technology is easy to do destructively, but there are a huge number of benefits if it can be done well.

We already have credit scores, resumes (which are effectively scores), and social media metrics. All of these are really crude.

Some examples of things that could be possi... (read more)

2Viliam2y

How would you design a review system that cannot be gamed (very easily)? For example: Someone sends a message to their 100 friends, and tells them to open the romantic partners app and falsely accuse you of date rape. Suppose they do. What exactly happens next? * You are forever publicly marked as a rapist, no recourse. * You report those accusations as spam, or sue the people... but, the same could be done by an actual rapist... assuming the victims have no proof. Both outcomes seem bad to me, and I don't see how to design a system that prevents them both. (And if we reduce the system to situations when there is a proof... well, then you actually don't need a mutual rating system, just an app that searches people in the official records.) Similar for other apps... politicians of the other party will be automatically accused of everything; business competitors will be reported as untrustworthy; people who haven't even seen your book will give it zero stars rating. (The last thing is somewhat reduced by Amazon by requiring that the reviewers actually buy the book first. But even then, this makes it a cost/benefit question: you can still give someone X fake negative reviews, in return for spending the proportional amount of money on actually buying their book... without the intention to read it. So you won't write negative fake reviews for fun, but you can still review-bomb the ones you truly hate.)

2ozziegooen2y

I think it's very much a matter of unit economics. Court systems have a long history of dealing with false accusations, but still managing to uphold some sort of standards around many sorts of activity (murder and abuse, for instance). When it comes to false accusations; there could be different ways of checking these to verify them. These are common procedures in courts and other respected situations. If 100 people all opened an application and posted at a similar time, that would be fairly easy to detect, if the organization had reasonable resources. Hacker News and similar deal with similar situations (though obviously much less dramatic) very often with various kinds of spamming attacks and upvote rings. There's obviously always going to be some error rate, as is true for court systems. I think it's very possible that the possible efforts that would be feasible for us in the next 1-10 years in this area would be too expensive to be worth it, especially because they might be very difficult to raise money for. However, I would hope that abilities here eventually allow for systems that represent much more promising trade-offs.