E. P. Cooper

AI 2040's space supplement says "[are] there any mitigating factors, e.g., the experience being necessary for a strongly positive life[...]" which could be construed as referring to mindcrime trading off against prediction quality, among other things, though barely. Is there some agreement to avoid talking about mindcrime more directly? Normally you don't have to, since the problem needs to be handled alongside malign entities and code that diverts your physical computer from the faithful execution of the abstract program/has physical side effects, e.g. ro... (read more)

My unusual claim in this area is that we had an aborted attempt to transcend the modernist representations when they proved insufficient in the face of problems in foundational math[...]

I think this would be worth working on (because of Löb) if we expected to have enough time, but plausible developments in logical induction should get within epsilon of perfect (in expectation), enough for the next few million years. Though, if you research just enough to write a document that makes the strategy semi-legible, work could be resumed if a credible long-term AI pause is implemented.

1. Agree UDT is a lot better than FDT.

I'll write this in a way that's generally useful to readers since I don't know what you like and dislike about UDT, and I don't know what the term "UDT" points to in your head. Do you want to elaborate? Before, you said you liked "updateless EDT" (which UDT is not, but it is of the same theme, maybe I'll get back to this later) more than FDT because it has fewer missing components.

Note that I'm not sure I understand Bomb correctly, so please give me feedback if I get it wrong. My understanding is that the bomb going off o... (read more)

Actually, the difficult procedure in E can, by itself, lose you certain commitment races against certain opponents, if you take my short summary written there as a guide on how to think.

Note that to avoid losing a commitment race (by reaching an "infohazard") to the maximum amount UDT 1.0 allows, you must be fully updateless (base your policy on the prior alone^[1]) and also have a "mathematical intuition subroutine" that is somehow optimal and doesn't (de facto) tell you too much. Intentionally making your mathematical intuition worse is an unsolved proble... (read more)

To reference, here is my list of issues that fairly sophisticated reasoners get wrong when thinking about FDT.

A: The utility function you imagine using is not gerrymandered enough. For example, it's not the decision theory's problem if you care a large amount about a small-in-prior region but forget to encode such.

B: There is not a proper attempt at writing a bounded procedure, instead the "decision theory" as imagined consists of a human trying to guess the output of a logically omniscient, unbounded procedure.

Both you and Nate Soares don't seem to have a... (read more)

Here’s the idea: you don’t really know if you are the algorithm being simulated in Newcomb’s problem or the actual person.

The history of that idea:

Garry Drescher and Eliezer Yudkowsky, in their most polished respective works, were careful to avoid positively claiming that the "agent" in Omega's prediction was the same as the actual agent, or had any experiences, and were also careful to make the reasoning work exactly the same either way.

However, this isn't what is done in many cases. Even restricting to researchers that have extremely strong claims that t... (read more)

How much worse is 1.0 than 1.1 in practical situations? UDT 1.1 has excellent theoretical properties, but to my knowledge hasn't been useful for further work so far. Non exhaustively, this seems to be because attempts to reduce the size of the outer loop and spread the computational work out over time don't work properly in 1.1, due to the use of an optimal global strategy.

I think you might benefit from designing your own UDT 1.0 variant (i.e. a variant of Wei Dai's original) that doesn't require logical omniscience/infinite compute. Note that UDT 1.1 is mentioned indirectly in the published Death in Damascus paper, but what is actually described works similarly to UDT 1.0. Martín Soto and Abram Demski have put some effort into writing useful notes. I think they're still missing a precise constructive description of Bayesian Logical Inductors (BLIs), but you could partner with a mathematician and go around to the researchers involved until you've collected enough mental model content to have it written up.

Maybe he could transfer the cadence over, since that's what I have to filter out as "not information" as I listen to the podcast. I'm not aware of any reliable way to do that however, since the highest quality synthesis and voice recognition methods rely on pretrained neural networks. If you could get reliable millisecond-level timestamps for when each word starts and stops, both in the original and in the initial generated speech, you could use the Rubberband library to build the final speech as a piecewise assembly of scaled segments such that the word b... (read more)

I'm unsure of the state of Mythos-preview at the moment, but at the absolute frontier there will be a gap in work of some size while Mythos 5 is shut down.

Companies that are part of the Glasswing project have non-citizen employees. I don't have the full list, though my assumption is that any exceptions would be in certain subdivisions of defense companies, and those subdivisions are not generally the ones responsible for writing common consumer and enterprise computer programs. When writing programs intended for worldwide release or use, proper internal co... (read more)

How much worse would a hypothetical "almost on policy" distillation be, compared to on policy distillation?
It would require some sort of mapping from an old version of the same model (that hadn't already started forgetting a skill) to the current version, so the delta there might have to be so tiny that it would never be economical.
RLVR would become an order of magnitude slower.

I want to make a few adjustments to my terminology and clarify a point about what it would really entail for a human to use the "correct" decision theory. The new terminology should better match that of Vladimir Nesov's newest comment on this subject^[1].

The restrictions I describe in my comment above are actually about the human's decision to be replaced by an agent that has a utility function (or similar parameter) programmed into the correct decision theory. The list of options the superintelligence presents is important because it is upstream of the hum... (read more)

(ancient) discussion here.)

Note that this link is broken. It should go to Eliezer's top comment here:^[1]
https://www.lesswrong.com/posts/SpHYBhkaeDZpZyRvj/what-can-you-do-with-an-unfriendly-ai?commentId=5p7nw3RzLShRftnt8

1. ^{^}

   https://web.archive.org/web/20220121014447/https://www.lesswrong.com/posts/SpHYBhkaeDZpZyRvj/what-can-you-do-with-an-unfriendly-ai

I'm somewhat concerned about the possible problems that the recent increased load of patches may cause during the creation of the Linux 7.0.1 release. In theory it's just a matter of checking the applicability of the entire set of patches to Linus's tree, but given the situation I think the consequence of something getting missed is higher than normal^[1].

I think an alternative solution of using the 6.19.XX series from Greg K-H until a few days after its last release is a better idea, but it's close, ~0.35 that it ends up worse^[2]. I think better automation... (read more)

The existence of such facts seems plausible because if there were facts about what is rational (which seems likely) but no facts about how to become rational, that would seem like a strange state of affairs.

There might be facts about what's rational, but not about what utility function^[1] it is right to use. Maybe a superintelligence could tell you (in a somewhat objective/convergent sense) what utility function to use, but the exact utility function would depend on the utility function of the superintelligence^[2].

In Vladimir Nesov's opinion^[3], even prese... (read more)

perhaps due to COVID stimulus money being lost / used up by retail traders

If most people in the US had a bank account that featured monthly payments anywhere close to the "interest rate," the government could reduce risky retail investments with little delay by raising rates. This is not the case. Assuming even highly bounded rationality, it seems like retail traders should still not be losing as much money as they do, so maybe I'm making a modeling mistake and it would turn out that people really dislike bank accounts. This may be a typical mind fallacy p... (read more)

In the third paragraph of the linked comment, I suggest a good thing the Glasswing companies could do for the rest of us. KVM is part of the Linux kernel, but surrounding host programs aren't. Someone should commit to looking through all of these with Mythos (in public) so all other computer users can start setting up their security with that stack, so they can await further software updates for those projects. This would require regular releases from the maintainers, however.

https://www.lesswrong.com/posts/rEiidwAug6htax2Wb/project-glasswing-anthropic-sho... (read more)

Outsiders like myself can do some things to take advantage of this program. Using software that is confirmed to get patches is the best option, but that can't cover all use cases. Use Chromium to watch videos^[1], listen to audio and read PDF/text/HTML documents, use Firefox to edit PDFs, use the latest Linux kernel from Greg Kroah-Hartman (not Linus's tree) from kernel.org or the repos of e.g. Debian testing or Arch Linux. I don't have a suggestion for reading `.epub` E-Books, except writing a Haskell program using pure functions from the pandoc project to... (read more)

Admittedly this both assumes that the "time of peril" hypothesis is correct and can be handled while maintaing human freedom, and the solution only (in maximum robustness) binds until the end of this time.

I'll note that "not being sure what utility functions are in use" is generally (in the colloquial sense) not how standard game theory works. It seems like I am not competent enough at standard game theory to clearly write down the edge cases I think might exist that could help with your understanding. This paragraph could serve as a placeholder for the case where I develop that competence.

As for non-standard game theory, you say you're reading the 2009 book The Bounds of Reason here^[1] and I wonder if you've heard of the newer Translucent players: Explaini... (read more)