All of Kyre's Comments + Replies

Very nice. This is the cleanest result on cognitive (or rationality) costs in co-operative systems that I've seen. Modal combat seems kind of esoteric compared to, say, iterated prisoners' dilemma tournaments with memory, but it pays off nicely here. It gives you the outcomes of a set of other-modelling agents (without e.g. doing a whole lot of simulation), and the box-operator depth then plugs in as a natural modelling-cost measure.

Did you ever publish any of your modal combat code (I have a vague recollection that you had some Haskell code ?) ?

Don't humans have to give up on doing their own science then (at least fundamental physics) ?

I guess I can have the FAI make me a safe "real physics box" to play with inside the system; something that emulates what it finds out about real physics.

If you failed you'd want to distinguish between (a) rationalism sucking, (b) your rationalism sucking, or (c) EVE already being full of rationalists.

Whether or not success in Eve is relevant outside Eve is debatable, but I think the complexity, politics and intense competition means that it would be hard to find a better online proving ground.

Good advice, but I would go further. Don't use your inbox as a to-do list at all. I maintain a separate to-do list for roughly three reasons.

(1) You can't have your inbox in chronological and priority order. Keeping an inbox and email folders in chronological order is good for searching and keeping track of email conversations.

(2) Possibly just my own psychological quirk, but inbox emails feel like someone waiting for me and getting impatient. I can't seem to get away from my inbox fundamentally representing a communications channel with people on the othe... (read more)

Not just the environment in which you share your goals, but also how you suspect you will react to the responses you get.

When reading through these two scenarios, I can just as easily imagine someone reacting in exactly the opposite way. That is, in the first case, thinking "gosh, I didn't know I had so many supportive friends", "I'd better not let them down", and generally getting a self-reinforcing high when making progress.

Conversely, say phase 1 had failed and got the responses stated above. I can imagine someone thinking "hey ... (read more)

My five minutes thoughts worth.

Metrics that might useful (on the grounds that in hindsight people would say that they made bad decisions): traffic accident rate, deaths due to smoking, bankruptcy rates, consumer debt levels.

Experiments you could do if you could randomly sample people and get enough of their attention: simple reasoning tests (e.g. confirmation bias), getting people to make some concrete predictions and following them up a year later.

Maybe something measuring people's level of surprise at real vs fake facebook news (on the grounds people should be more surprised at fake news) ?

Doing theoretical research that ignores practicalities is sometimes turns out to be valuable in practice. It can open a door to something you assumed to be impossible; or save a lot of wasted effort on a plan that turns out to have an impossible sub-problem.

A concrete example of first category might be something like quantum error correcting codes. Prior to that theoretical work, a lot of people thought that quantum computers were not worth pursuing because noise and decoherence would be an insurmountable problem. Quantum fault tolerance theorems did nothi... (read more)

Will second "Good and Real" as worth reading (haven't read any of the others).

Maybe translating AI safety literature into Japanese would be a high-value use of your time ?

That's true, 20 years wouldn't necessarily bring to light a delayed effect.

However the GMO case is interesting because we have in effect a massive scale natural experiment, where hundreds of millions of people on one continent have eaten lots of GMO food while hundreds of millions on another continent have eaten very little, over a period of 10-15 years. There is also a highly motivated group of people who bring to the public attention even the smallest evidence of harm from GMOs.

While I don't rule out a harmful long-term effect, GMOs are a long way down on my list of things to worry about, and dropping further over time.

Heh, that was really just me trying to come up with a justification for shoe-horning a theory of identity into a graph formalism so that Konig's Lemma applied :-)

If I were to try to make a more serious argument it would go something like this.

Defining identity, whether two entities are 'the same person' is hard. People have different intuitions. But most people would say that 'your mind now' and 'your mind a few moments later' are do constitute the same person. So we can define a directed graph with verticies as mind states (mind states would probably have... (read more)

If we take "immortality" to mean "infinitely many distinct observer moments that are connect to me through moment-to-moment identity", then yes, by Konig's Lemma.

(Every infinite graph with finite-degree verticies has an infinite path)

(edit: hmmm, does many-worlds give you infinite-branching into distinct observer moments ?)

Procedural universes seemed to see a real resurgence from around 2014, with e.g. Elite Dangerous, No Man's Sky, and a quite a few others that have popped up since.

I love a beautiful procedural world, but I think things will get more interesting when games appear with procedural plot structures that are cohesive and reactive.

Then multiplayer versions will appear that weave all player actions into the plot, and those games will suck people in and never let go.

For 5 minutes suspension versus dreamless deep sleep - almost exactly the same person. For 3 hours dreamless deep sleep I'm not so sure. I think my brain does something to change state while I'm deep asleep, even if I don't consciously experience or remember anything. Have you ever woken up feeling different about something, or with a solution to a problem you were thinking about as you dropped off ? If that's not all due to dreaming, then you must be evolving at least slightly while completely unconscious.

Would a slow cell by cell, or thought by thought / byte by byte, transfer of my mind to another medium: one at a time every new neural action potential is received by a parallel processing medium which takes over? I want to say the resulting transfer would be the same consciousness as is typing this but then what if the same slow process were done to make a copy and not a transfer? Once a consciousness is virtual, is every transfer from one medium or location to another not essentially a copy and therefore representing a death of the originating version

Not sure if it's a scientific or engineering achievement, but this Nature letter stuck in my mind:

An aqueous, polymer-based redox-flow battery using non-corrosive, safe, and low-cost materials

Oh, I think I see what you mean. No matter how many or how detailed the simulations you run, if your purpose is to learn something from watching them, then ultimately you are limited by your own ability to observe and process what you see.

Whoever is simulating you only has to run the simulations that you launch to the level of fidelity such that you can't tell if they've taken shortcuts. The deeper the nested simulation people are, the harder it is for you to pay attention to them all, and the coarser their simulations can be.

If you are running simulations... (read more)

I was thinking more like a random power surge, programming error,or political coup within our simulation that happened to shut down the aspect of our program that was hogging resources. If the programmers want the program to continue, it can.

You're right - branch (2) should be "we don't keep running run more than one". We can launch as many as we like.

The single actor is not going to experience every aspect of the simulation in full fidelity, so a low-res simulation is all that is needed. (The actor might think that it is a full simulation,

That's the unbounded computation case.

It seems like there is a lot of room between "one simulation" and "unbounded computational resources"

Well the point is that if we are running on bounded resources, then the time until it runs out depends very sensitively on how many simulations we (and simulations like us) launch on average. Say that our simulation has a million years allocated to it, and we launch simulations starting a year back from the time when we launch a simulation.

If we don't launch any, we get a million years.

If we launch one, but that one doesn't launch any... (read more)

Here is a second Simulation Trilemma.

If we are living in a simulation, at least one of the following is true:

1) we are running on a computer with unbounded computational resources, or

2) we will not launch more than one simulation similar to our world, or

3) the simulation we are in will terminate shortly after we launch our own simulations.

Here 'short' is on the order of the period between the era we start the simulation at and when the simulation reaches our stage.

I heard strawberry jam can be made with just strawberries, water and sugar on a frying pan on the radio.

I'd use a stove.

A short hook headline like “avoiding existential risk is key to afterlife” can get a conversation going. I can imagine Salon, etc. taking another swipe at it, and in doing so, creating publicity which would help in finding more similar minded folks to get involved in the work of MIRI, FHI, CEA etc. There are also some really interesting ideas about acausal trade ...

Assuming you get good feedback and think that you have an interesting, solid arguments ... please think carefully about whether such publicity helps the existential risk movement more than it... (read more)

It's a trade-off. The example is simple enough that the alignment problem is really easy to see, but it also means that it is easy to shrug it off and say "duh, just the use obvious correct utility function for B".

Perhaps you could follow it up with an example with more complex mechanics (and or more complex goal for A) where the bad strategy for B is not so obvious. You then invite the reader to contemplate the difficulty of the alignment problem as the complexity approaches that of the real world.

Nitpick: we have equations for (special) relativistic quantum physics. Dirac was one of the pioneers, and the Standard Model for instance is a relativistic quantum field theory. I presume you mean general relativity (gravity) and quantum mechanics that is the problem.

(Douglas_Knight) Moreover, the predictions that QFT makes about chemistry are too hard. I don't think it is possible with current computers to compute the spectrum of helium, let alone lithium. A quantum computer could do this, though.

In the spirit of what Viliam suggested, maybe you could... (read more)

Things that are unsexy but I can actually verify as having been useful more than once:

In wallet, folded up tissue. For sudden attack of sniffles (especially on public transport), small cuts, emergency toilet paper.

In bag I carry every day: small pack of tissues, multitool, tiny torch, ibuprofin, pad and pencil, USB charging cable for phone, plastic spork, wet wipe thing from KFC (why do they always shovel multiples of those things in with my order ?).

Very rough toy example.

Say I've started a project which I can definitely see 5 days worth of work. I estimate there'll be some unexpected work in there somewhere, maybe another day, so I estimate 6 days.

I complete day one but have found another day's work. When should I estimate completion now ? Taking the outside view, finishing in 6 days (on day 7) is too optimistic.

Implicit in my original estimate was a "rate of finding new work" of about 0.2 days per day. But, now I have more data on that, so I should update the 0.2 figure. Let's see, 0.2 is... (read more)

Thank you, you saved me a lot of typing. No amount of straight copying of that GIF will generate a conscious experience; but if you print out the first frame and give it to a person with a set of rules for simulating neural behaviour and tell them to calculate the subsequent frames into a gigantic paper notebook, that might generate consciousness.

Thanks for replying ! Sorry if the bit I quoted was too short and over-simplified.

That does clarify things, although I'm having difficulty understanding what you mean by the phrase "causal structure". I take it you do not mean the physical shape or substance, because you say that a different computer architecture could potentially have the right causal structure.

And I take it you don't mean the cause and effect relationship between parts of the computer that are representing parts of the brain, because I think that can be put into one-to-one corr... (read more)

That is very interesting; there does seem to be quite rapid progress in this area.

From the blog entry:

... the reason for this is because simulating the neural activity on a Von Neumann (or related computer) architecture does not reproduce the causal structure of neural interactions in wetware. Using a different computer architecture may avert this problem ...

Can anyone explain what that means ? I can't see how it can be correct.

Tattoo private key on inside of thigh.

What's to stop the AI from instead learning that "good" and "bad" are just subjective mental states or words from the programmer, rather than some deep natural category of the universe? So instead of doing things it thinks the human programmer would call "good", it just tortures the programmer and forces them to say "good" repeatedly.

The pictures and videos of torture in the training set that are labelled "bad".

It is not perfect, but I think the idea is that with a large and diverse training set the hope... (read more)

I'm not sure succeeding at number 4 helps you with with the unattractiveness and discomfort of number 3.

Say you do find some alternative steel-manned position on truth that is comfortable and intellectually satisfying. What are the odds that this position will be the same position as that held by "most humans", or that understanding it will help you get along with them ?

Regardless of the concept of truth you arrive at, you're still faced with the challenge of having to interact with people who have not-well-thought-out concepts of truth in a way that is polite, ethical, and (ideally) subtly helpful.

I thought CLARITY was an interesting development - a brain preservation technique that renders tissue transparent. I imagine in the near future there's likely to be benefits going both was from preservation and imaging research.

Buffy / Xander, Motoko / Batu, Deunan / Briareos

(although I'm not sure "Sidekick" is exactly right here)

I thought this sounded familiar

Ah, my mistake, thanks again.

Downvoted for bad selective quoting in that last quote. I read it and thought, wow, Yudkowsky actually wrote that. Then I thought, hmmm, I wonder if the text right after that says something like "BUT, this would be wrong because ..." ? Then I read user:Document's comment. Thank you for looking that up.

I believe this is incorrect. The required proportion of the population that needs to be immune to get a herd immunity effect depends on how infectious the pathogen is. Measles is really infectious with an R0 (number of secondary infections caused by a typical infectious case in a fully susceptible population) of over 10, so you need 90 or 95% vaccination coverage to stop it speading - and why it didn't much of a drop in vaccination before we saw new outbreaks.

R0 estimates for seasonal influenza are around 1.1 or 1.2. Vaccinating 100% of the population with... (read more)

My current rationalisation for my level of charitable giving is "if, say, the wealthiest top billion humans gave as much as me, most of the worlds current problems that can be solved by charity would be solved in short order".

I use this as a labor-saving angst prevention device.

Me: "Am I a good person ? Am I giving too little ? How should I figure out how much to give ? What does my giving reveal about my true preferences ? What would people I admire think of me if they knew ?"

Me: "Extra trillions thing. Get back to work."

1 - All but one of our ships BUILT for space travel that have gone on to escape velocity have failed after a few decades and less than 100 AUs. Space is a hard place to survive in.

Voyagers 1 and 2 were launched in 1977, are currently 218 and 105 AU from the Sun, and are both are still communicating. They were designed to reach Jupiter and Saturn - Voyager 2 had mission extensions to Uranus and Neptune (interestingly, it was completely reprogrammed after the Saturn encounter, and now makes use of communication codes that hadn't been invented when it was ... (read more)

Thanks, that is interesting.

That's not why it's useful. It's useful because it provides liquidity and reduces the costs of trading.

Absent other people getting their trades completed slightly ahead of you, getting your trades completed in a millisecond instead of a second is that valuable ? I'm not being rhetorical - I know very little about finance. What processes in the rest of the economy are happening fast enough to make millisecond trading worthwhile ?

I would have guessed a failure to solve a co-ordination problem. That is, at one time trades were executed on the timescale of ... (read more)

I don't know if there's a name for it. In general, consequentialism is over the entire timeline.

Yes, that makes the most sense.

It seems likely that your post is due to a misunderstanding of my post, so let me clarify. I was not suggesting killing Alice to make way for Bob.

No no, I understand that you're not talking about killing people off and replacing them, I was just trying (unsuccessfully) to give the most clearest example I could.

And I agree with your consequentialist analysis of indifference between the creation of Alice and Bob if they have the same utility ... unless "playing god events" have negative utility.

Is there a separate name for "consequentialism over world histories" in comparison to "consequentialism over world states" ?

What I mean is, say you have a scenario where you can kill of person A and replace him with a happier person B. As I understand the terms, deontology might say "don't do it, killing people is bad". Consequentialism over world states would say "do it, utility will increase" (maybe with provisos that no-one notices or remembers the killing). Consequentialism over world histories would say "the utility contribution of the final state is higher with the happy person in it, but the killing event subtracts utility and makes a net negative, so don't do it".

OK now I have to quote this:

Bernard Woolley: What if the Prime Minister insists we help them?

Sir Humphrey Appleby: Then we follow the four-stage strategy.

Bernard Woolley: What's that?

Sir Richard Wharton: Standard Foreign Office response in a time of crisis.

Sir Richard Wharton: In stage one we say nothing is going to happen.

Sir Humphrey Appleby: Stage two, we say something may be about to happen, but we should do nothing about it.

Sir Richard Wharton: In stage three, we say that maybe we should do something about it, but there's nothing we can do.

Sir Humph

Current beliefs on climate change: I would defer to the IPCC.

I would have first came across the subject while I was at school about 25 years ago (probably not at school, or at least only in passing). I think I accepted the idea as plausible based on a basic understanding of the physics and on scientific authority (probably of science popularisers). I don't remember anyone mentioning quantitative warming estimates, or anyone being particularly alarmist or alarmed.

My current views aren't based on detailed investigation. I would say they are based mostly on (... (read more)

Another text file user. My current system is a log.txt file that has to-do lists at the top, followed by a big list of ideas waiting to be filed, followed by datestamped entries going downwards. That way the next thing to do is right at the top, and I can just cat the date to the bottom to write an entry. I keep this in the home directory on my notebook, but regularly copy it up to my Dropbox. When it gets really long I cut the log part off and save it.

I have another set of files with story ideas, productivity notes, personal thoughts, wildlife sightings, ... (read more)

Rather flogs a dead horse, but highlights an important difference in perspective. You tell your AI to produce paperclips, and eventually it stops and asks if you would like it to do something different.

You could think "hey, cool, its actually doing friendly stuff I didn't ask for", or you could think "wait ... how would knowing what I really want help it produce more paperclips ... "

I agree, I think there is a common part of the story that goes "once connected to the internet, the AI rapidly takes over a large number of computers, significantly amplifying its power". My credence that this could happen has gone way up over the last 10 years or so. Also my credence that an entity could infiltrate a very large number of machines without anyone noticing has also gone up.