All of Celarix's Comments + Replies

I'd say kind of... you definitely have to keep your attention and wits about you on the road, but if you're relying on anxiety and unease to help you drive, you're probably actually doing a bit worse than optimal safety - too quick to assume that something bad will happen, likely to overcorrect and possibly cause a crash.

I'm afraid I don't have the time for a full writeup, but the Stack Exchange community went through a similar problem: should the site have a place to discuss the site? Jeff Atwood, cofounder, said [no](https://blog.codinghorror.com/meta-is-murder/) initially, but the community wanted a site-to-discuss-the-site so badly, they considered even a lowly phpBB instance. Atwood eventually [realized he was wrong](https://blog.codinghorror.com/listen-to-your-community-but-dont-let-them-tell-you-what-to-do/) and endorsed the concept of Meta StackExchange.

I think if you model things as just "an internet community" this will give you the wrong intuitions. 

This, plus Vaniver's comment, has made me update - LW has been doing some pretty confusing things if you look at it like a traditional Internet community that make more sense if you look at it as a professional community, perhaps akin to many of the academic pursuits of science and high-level mathematics. The high dollar figures quoted in many posts confused me until now.

Yeah, that does seem like what LW wants to be, and I have no problem with that. A payout like this doesn't really fit neatly into my categories of what money paid to a person is for, and that may be on my assumptions more than anything else. Said could be hired, contracted, paid for a service he provides or a product he creates, paid for the rights to something he's made, paid to settle a legal issue... the idea of a payout to change part of his behavior around commenting on LW posts was just, as noted on my reply to habryka, extremely surprising.

The amount of moderator time spent on this issue is both very large and sad, I agree, but I think it causes really bad incentives to offer money to users with whom moderation has a problem. Even if only offered to users in good standing over the course of many years, that still represents a pretty big payday if you can play your cards right and annoy people just enough to fall in the middle between "good user" and "ban".

I guess I'm having trouble seeing how LW is more than a (good!) Internet forum. The Internet forums I'm familiar with would have just susp... (read more)

by offering him like $10k-$100k to change his commenting style or to comment less in certain contexts

What other community on the entire Internet would offer 5 to 6 figures to any user in exchange for them to clean up some of their behavior?

how is this even a reasonable-

Isn't this community close in idea terms to Effective Altruism? Wouldn't it be better to say "Said, if you change your commenting habits in the manner we prescribe, we'll donate $10k-$100k to a charity of your choice?"

I can't believe there's a community where, even for a second, having a spe... (read more)

I don't have much to contribute on AI risk, but I do want to say +1 for the gutsy title. It's not often you see the equivalent of "Contra The Founding Mission of an Entire Community".

A lot of good stuff here, especially about the part on being wrong about the emotions having a subtle noticeability. I feel like this supports somewhat tighter cycle times and checking in with subagents more often so one doesn't spend years chasing the wrong ideas.

Thanks for the reply!

This is one of my favorite sequences on this site and I'm quite glad to see a new entry. I do have a question regarding the last section:

Rather, I would suggest opening up to feelings. Becoming familiar with them, understanding where they come from and what they are trying to do, and allowing them to become updated with new evidence and feedback.

How does one gain confidence that the read on their own emotions is an accurate description of the message they're trying to communicate? That is, how can one be more sure that they're actually listening to th... (read more)

Reflection: this adds a lot of extra writing in order to answer the questions. Worth being aware of; adding GPT-4 to your journaling process may increase the time it takes to complete it each day/week/etc.

I might try this with GPT 3.5, as I can't yet justify spending the $20/month on GPT-4. Something like this seems quite interesting!

(oh god we're going to have to be using this near-AGI stuff for everything soon, aren't we?)

3. Needing help with depression, akrasia, or medical advice with confusing mystery illness.

Bit of a shame to see this one, but I understand this one. It's crunch time for AGI alignment and there's a lot on the line. Maybe those of us interested in self-help can go to/post their thoughts on some of the rationalsphere blogs, or maybe start their own.

I got a lot of value out of the more self-help and theory of mind posts here, especially Kaj Sotala's and Valentine's work on multiagent models of mind, and it'd be cool to have another place to continue discussions around that.

There's only one way to save us from AI armageddon. Either the AI mercilessly tears us apart, or we choose our own dignified exit.

Cause vacuum decay. Obliterate the universe. Guarantee that nothing will ever occur again.

COVID at least had some policy handles that the government could try to pull: lockdowns, masking, vaccines, etc. What could they even do against AGI?

I like this question - if it proves true that GPT-4 can produce recognizable ASCII art of things, that would mean it was somehow modelling an internal sense of vision and ability to recognize objects.

The pedant in me wants to say that three tails means 7 (tails, tails, tails > 111 > 1 + 2 + 4 > 7).

EDIT: Ah, now I see you started with 1, so the max value is indeed 8.

The voting response seems a bit harsh to this one; I read it as "where there is opportunity, there will be people selling pickaxes and shovels, and there will also be grifters teaching courses on how to market pickaxes and shovels."

It's like how on days when you're sick or depressed, you think that life is always like this, and you can't remember what it's like to feel happy or healthy, and then a week later when you're happy and healthy, it feels like you've always been that way.

Can confirm. I call it the "Valley of Fog" effect - either you're in the valley (sickness, pandemic) among the sharp rocks and rough terrain and you can't see the sun (happiness, wellness, bustling streets), or you're above the valley and can't see the sharp rocks through all the fog. You remember that things used to be bad but you forget the feelings attached to it.

Also not Zvi, but reducing the death rate from 100% still requires at least some of humanity to survive long enough to recognize those gains. If the AI paperclips everyone, there'd be no one left to immortalize, unless it decides to make new humans down the road for some reason.

With regard to the "does Bob love dogs?" question, is the answer "he's just doing it to make Sandra happy, but I have low confidence in that statement" the right one?

I'd imagine Gerald's "probability 0" is something like Metaculus's "resolved as yes" - that is, the even in question has already happened.

I generally agree, but I think we'd also need to sort out AI alignment while it's asleep. I have no problems with aligned humans and aligned AIs both getting to live.

But, as the last decade+ has shown, alignment is hard. It seems, say, most of MIRI's P(doom) is quite high, and Eliezer thought the task would be so hard that he had to invent/summarize/revive/grow rationality and write the Sequences just to bootstrap enough people into seeing the problem and maybe being able to contribute!

Hence my hardline stance. If Bing Chat gets cleaned up and goes GA, tha... (read more)

The debate around whether LLMs are conscious/sentient or not is not one I want to take a strong opinion on, but I still feel afraid of what follows after Bing Chat.

Note this Twitter thread: https://twitter.com/repligate/status/1612661917864329216 by @repligate. LLMs like ChatGPT and Bing Chat are tuned to "play a character" - that is, I think, the distribution of probabilities over certain words and sequences is changed by humans to avoid certain outcomes. ChatGPT becomes middle management PR drone, Bing Chat becomes... that. I could claim that this is mer... (read more)

Ouch. Bing Chat delenda est, as soon as possible.

It really scares me that Google and Bing felt threatened enough by ChatGPT to put these AI chatbots together for their search engine in just a few months. I don't know if the general AI community has learned a damn thing from all of MIRI's or LW's work on alignment.

Alas, the commands to open a cloud computing account and allocate a million VMs are very low-energy. AI can scale in ways that living things cannot.

Dutch supermarket chain introduces intentionally slow checkout lines so lonely people, especially the elderly, can have a chat. This seems like a good idea for all concerned, while also pointing towards a severe problem that it there is such a need for it.

This sounds awesome. We should totally do this for some of the checklanes here.

This is true, too. Perhaps we can refine it as never interrupt someone when they're doing something you want, unless their reasons would incur other damage - this lets us ban GoF while only caring about taxpayer dollars (true but not the main reason) while avoiding banning GoF for, say, pwning the Democrats (bad if we start doing more things to pwn the Democrats).

We must be vigilant, they remind us, about taxpayer dollars. The important thing about creating the next pandemic is the same as the important thing about preventing the next pandemic, which is making sure our tax dollars do not pay for it.

I think, as a corollary to never interrupt your enemy while they're in the process of making a mistake, we may adopt never interrupt someone when they're doing something you want, even for the wrong reason. Or, at least, maybe "taxpayer dollars" is the good-sounding excuse and they actually want to ban GoF for its world-ending powers and just can't say that directly.

If an AGI wants something from humans it needs to leave us alive and happy enough to produce it

No, I imagine an AGI would have many creative ways to force humans to do what it wants - directly pumping nutrients into your blood, removing neurotransmitters from your brain, overwriting your personality with an incorrigible desire to Do The Thing...

Ants are tiny and hard to find; they could plausibly take your money, defect, and keep eating for a long time before you found them again. Then you need to buy ant poison, anyway.

You know, in a weird sort of way, I think your comment actually makes this more helpful for people who have this impairment in ability. We try so damn hard to "fix" what's wrong with us and are so quick to self-judgment when something doesn't work. By framing this as a description of what is, I think it helps reinforce the idea of not just trying harder via application of more force, more self-hatred, etc.

(p.s. I saw your reply to my comment about subagents which want bad things and really appreciate it. I'm still trying to process it; you should see a reply soon)

That, uh, is a good question. Now I'm not sure myself.

I think what I was going for is the idea that, yes, subagents matter, but no, you're not always going to be able to use these methods to get better at dealing with them. So don't feel too bad if you have a condition that renders this more difficult or even impossible.

One question I've wanted to ask about subagents: what should you do if you determine that a subagent wants something that's actually bad for you - perhaps continuing to use some addictive substance for its qualities in particular (rather than as a way to avoid something else), being confrontational to someone who's wronged you, or other such things?

In other words, what do you do if the subagent's needs must be answered with no? I don't know how that fits in with becoming trustworthy to your subagents.

There are basically four problems with forcing instead of listening:

I will note that, sometimes, the connection between "you" and the part of your brain you're trying to communicate with is that the connection isn't great and that part's memory is terrible - this is the heart of executive dysfunction.

It does not seem like cherry-picking to then ignore things that are not diseases. That is the category.

It seems some of the stuff filtered out were cancer and heart disease, both of which I'd call diseases.

I don't check my priors at the door. I am >99% confident that heroin and drugs like it are deeply bad for you, and it's up to the research to prove that I should update.

I'd like to push back a bit against the downsides of being overconfident, which I think you undersell. Investing in a bad stock could lose you all your investment money (shorting even more so). Pursuing an ultimately bad startup idea might not hurt too much, unless you've gotten far enough that you have offices and VC dollars and people who need their paychecks. For something like COVID, mere overstocking of supplies probably won't hurt, but you'll lose a lot of social clout if you decide to get to a bunker for something that may end up harmless.

Risk is ri... (read more)

Deploying UVC to disinfect large spaces might be infeasible, but would it be easier to have smaller UVC lamps inside ventilation ducts and let the air pass under them at a higher rate? You get a much closer lamp, much more airflow, and don't have to expose anyone to UVC directly.

Not sure if this is helpful, but I feel like something like this happens within me when I try to change the current thing I'm doing (get ready for work, go to bed, what have you). My elephant really doesn't like changing gears, and it can be quite difficult to follow through.

The elephant's response is mostly a strong wave of negative emotions. The emotions are similar to ones where you might be told about a new pile of work you have to do, not the ones you might have when encountering something repulsive. Something like "ughhhhh", not "ewwww". Parsing out ... (read more)

Makes sense, thanks.

I mostly want to let 1000 flowers bloom on this

I am not sure what you mean by this. Do you mean this comment is great and worth the recommendation, or should be buried so that its corpse produces a thousand flowers? Genuinely unsure.

I don't know that we would have the political will to clearcut the Amazon and switch out our crop supply, even in the face of >70% death by starvation. Political polarization is very high right now. When either side proposes anything, the other will oppose it, even if it's saving humanity.

You might rightly say "but starvation is a powerful motivator!" and it is, but the people doing the starving won't be the people who could move the crops - the farmers and the world's lumber industry, who would be foiled by politics at every turn. The starving people will be too hungry to really do anything.

So maybe not 5 billion dead, but I wouldn't be surprised at about 3 billion.

Ancient creation myths filled the universe with human figures – jealous lovers and loving fathers and whatnot. Later it turned out these myths were more like mirrors than telescopes, and only by ditching them in favor of real telescopes and the cold abstractions of mathematics could we make progress.

Excellent saying.

This is in the heart of wine country where grapes grow in abundance and wheat waves like golden seas- but not now. Now the wheat burns and the grapes whither to raisins on the vine. This is the end of days. And on Monday morning I'll return to work and pretend this isn't happening. It's complete madness.

This seems like a pretty common pattern in argument and debate, which I'll tentatively call "piggybacked claims" - make a claim with some evidence ("this river's dry, it's really rare, here's a picture"), then add on additional claims that may logically fol... (read more)