winstonBosan's Shortform

winstonBosan

winstonBosan's Shortform — LessWrong

winstonBosan's Shortform

13th Feb 2022

1 min read

2

This is a special post for quick takes by winstonBosan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

30 comments, sorted by

top scoring

Click to highlight new comments since: Today at 12:35 AM

[-]winstonBosan3mo6310

Gwern has retired from fulltime writing + pseudonymity and is now working full time on Guardian Angel - definitely didn't have this on my bingo card.

@Gwern Update: I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel Inc and bring GAs to life. We are looking for good people. If you are interested, contact me.

[-]Thane Ruthenis3mo*216

Not sure how I feel about this.

Going off Gwern's introductory post, the idea is that these Guardian Angels would learn their principals' preference and personalities to a very high degree of faithfulness, that as part of this "faithfulness" they would be as creative as the principals (rather than being mode-collapsed), that they would continually update those preferences over time, that you'd be able to trust them to operate autonomously on long complex tasks, and that they'd be capable enough to represent the principal's interests against hostile increasingly smarter external AIs.

That is,

Improving training sample-efficiency is explicitly one of the main research goals.
Improving conceptual resolution is explicitly one of the main research goals.
Improving creativity/ability to innovate is explicitly one of the main research goals.
Achieving a form of continual learning is explicitly one of the main research goals.
The underlying model having long time horizons is a must.
The underlying model having high general capabilities is a must.

Gwern does provide the doom story to which this is the natural answer, but, like... Yeah, I can very much see a world in which this just becomes another part of the problem.

This seems like exactly the type of research that is well-intentioned and motivated by a plausible-sounding story about how it's differentially more helpful for alignment (making models better aligned to their principals), but which then fails to achieve sufficient degree of alignment (I don't expect to be able to trust these GAs, trained as stated, to faithfully represent me in any remotely complicated situation; see one argument here) while still making a bunch of progress on general capabilities that then gets co-opted for yet higher p-of-doom (inasmuch as any of those ideas work, I expect them to be pretty dual-use).

... At least, if the idea works well enough for this startup's models to achieve frontier-level capabilities. And without that, I expect them to be mostly useless.

Well, now that I wrote it all out, I suppose I did figure out how I should feel about this.

Is this an unfair characterization? That this is Gwern's project does make me want to give it the benefit of the doubt, but on its merits, it sure doesn't look good.

[-]Richard_Kennaway3mo100

For those not in the very select group of Gwern's Twitter followers: "Guardian Angels: LLM Personalization for Productivity and Security".

[-]_rpd3mo712

aka Friendly Shoggoths

[-]anaguma3mo50

author: Gwern, Ethan Roland, Zeke Reffe-Hogan, GBT-1-397b

Who/what is GBT-1-397b?

[-]Adele Lopez3mo173

My guess is that it stands for "Gwern Branwen Transformer" and is a proof of concept (and likely a finetune of Qwen 3.5 397B A17B).

[-]Richard_Kennaway3mo62

I'm wondering who Ethan Roland and Zeke Reffe-Hogan are. There are google hits, but none of the candidates look very plausible.

[-]DW113mo40

Zeke Reffe-Hogan is listed as working for this non-profit, which I've never heard of even though it has Kelsey Piper as a director: https://philanthropy.org/990/report/922979832/ai-governance-project

Ethan Roland is probably this guy? https://ethan-w-roland.com/

[-]winstonBosan3mo30

My speculation and guess that Zeke is Ezekiel Reffe-Hogan - of AI governance project. Hunch based on their association with Kelsey Piper, generally associated with AI stuff etc

[-]Mateusz Bagiński3mo40

Is he implying that his legal name is actually Gwern Branwen? Or did he change his legal name to match his pseudonym?

[-]DW113mo64

Gwern Branwen is a name combining figures from Welsh mythology so it's vanishingly unlikely to be his real name. I'm guessing he'll drop the pseudonym once this project progresses.

[-]Mo Putera3mo20

He might, but I hope he doesn't.

[-]Dean Valentine3mo3-7

Seems good for the world.

[-]Dean Valentine3mo32

Interested in learning the disagree-reasoning? I don't think they're going to accelerate the frontier; they're just tuning existing open weight models so that regular people can trust them. Governance seems like it could get way better, and most of the research is literally alignment research.

[-]winstonBosan3mo64

I didn't vote disagree, but my concern is that, if this were to work, it is just gradual dis-empowerment by another name and something very whispering earring shaped.

[-]Richard_Kennaway3mo71

Is disempowerment a convergent result of having above-human AI at all, with a Butlerian jihad the only way out?

[-]Thane Ruthenis3mo20

See there.

[-]winstonBosan1mo120

Who are the merchants of cope for AI?

So far, the one that comes to mind is Arvind Narayanan. Who else are the greatest peddler of normalcy in our age?

[-]Komodo1mo160

There's a few major camps

It's all sci-fi speculation, it's going to be fine.

Cal Newport

It's not agents it's just prompt loops. AI's are just pattern matching from all the scifi stories when they talk about "permadeath". It is not misalignment, long running unsupervised prompt loops should be expected to run amok. The reasoning traces aren't actually reasoning. AI companies in an effort to make models better when scaling slowed just decided to reward LLM's for outputting "reasoning traces" but it's not actual reasoning, it's just spitting random stuff out because it was rewarded for it. source: Chain-Of-Thought Reasoning In The Wild Is Not Always Faithful. Misalignment is not a real thing, there's no entity behind the screen. It's a dog with a weed wacker on it running around.

Let me just give you a blue-sky thought experiment. Imagine we basically said you can’t have long-horizon, LLM-driven, actuated agents, programs whose actions are largely determined by the results of LLM prompts, without human supervision. The way you would do this is by imposing a depth limit. That type of system could do, say, half a dozen prompts unsupervised, and that’s it. I don’t know if that’s the right number, but something like that. So it could do a few prompts to try to solve a very immediate thing you asked it to do, but none of this long-horizon stuff. Just make that illegal.

Yann LeCun

LLM's are nowhere close to human level intelligence and will take us nowhere close, AGI will not cause mass unemployment or a "singularity". Technological progress will hit limits and not just shoot off into infinity.

Ed Zitron

LLM's have some uses but are gimmicks and nowhere close to real AGI. They rely on massive amounts of money to make benchmark number go up, while in practice they remain unreliable. and not very useful All the companies are passing money around to pump up the numbers and the entire funding scheme is smoke and mirrors.

Something might go really really bad, but we won't go extinct

Gary Marcus

Generally the historic view was LLM's are good at some things, but we need neurosymbolic reasoning for true AGI, LLM's are nowhere close. His view is shifting though. In his eyes current models are actually neurosymbolic (harnesses are neurosymbolic, Astra making algebraic notation on the fly to solve puzzles is neurosymbolic reasoning).

and in another post talking about why he doubts extinction he said

they reduce human beings to helpless stick figures, we are not. they also abstract away from military strategy and the reality that no military attack against humanity on the scale doomers imagine would ever go perfectly on the first time, which means humans would launch mighty counterattacks

which is a take I guess. I saw a similar take from Andrew McAfee (AI won't be able to take out all of humanity, we are not that weak, I saw geohot say similar things in his debate with yudkowsky on the dwarkesh pod)

I think the Gary Marcus view is gaining traction (it poses large dangers, but don't really believe in catastrophic risks). I see it with Ezra Klein and Bill Gates. I could definitely find some quotes from them but it's late at night.

AI is going to transform the world and doomers are just pessimists

Accelerationists

A lot of these guys don't even really think we'll solve alignment. From there it ranges from just being happy to be along for the ride (accelerate harder subset), just happy to be along for the ride (can't do anything about it so may as well enjoy it subset).

Also a sector think that alignment is solvable, a sector think the AI would keep us around because we would be useful (thinking it would kill us all for some mysterious reason is insane pessimism), a sector think as it becomes smarter it would be kind, a sector think that since it starts from human data, it is like a average of humans and wouldn't be a weird alien.

A sector are just successionists.

Lot's of views under this camp

Jensen Huang / Marc Andreessen

AI is normal technology that is going to transform the world. Arguments about doom are just pessimists just trying to slow us down. Alignment is an engineering problem that can be solved, and AI is built, designed, and controlled by us. I think the high level view is the same, Marc is just more singularity pilled.

Accelerationists and Huang-Andreessen exist on a sort of spectrum, sort of two axes, one is the future will be normal / singularity and the other is AI is software / this is an alien.

Huang-Andreessen is defo a view I see with ambitious high functioning go getters. Singularity pilled accelerationists I don't really see much in reality (Mike Israetel but nobody else really comes to mind).

This is probably sloppily written it's late at night but I think I covered all the major stuff.

[-]DW111mo31

Timnet Gebru is still flying the flag for the original Marcus view that LLMs aren't intelligent at all.

[-]I.M.J. McInnis1mo42

Canonically Ed Zitron, whom I actually recommend every deep LWer read, if only for resistance training.

[-]winstonBosan1mo98

Absolute banger (also opus 5.5 is a good model yes sir)

https://x.com/donaldjewkes/status/2102801274173587569?s=46

[-]CstineSublime1mo10

I kind of like how the digital inking of the orange on the main singer is imperfect shudders and is imperfect. It makes it feel somewhat analogy.

[-]winstonBosan7d80

Zuckerberg’s released deposition about politicians been exempted from fact checking:

https://m.youtube.com/watch?v=C7UC2ie0XIE&ra=m

i had a lot of fun watching it. despite it been dry and pretty monotonic, Zuckerberg is definitely trained about how to answer without exposing himself, and how the plaintiff/deposing attorney sets up her questions in a seemingly obv and repetitive manner to ready her snares is skillful.

[-]Viliam7d31

"You want to increase the value of your company 500 times, is that right?" "No." "Here is the interview where you said that." "That says to increase the value by 500%, that's not the same thing." :)

[-]winstonBosan2mo30

I wondered how the Muslim world viewed the fall of Baghdad to the mongols.

Googling it would take a few seconds, and asking Claude would likely fetch me something more coloured. But I prefer to keep this one away from them, because this is not a fact finding mission. I wanted to roll the thoughts in my tongue like a lollipop and pretend that I could taste the bloody Tigris. Christians bones and Muslim bones, carelessly toss'ed, to depths light never shone.

I prefer to keep this one inaccurate, just for a bit.

[-]winstonBosan2mo30

I have recently acquired a cat, and now she drinks from a tall beer glass.

Tabitha had her stainless water dish. As all curious creatures would, she saw me and my partner drinking from tall glasses, so she started drinking from our glasses and would hiss at us should we try to take it away. Hearing her complaints, and on the grounds of egalitarianism and pan-species cooperation, I got my cat an beer glass and all was well.

Lately, she noticed that the magic was not in the glass but in it being elevated on a desk. Water somehow tastes better on top of a veneered IKEA table. As all unprincipled man would, I relented and gave more grounds so now my cat dines with us in high fashion.

[-]winstonBosan5y30

Today, there is a distinct lack of faith.
This lack of faith has permeated all pairs of 16oz denims, event merch hoodies and smartwool socks. The lack of faith is insidious, bad, and all things molochian.

Actually, screw this manifesto writing style. I am about to meta criticize myself for a style that has NOT aged well and tends to be way too authoritarian in tone but measly in effective information communicated.

Actually, screw this meta-criticism. In all good faith, I didn't have to write the criticism. I could have simply done the internal self-criticism and just deleted what I have written to start anew. Instead, I have now left it as a slightly well-disguised monument to my apparent self-improved and evolution. But really, just to stroke my ego and fill the word count.

Actually, have I written anything of content yet? Have I put my pen to paper and allow the ink to betray my trust when my back is turned? (also known as drying)

Oh no... Better start counting the trains that leave Datong for the trail of coal crumbs.

Real Content Soon™

[-]winstonBosan2mo20

The Spire is Neow.

According to the epochs, the Architect have created the Ancients as an improvement over the native flora and fauna on the planet. All of the ancients, Neow included, started out as tiny seeds and grew into different trees shaped by its surroundings. Eventually, as a tree reached full maturity, an ancient is birthed from the tree.

Neow was jealous of the Architect. So she devised the Blight to incapacitate the Architect to grow her own tree. Captured by her peers and then out cast, Neow's tree fused with the Blight and robbed the land of its nourishment, until it too lost its branches and foliage and became the Spire we know today.

[-]winstonBosan2y10

Could it be worth it to buy 23andme stocks if you want some of their user's data?

Naively, the sticker price of 75M USD (today's market cap) for all of their user data might seem cheap - all together for genomes of roughly 12 million users. It seems reasonably cheap to me on a replacement cost and opportunity-cost basis.

However, the 49% basically-majority shareholder is CEO Anne Wojcicki and a "possibility is that Wojcicki has unreasonable plans to take the company private at a bargain-basement price^[1]". If you takes this path forward as a decidedly important but minority shareholder, the only clear way forward is to hope for a shareholder oppression lawsuit in your favour. And instead of cashing out, you hope you can turn your skill set of "being annoying enough" into data in-lieu of cash (Note: if this is done through bankruptcy process, I can't see the trustee agreeing to this plan.)

Even given the obstacles, is this a deal? Or is this just a bad price for this particular bag of information? Is 23andme's information sufficiently low fidelity that it might not be useful at all except to relate yourself to fractional vikings?

^{^}
Per Matt Levine - Money Stuff, Sep 19, 2024

[+][comment deleted]2y10

Moderation Log

It's all sci-fi speculation, it's going to be fine.

Cal Newport

Let me just give you a blue-sky thought experiment. Imagine we basically said you can’t have long-horizon, LLM-driven, actuated agents, programs whose actions are largely determined by the results of LLM prompts, without human supervision. The way you would do this is by imposing a depth limit. That type of system could do, say, half a dozen prompts unsupervised, and that’s it. I don’t know if that’s the right number, but something like that. So it could do a few prompts to try to solve a very immediate thing you asked it to do, but none of this long-horizon stuff. Just make that illegal.

Yann LeCun

Ed Zitron

Something might go really really bad, but we won't go extinct

Gary Marcus

and in another post talking about why he doubts extinction he said

they reduce human beings to helpless stick figures, we are not. they also abstract away from military strategy and the reality that no military attack against humanity on the scale doomers imagine would ever go perfectly on the first time, which means humans would launch mighty counterattacks

AI is going to transform the world and doomers are just pessimists

Accelerationists

A sector are just successionists.

Lot's of views under this camp

Jensen Huang / Marc Andreessen

Accelerationists and Huang-Andreessen exist on a sort of spectrum, sort of two axes, one is the future will be normal / singularity and the other is AI is software / this is an alien.

This is probably sloppily written it's late at night but I think I covered all the major stuff.

[-]DW111mo31

Timnet Gebru is still flying the flag for the original Marcus view that LLMs aren't intelligent at all.

[-]I.M.J. McInnis1mo42

Canonically Ed Zitron, whom I actually recommend every deep LWer read, if only for resistance training.

[-]winstonBosan1mo98

Absolute banger (also opus 5.5 is a good model yes sir)

https://x.com/donaldjewkes/status/2102801274173587569?s=46

[-]CstineSublime1mo10

I kind of like how the digital inking of the orange on the main singer is imperfect shudders and is imperfect. It makes it feel somewhat analogy.

[-]winstonBosan7d80

Zuckerberg’s released deposition about politicians been exempted from fact checking:

https://m.youtube.com/watch?v=C7UC2ie0XIE&ra=m

[-]Viliam7d31

"You want to increase the value of your company 500 times, is that right?" "No." "Here is the interview where you said that." "That says to increase the value by 500%, that's not the same thing." :)

[-]winstonBosan2mo30

I wondered how the Muslim world viewed the fall of Baghdad to the mongols.

I prefer to keep this one inaccurate, just for a bit.

[-]winstonBosan2mo30

I have recently acquired a cat, and now she drinks from a tall beer glass.

[-]winstonBosan5y30

Actually, have I written anything of content yet? Have I put my pen to paper and allow the ink to betray my trust when my back is turned? (also known as drying)

Oh no... Better start counting the trains that leave Datong for the trail of coal crumbs.

Real Content Soon™

[-]winstonBosan2mo20

[-]winstonBosan2y10

Could it be worth it to buy 23andme stocks if you want some of their user's data?

^{^}
Per Matt Levine - Money Stuff, Sep 19, 2024

[+][comment deleted]2y10

Moderation Log