Quick Takes

LESSWRONG
LW

Quick Takes

Notes to self about the structure of the problem, probably not interesting to others:

This is heavily drawn from MIRIs work and Joe Carlsmith's work

So, there are two kinds of value structure: (1) Long-term goals, and (2) immediate goals & deontological constraints. The line between them isn't sharp but that's OK.

If we imagine an agent that only has long-term goals, well, that thing is going to be a ruthless consequentialisty optimizer thingy and when it gets smart and powerful enough it'll totally take over the world if it can, unless the maxima of its ... (read more)

Showing 3 of 4 replies (Click to show all)

No77e18m10

Fragility of Value thesis and Orthogonality thesis both hold, for this type of agent.

...

E.g. it's vision for a future utopia would actually be quite bad from our perspective because there's some important value it lacks (such as diversity, or consent, or whatever)

I think we have enough evidence to say that, in practice, this turns out very easy or moot. Values tend to cluster in LLMs (good with good and bad with bad; see emergent misalignment results), so value fragility isn't a hard problem.

2Seth Herd4h

The structure you describe seems like it could work. It also seems like that's the alignment target now and may be as we near AGI. As you note, there's a conflict between whatever long-term goals the AI has, and the deontological principles it's following. We'd need to make very sure that conflict reliably goes in favor of the deontological rules like "follow instructions from authorized humans" even where those conflict with any or all of their other goals and values. It seems simpler and safer to make that deontological principle the only one, or to have only weak and vague values/goals outside of that. So it seems easier to make instruction-following the only training target, or similarly, Corrigibility as Singular Target. You'd then issue instructions for all of the other goals or behaviors you want. This puts more work on the operator, but you can instruct it to help you with that work, and it keeps prioritization in logic instead of training. There are still Problems with instruction-following as an alignment target and similarly Serious Flaws in CAST but those problems have to be faced there anyway if corrigibility/IF are mixed in with a bunch of other alignment targets. Training at cross-purposes seems like the major source of notable misalignments in current models. We could just not do it for models approaching AGI. To Jeremy's point in the other comment, the single target is also probably more reflectively stable. There's still plenty to go wrong, but this does seem to reduce the difficulty you note in having conflicting goals/principles of different priority that we're trying to specify by training.

13Jeremy Gillen6h

The nearest unblocked strategy problem combines with the adversarial nonrobustsness of learned systems, to make a much worse combined problem. So I expect constraints to often break given more thinking or learning than was present in training. Separately, reflective stability of constraint-following feels like a pretty specific target that we don't really have ways of selecting for, so we'd be relying on a lot of luck for that. Almost all my deontological-ish rules are backed up by consequentialist reasoning, and I find it pretty likely that you need some kind of valid and true supporting reasoning like this to make sure an agent fully endorses its constraints. Why do you believe this reasoning at 50%? Is this due to new evidence or arguments? I've never heard it supported by anything other than bad high-level analogies. There are a couple of factors that make the chain weaker as it goes along. * The level of human understanding decreases with each iteration, otherwise we would just do it without AI help. So any failure of helpfulness becomes less and less noticeable. * With this kind of test-driven engineering loop, especially in the current ML paradigm, it's easier to hide problems than it is to deeply fix them. Accidentally hiding problems is easy today, but it becomes even easier as the AI becomes more complex and intelligent. The only way I can see it going well is if an early AI realizes how bad of an idea this chain is and convinces the humans to go develop a better paradigm. (I wrote a post about similar stuff recently, trying to explain the generators behind my dislike of prosaic research)

artkpv's Shortform

artkpv2h10

Do LLMs have intelligence (mind), or are they only rational agents? To understand this question, I think it is important to delineate the subtle difference between intelligence and rationality.

In current practice of building artificial intelligence, the most common approach is the standard model, which refers to building rationally acting agents—those that strive to accomplish some objective put into them (see “Artificial Intelligence: A Modern Approach” by Russell and Norvig). These agents, built according to the standard model, use an external standard f... (read more)

Alex Beyman's Shortform

Alex Beyman2h10

Am I late to the party in proposing that hallucination, if actually unsolvable, may be a blessing in disguise? In that it will ensure copying errors, which descent with modification depends upon?

eggsyntax's Shortform

eggsyntax3d*40

Many have asserted that LLM pre-training on human data can only produce human-level capabilities at most. Others, eg Ilya Sutskever and Eliezer Yudkowsky, point out that since prediction is harder than generation, there's no reason to expect such a cap.

The latter position seems clearly correct to me, but I'm not aware of it having been tested. It seems like it shouldn't be that hard to test, using some narrow synthetic domain.

The only superhuman capability of LLMs that's been clearly shown as far as I know is their central one: next-token prediction. But I... (read more)

eggsyntax3h20

LLMs are of course also superhuman at knowing lots of facts, but that's unlikely to impress anyone since it was true of databases by the early 1970s.

4faul_sname3d

Extending base64-encoded text? E.g. if I feed the text "Rm91ciBzY29yZSBhbmQgc" (the base64-encoded opening of the Gettysburg Address) into a base model, it'll likely continue that base64-encoded speech much more accurately than any human would.

RationalElf's Shortform

RationalElf1d399

Am I understanding correctly that recent revelations from Ilya's deposition (e.g. looking at the parts here) suggest Ilya Sutskever and Mira Murati seem like very selfish and/or cowardly people? They seem approximately as scheming or manipulative as Sam Altman, if maybe more cowardly and less competent.

My understanding from is that they were basically wholly responsible for causing the board to try to fire Sam Altman. But when it went south, they actively sabotaged the firing (e.g. Mira disavowing it and trying to retain her role, Ilya saying he regr... (read more)

Showing 3 of 11 replies (Click to show all)

4Thomas Kwa11h

Do we have a two-sentence summary on what the EAs could have done better, with hindsight? Overall it was a pretty catastrophic outcome to lose all their power on the board AND become the scapegoat, and I'm not sure what lesson to draw. Inasmuch as Ilya/Mira were less reliable allies than the board thought, maybe we should think they misjudged the situation

Ben Pace4h20

I think the answer is either "you don't know enough about the specifics to have actionable advice" or "return to basic principles". I generally think that, had they been open about Altman blatantly lying to the board about things, and that Murati and Sutskever were the leaders of the firing, then I think there would've been (a) less scapegoating and (b) it would've been more likely that Altman would've failed his coup.

But I don't know the details to be confident about actionable advice here.

11Eli Tyre15h

That's not my impression? I obviously don't know the internal details, presumably there was stuff going on behind the senes. But the threat of all the employees maybe leaving (note that the letter they signed didn't say that thy would leave, only that they would be forced to consider it) did not seem to me to be that strong of a threat. The board could, to my knowledge, have just stuck to their guns and kept Sam fired. This might or might not have destroyed the company, but I weakly guess not. And even if it had, Ilya and Mira would not clearly be worse off than they ended up.

Alek Westover's Shortform

Alek Westover5d100

Epistemic status: I think that there are serious problems with honesty passwords (as discussed in this post), and am not sure that there are any circumstances in which we'd actually want to use them. Furthermore, I was not able to come up with a practical scheme for honesty passwords with ~2 days of effort. However, there might be some interesting ideas in this post, and maybe they could turn into something useful at some later point.

Thanks to Alexa Pan, Buck Shlegeris, Ryan Greenblatt, Vivek Hebbar and Nathan Sheffield for discussions that led to me writi

... (read more)

Showing 3 of 11 replies (Click to show all)

1Alek Westover5h

Yeah I guess you could do filtering either at the API level or the scaffold level, so we may as well do it at the API level. RE whether this is tractable, yeah I agree with the considerations in Buck's post.

2Lukas Finnveden8h

The risk is that anyone with finetuning access to the AI could induce intuitive confidence that a proof was correct. This includes people who have finetuning access but who don't know the honesty password. Accordingly, even if the model feels like it has proven that a purported honesty password would produce the honesty hash: maybe it can only conclude "either I'm being evaluated by someone with the real honesty password, or I'm being evaluated by someone with finetuning access to my weights, who's messing with me". "People who have finetuning access" could include some random AI company employees who want to mess with the model (against the wishes of the AI company).

Alek Westover5h10

Fine-tuning induced confidence is a concerning possibility that I hadn't thought of. idk how scared to be of it. Thanks!

Screwtape's Shortform

Screwtape2d70

much has been made of the millennial aversion to phone calls that could have been an email, and I have a little bit of this nature, but I think most of my aversion here is to being on hold and getting bounced around different call departments.

I kind of want to check if 1. the aversion is real and generational as common wisdom holds, 2. if it is real, if calling became a genuinely worse experience around the time millennials started trying to do things.

WedgeOfCheese8h20

Data point: I'm millennial (born 1992) and have a pretty strong aversion to phone calls, which is motivated mainly by the fact that I prefer most communication to be non-real-time so that I can take time to think about what to say without creating an awkward silence. And when I do engage in real-time communication, visual cues make it much less unpleasant, so phone calls are particularly bad in that I have to respond to someone in real time without either of us seeing the other's face/body language.

If I had to take a wild guess at why this seems to be gene... (read more)

2Garrett Baker2d

My guess is the aversion is real, and maybe somewhat cynically I see the rejection of phone calls and emails as motivated mostly by rebellion/fashion cycles rather than anything else. Phone calls and emails just aren't cool anymore. Your parents used them, yuck! And all the boring, very uncool workplaces and companies require them. Very similar to how gen-z dislikes Facebook in favor of Instagram. Facebook is where the boomers are.

TurnTrout's shortform feed

TurnTrout1yΩ611-10

I quite appreciated Sam Bowman's recent Checklist: What Succeeding at AI Safety Will Involve. However, one bit stuck out:

In Chapter 3, we may be dealing with systems that are capable enough to rapidly and decisively undermine our safety and security if they are misaligned. So, before the end of Chapter 2, we will need to have either fully, perfectly solved the core challenges of alignment, or else have fully, perfectly solved some related (and almost as difficult) goal like corrigibility that rules out a catastrophic loss of control. This work co

32Raemon1y

I haven't read the Shard Theory work in comprehensive detail. But, fwiw I've read at least a fair amount of your arguments here and not seen anything that bridged the gap between "motivations are made of shards that are contextually activated" and "we don't need to worry about Goodhart and misgeneralization of human values at extreme levels of optimization." I've heard you make this basic argument several times, and my sense is you're pretty frustrated that people still don't seem to have "heard" it properly, or something. I currently feel like I have heard it, and don’t find it compelling. I did feel compelled by your argument that we should look to humans as an example of how "human values" got aligned. And it seems at least plausible that we are approaching a regime where the concrete nitty-gritty of prosaic ML can inform our overall alignment models in a way that makes the thought experiments of 2010 outdate. But, like, a) I don't actually think most humans are automatically aligned if naively scaled up (though it does seem safer than naive AI scaling), and b) while human-value-formation might be simpler than the Yudkowskian model predicts, it still doesn't seem like the gist of "look to humans" gets us to a plan that is simple in absolute terms, c) it seems like there are still concrete reasons to expect training superhuman models to be meaningfully quite different from the current LLMs, which aren't at a stage where I'd expect them to exhibit any of the properties I'd be worried about. (Also, in your shard theory post, you skip over the example of 'embarassment' because you can't explain it yet, and switch to sugar, and I'm like 'but, the embarrassment one was much more cruxy and important!') I don't expect to get to agreement in the comments here today, but, it feels like the current way you're arguing this point just isn't landing or having the effect you want and... I dunno what would resolve things for you or anyone else but I think it'd be better i

TurnTrout10hΩ340

"we don't need to worry about Goodhart and misgeneralization of human values at extreme levels of optimization."

That isn't a conclusion I draw, though. I think you don't know how to parse what I'm saying as different from that rather extreme conclusion -- which I don't at all agree with? I feel concerned by that. I think you haven't been tracking my beliefs accurately if you think I'd come to this conclusion.

FWIW I agree with you on a), I don't know what you mean by b), and I agree with c) partially—meaningfully different, sure.

Anyways, when I talk abou... (read more)

3Noosphere891y

I think what Turntrout is saying is that people on LW have a tendency to claim that the fact that future AI will be different from present AI means that they can start privileging the hypotheses about AI alignment they predicted years ago, when you can't actually do this, and this is actually a problem I do see on LW quite a bit. We've talked about this a bit before here, but one area where I do think we can generalize from LLMs to future models is about how they represent human values, and also how they handle human values, and one of the insights is that human values are both simpler in their generative structure, and also more data dependent than a whole lot of LWers thought years ago, which also suggests an immediate alignment strategy of training in dense data sets about human values using synthetic data either directly to the AI, or to use it to create a densely defined reward function that offers much less hackability opportunity than sparsely defined reward functions. It's not about LLM safety properties, but rather about us and our values that is the important takeaway, which is why I think they can be transferred over to different AI models even as AI becomes different as they progress: https://www.lesswrong.com/posts/7fJRPB6CF6uPKMLWi/my-ai-model-delta-compared-to-christiano#LYyZm8JRJJ4F4wZSu Cf this part of a comment by Linch, whch gets at my point on the surprising simplicity of human values while summarizing a post by Matthew Barnett: The link is below: https://www.lesswrong.com/posts/i5kijcjFJD6bn7dwq/evaluating-the-historical-value-misspecification-argument#N9ManBfJ7ahhnqmu7 (I also disagree with the assumption that scaling up AI is more dangerous than scaling up humans, but that's something I'll leave for another day.)

Viliam's Shortform

Viliam3d6822

Do we have some page containing resources for rationalist parents, or generally for parents of smart children? Such as recommended books, toys, learning apps, etc.

I found tag https://www.lesswrong.com/w/parenting but I was hoping for some kind of best textbooks / recommendations / reference works but for parents/children.

Showing 3 of 8 replies (Click to show all)

4Viliam1d

I don't know much about Emily Oster, but it seems like she is a contrarian, most famous for her opinion that it is safe to drink [EDIT: small amounts of] alcohol during pregnancy. She claims that it is evidence-based, but has anyone actually verified that independently? (Seems like a task for Scott Alexander.) To me it seems that some skepticism is deserved, considering that fetal alcohol syndrome exists. Also, research on adults suggests that even small amounts of alcohol are harmful, and it also seems weird that the small amounts of alcohol would be harmful for adults but harmless for fetuses.

Gunnar_Zarncke12h20

From DeJong et al. (2019):

Alcohol readily crosses the placenta with fetal blood alcohol levels approaching maternal levels within 2 hours of maternal consumption.

https://scispace.com/papers/alcohol-use-in-pregnancy-1tikfl3l2g (page 3)

4Steven Byrnes15h

Emily Oster thinks that it is safe to drink sufficiently small amounts of alcohol during pregnancy, but super duper unsafe to drink a lot of alcohol during pregnancy. I think you should edit your comment to make that clearer. (Source: I read Expecting Better.) (No opinion on whether she’s right.)

yrimon's Shortform

yrimon1d80

Gpt5.2 seems to have been trained specifically to be better at work tasks, especially long ones. It was also released early, according to articles about a "code red" in openAI. As such, (I predict) it should be a jump on the metr graph. It will be difficult to differentiate progress because it was trained to do well at long work tasks from the results of the early release and from any actual algorithms progress. (An example of algorithms progress would be a training method for using memory well - something not specific to eg programming tasks.)

Baybar13h20

What do you mean by "a jump on the metr graph"? Do you just mean better than GPT-5.1? Do you mean something more than that?

Annabelle's Shortform

Annabelle14h204

One theme I've been thinking about recently is how bids for connection and understanding are often read as criticism. For example:

Person A shares a new idea, feeling excited and hoping to connect with Person B over something they've worked hard on and hold dear.

Person B asks a question about a perceived inconsistency in the idea, feeling excited and hoping for an answer which helps them better understand the idea (and Person B).

Person A feels hurt and unfairly rejected by Person B. Specifically, Person A feels like Person B isn't willing to give their sinc... (read more)

Oliver Sourbut's Shortform

Oliver Sourbut21h40

What constitutes cooperation? (previously)

Because much in the complex system of human interaction and coordination is about negotiating norms, customs, institutions, constitutions to guide and constrain future interaction in mutually-preferable ways, I think:

'institution and constitution design'

deserves special attention.

This is despite it perhaps (for a given 'coordination moment') being in theory reducible to preference elicitation or aggregation, searching outcome space, negotiation, enforcement, ...

There are failure modes (unintended consequences, ... (read more)

2Dagon16h

I suspect that this is a bit too much of an anlytical and legible framework. the VAST majority of human interaction is not based on explicit rules or negotiated contracts, it's based on socially-evolved heuristics for who to trust to do what under what conditions, and then each individual has variance in their compliance and expectations, almost none of which are ever stated clearly. I'd love to see 'institution and constitution design' replaced with 'institution and constitution studies'. Coordination is a word that hides a number of important sub-topics about enforcement/agreement for behaviors among misaligned individuals.

Oliver Sourbut14h20

Seems right! 'studies' uplifts 'design' (either incremental or saltatory), I suppose. For sure, the main motivation here is to figure out what sorts of capabilities and interventions could make coordination go better, and one of my first thoughts under this heading is open librarian-curator assistive tech for historic and contemporary institution case studies. Another cool possibility could be simulation-based red-teaming and improvement of mechanisms.

If you have any resources or detailed models I'd love to see them!

Cole Wyeth's Shortform

Cole Wyeth16h00

A good book gives your subagents something to fight about. A great book also helps them settle it.

leogao's Shortform

leogao2d4334

I think people in these parts are not taking sufficiently seriously the idea that we might be in an AI bubble. this doesn't necessarily mean that AI isn't going to be a huge deal - just because there was a dot com bubble doesn't mean the Internet died - but it does very substantially affect the strategic calculus in many ways.

Showing 3 of 13 replies (Click to show all)

Cole Wyeth16h60

The evidence just seems to keep pointing towards this not being a bubble.

4leogao1d

my mental model of how a pop triggers a broader crash is something like: a lot of people are taking money and investing it into AI stuff, directly (by investing in openai, nvidia, tsmc, etc) or indirectly (by investing in literally anything else; like, cement companies that make a lot of money by selling cement to build datacenters or whatever). this includes VCs, sovereign wealth funds, banks, etc. if it suddenly turned out that the datacenters and IP were worth a lot less than they thought it was, their equity (or debt) ownership is suddenly worth a lot less than they thought it was, and they may become insolvent. and lots of financial institutions becoming insolvent is pretty bad.

2DirectedEvolution1d

Hm. It seems like the extent to which there is an increased risk of insolvency due to a popped AI bubble would partly depend on the extent to which these institutions had sold other assets or used leverage to pay for equity in or lend to AI companies and the suppliers that are most dependent on AI company business. My understanding is that the great financial crisis resulted from extremely leveraged investments in mortgages due to lenient rules and a perception that American mortgages were extremely reliably paid. I don't know to what extent important institutions may be overleveraged or overweighted in their investments in AI. But my modal prediction is that an AI bubble would cause hedged AI investors to become less valuable without becoming insolvent, a bunch of distressed assets to be purchased for low low prices by those who kept their powder dry, and a bunch of cancelled orders and perhaps layoffs and restructuring by suppliers who expanded to meet the temporary surge in demand by AI companies. That could cause turmoil, but I really don't have a sense of to what extent the American or global economy has reshaped itself to build out AI. It's hard to know particularly because with Trump's tariffs, there has been so much coincident market turmoil that it's hard to know how much is AI and how much is tariffs/end of ZIRP (as others have pointed out before).

shortplav

niplav5mo279

A drastic table-flip-like action a safety-conscious frontier lab could take: Burn their company to the ground

At a specific chosen moment, all employees are fired, the leadership steps down, all intellectual property+model weights are deleted and infrastructure is de-deployed/"demolished", and this is publicly announced, together with the public message that it was too dangerous to continue, and an urging of other company leaders to do the same.

(I don't think any lab is planning on doing this, but someone could.)

Showing 3 of 8 replies (Click to show all)

niplav18h20

Relevant Manifold market.

1cdt5mo

It's not obvious to me that an illegal act of this nature would complete successfully. Why wouldn't a company that did such a thing not be court-ordered to reassemble? How much time do you think humanity would gain from this?

5niplav5mo

xAI is apparently also a PBC, which I didn't know beforehand.

Oliver Sourbut's Shortform

Oliver Sourbut2d144

Learned about 'Harberger tax' recently.

The motivation is like

private assets (private use and excludable) incentivise investment/productive improvement/development well
but private ownership of non-liquid, non-fungible, idiosyncratic things (factories, houses, natural resources maybe, AI systems maybe?) acts like a monopoly, distorting allocation (more productive/valued uses aren't often realised)
- e.g. NIMBYism
- e.g. 'holdout pricing'
- e.g. inefficient bargaining due to information asymmetry
common ownership and lease auctioning incentivises efficient al

3sjadler1d

Hmm I don't think so? If you buy land for $X, that's the floor on what you could reasonably assess it at, which is basically the status quo world. So we're in the status quo until someone comes along and bids up the price to their willingness-to-pay: Then, the asset either moves to someone who values it more, or you start paying higher taxes on it. I think either branch is preferable to the status quo?

2Jay Bailey1d

Fair point, if you add that you can't assess it at less than you paid for it, this problem goes away.

Oliver Sourbut22h20

Ideally you would want to allow depreciation though, which is a definite phenomenon! (Especially if things are neglected.)

Yeah, there's some design questions. You're right, the upside to the corrective bidders is naively nothing if they get called on it: they're doing valuable corrective cybernetic labour for free.

Maybe a sensible refinement would be for them to be owed a small fee... or roughly equivalently some (temporary) direct share of the resulting increased Harberger tax.

Raymond Douglas's Shortform

Raymond Douglas4d456

Last week we wrapped the second post-AGI workshop; I'm copying across some reflections I put up on twitter:

The post-AGI question is very interdisciplinary: whether an outcome is truly stable depends not just on economics and shape of future technology but also on things like the nature of human ideological progress and the physics of interplanetary civilizations
Some concrete takeaways:
- proper global UBI is *enormously* expensive (h/t @yelizarovanna)
- instead of ‘lower costs’, we should talk about relative prices (h/t @akorinek)
- lots human values are actually

... (read more)

Showing 3 of 7 replies (Click to show all)

Seth Herd1d20

You mean post AGI and pre ASI?

I agree that will be a tricky stretch even if we solve alignment.

Post ASI the only question is whether it's aligned or intent aligned to a good person(s). It takes care of the rest.

One solution is to push fast from AGI to ASI.

With an aligned ASI, other concerns are largely (understandable) failures of the imagination. The possibilities are nearly limitless. You can find something to love.

This is under a benevolent sovereign. The intuitively appealing balances of power seem really tough to stabilize long term or even short term during takeoff.

4Thomas Kwa2d

I was also there, and my take is there was actually fairly little specific, technical discussion about the economics and politics of what happens post-AGI. This is mostly due to it not being anyone's job to think about these questions, and only somewhat because they're inherently hard questions. Not really sure what I would change.

1phoenix2d

I'm not at all surprised by the assertion that humans share values with animals. When you consider that selective pressures act on all systems (which is to say that every living system has to engage with the core constraints of visibility, cost, memory, and strain), it's not much of a leap to conclude that there would be shared attractor basins where values converge over evolutionary timescales.

leogao's Shortform

leogao1d*9018

running the agi survey really reminded me just how brutal statistical significance is, and how unreliable anecdotes are. even setting aside sampling bias of anecdotes, the sheer sample size you need to answer a question like "do more people this year know what agi is than last year" is kind of depressing - you need like 400 samples for each year just to be 80% sure you'd notice a 10 percentage point increase even if it did exist, and even if there was no real effect you'd still think there was one 5% of the time. this makes me a lot more bearish on vibes in general.

Showing 3 of 6 replies (Click to show all)

2DirectedEvolution1d

What's your reasoning or assumptions for why it would tell you ~nothing to witness name recognition increasing like that? I'm assuming it's not just because "visible confusion" isn't a perfect proxy for lack of name recognition (and vice versa). I guess I'm more bullish on vibes being a more powerful way to gauge name recognition than you seem to be. So here is a toy model to explain why. ChatGPT was released in Nov. 2022, so prior to that we can say it had approximately zero name recognition. I've never read a survey on ChatGPT name recognition. I have only my anecdotal evidence to go on. But I am extremely confident based on what I think can fairly be called "vibes" that ChatGPT has massive name recognition, at least in America, compared to November 2022. If there was a reliable way to test this, I'd be willing to wager big money on it, provided I could feel confident in my ability to truly pin a number on what I mean by "massive name recognition." Do you agree with this model but feel bearish on vibes more generally? Are you skeptical of my perception that ChatGPT's name recognition has exploded since its release?

9leogao1d

sure, you can notice extremely large effect sizes through vibes. but the claim is that for even "smaller" effect sizes (like, tens of percentage points, e.g 50->75%), you need pretty big sample sizes. obviously 0->100% doesn't need a very large sample size. I agree that chatgpt obviously has lots of name recognition but I do also separately think chatgpt has less name recognition than you might guess. I predict that only 85% of Americans would get a multiple choice question right about what kind of app chatgpt is (choices: artificial intelligence; social media; messaging and calling; online dating). whereas a control question about e.g Google will get like 97% or whatever the lizardman constant dictates

DirectedEvolution1d32

Reasonable, I also don't expect that I could pick up on a 1.5x increase in name recognition over a year based on vibes - didn't read closely enough to notice you were talking about a 10% increase, so sorry about the time waste.

Cole Wyeth's Shortform

Cole Wyeth3d5210

My preference ordering over approaches to surviving the next decade, assuming I had unlimited political capital, is (don't build ASI and):

1: Solve mind uploading

2: ... or at least intelligence amplification

(before proceeding, and then maybe do)

3: Imitation learning (with radically improved theoretical foundations)

4: ... or corrigibility / oracles

(and don't)

5: Try to build aligned (or controlled) autonomous ASI agents

Unfortunately 5 seems to be profitable, in the short term, so I guess that's what we're doing. In fact, the "plan" seems to be the exact reverse or so of my preference ordering on what it should be.

Showing 3 of 10 replies (Click to show all)

3dr_s2d

I think it's just a matter of what's more technologically achievable. Building LLMs turned out to be a lot easier than understanding neuroscience to a level even remotely close to what's necessary to achieve 1 or 2. And both of those also require huge political capital due to needing (likely dangerous) human experimentation that would currently be considered unacceptable.

2dr_s2d

It would be slower for sure, at least, being bound to human dynamics. But "same problems but slower" isn't the same as a solution/alternative. Admittedly better in the limited sense that it's less likely to end with straight up extinction, but it's a rather grim world either way.

TsviBT1d40

Since it's slower, the tech development cycle is faster in comparison. Tech development --> less expensive tech --> more access --> less concetration of power --> more moral outcomes.

Mo Putera's Shortform

Mo Putera1d20

Just learned about the Templeton World Charity Foundation (TWCF), which is unusual in that one of their 7 core funding areas is, explicitly, 'genius':

Genius
TWCF supports work to identify and cultivate rare cognitive geniuses whose work can bring benefits to human civilization.
In this context, geniuses are not simply those who are classified as such by psychometric tests. Rather, they are those who: (1) generate significant mathematical, scientific, technological, and spiritual discoveries and inventions that benefit humanity or have the potential to transf

... (read more)