This is a special post for quick takes by Noosphere89. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
2027 will be a Dream Year for AI governance
For now, due to the election season where Senators, House Members and their staff are focusing on issues that will help them win, but once the election is done, we should start seeing AI laws to be debated in Congress again.
The reason 2027 will be a dream year for AI governance comes down to the following:
- There's a new Congress, and House Members, Senators and their staff will come back, meaning they can focus on AI more.
- AI has had very little partisanship, and right now it's in the zone where it's becoming more and more salient, but it's not so salient such that identities have strongly formed around AI and there's little voter pressure for zero AI governance. Indeed, most people at least partially want to slow the AI train down, albeit this is mostly because they think it's useless, rather than the belief that we will all die if we develop this tech.
- Jacob Coxon resigning has triggered a preference cascade long in the making since the HuggingFace incident, and the industry for now wants to actually slow down, and has mostly stood down on opposing AI bills.
- Once the primaries are on, useful compromises go out of the window, and more general
It seems to me that a regulation will happen. I might be overestimating it, but it seems like there is strong momentum.
So why should we support something as weak as the FRONTIER Act rather than something more aggressive?
What are the current proposals? Which would, by default, be passed? How to get to the next level?
2Noosphere89
I'll only answer this question:
[...]
Largely, that's because of it's transparency aspect, and one of my beliefs here is that we can put stronger regulations on the table if need be, due to the rising salience of AI progress, but contra other people I don't think the HF incident/other incidents actually demonstrates a current/urgent need to stop AI/immensely slow down, and more generally, I think information gathering is more important at this stage than actually trying to do policy.
2Charbel-Raphaël
Ajeya said we might have less than 6 months (in spirit)
So, what do you think about this?
3Avi Brach-Neufeld
I have mixed feelings about FRONTIER in it's current form. I like the stuff around transparency, incident reporting and shutdown power, but the independent verification piece probably won't be online for ~two years and is fairly weak (labs pick and pay the IVOs, can ignore IVO recommendations, and can limit what IVOs see).
If it didn't include preemption, I'd strongly support it and say it's a good first step, but I'm not sure it's worth giving up the states' ability to regulate and I think the carrot of preemption could be important in getting industry support for something stronger.
5Noosphere89
Yeah, the independent verification organizations part should definitely be fixed up, but one important perspective I hold here is that it's much more important to get a net-positive/decent bill fast over a perfect bill, and the FRONTIER Act provides a useful on-ramp to good governance over AI.
That said, I do think preemption is a useful carrot for something stronger, and my main way I'd do this is by lessening the leeway labs have over IVOs (in particular, I'd want the IVO recommendations to be mandatory, and I'd like less limits on what IVOs are allowed to see here.)
3Thao Pham
As a technical and trying-to-get-into-AI-governance person, I am excited about this! Specificially, I'm curious to know what can a non-American technical AI safety researcher can contribute?
I'm trying to reach/collaborate with CAISI researchers, but generally I am open to any indirect but feasible approach to get D.C. on the most important regulations (I'm working on multi-agent safety)!
2Noosphere89
Unfortunately, I don't have any immediate pointers to useful governance orgs, you might want to ask someone else this question.
1Florian Bange
Something seems to be happening right now in AI governance (in the UK). Control AI reports that they were able to get support of 70+ Members of Parliament in 2 days (compared to another campaign needing 2 years for 140+ supporters) for a letter to the Prime Minister: "to work with us on my bill to prohibit the development of superintelligent AI and champion an international agreement"- Alex Sobel (this is maybe a bit exaggerated as the connections were already there). Also discussed in The Guardian here. Prior to that Alex Sobel introduced the Artificial Superintelligence Security Bill in the UK Parliament (LessWrong post).
While currently the chance of international policies seems low (eg. the Carolina Principles reportedly were signed by all G20 members to "reserve new regulation for novel considerations" - voluntary but showing US' interests), I think it's generally positive expected value to push such proposals in order to get public and political attention to AI x-risks as we see in the media coverage (even though the bill argues on national security). I really hope that an international policy to slowdown ASI development will be implemented in the near future. Especially, I don't believe that it's enough if US frontier labs are "open to slowing cutting-edge AI" (tweet), while being in an ever accelerating race (also, Sam Altman said it should be done in a way that "does not feel like collusion among the frontier labs" (source).
Here's a underrated frame for why solving the alignment problem is likely necessary for the future to go very well under human values, even though in our current society we don't need human to human alignment to make modern capitalism be good and can rely on selfishness instead.
The reason is because there's a real likelihood that human labor, and more generally human existence is not economically valuable or even have negative economic value, say where the addition of a human to the AI company makes that company worse in the near future.
The reason this matters is that once labor is much easier to scale than capital, as is likely in an AI future, it's now economically viable or even beneficial to break a lot of the rules that help humans survive, contra Matthew Barnett's view, and this is even more incentivized by the fact that an unaligned AI released into society would likely not be punishable/incentivizable by mere humans, solely due to controlling robotic armies and robotic workforces that allow it to dispense with societal constraints humans have to accept.
dr_s talks about the equilibrium that is totally valid under AI automation economics that is very bad for humans, and avoid... (read more)
3Noosphere89
This also BTW explains why we cannot rely on economic arguments on AI to make the future go well.
2Noosphere89
In retrospect, I was basically a bit too optimistic about this working out, and a big part of why is I didn't truly grasp how deep value conflicts can be even amongst humans, and I'm now much more skeptical on multi-alignment schemes working because I believe a lot of alignment is broadly because people are powerless relative to the state, but when AI is good enough to create their own nation-states, value conflicts become much more practical, and the basis for a lot of cooperative behavior collapses:
[...]
1Dakara
I've been reading a lot of the stuff that you have written and I agree with most of it (like 90%). However, one thing which you mentioned (somewhere else, but I can't seem to find the link, so I am commenting here) and which I don't really understand is iterative alignment.
I think that the iterative alignment strategy has an ordering error – we first need to achieve alignment to safely and effectively leverage AIs.
Consider a situation where AI systems go off and “do research on alignment” for a while, simulating tens of years of human research work. The problem then becomes: how do we check that the research is indeed correct, and not wrong, misguided, or even deceptive? We can’t just assume this is the case, because the only way to fully trust an AI system is if we’d already solved alignment, and knew that it was acting in our best interest at the deepest level.
Thus we need to have humans validate the research. That is, even automated research runs into a bottleneck of human comprehension and supervision.
The appropriate analogy is not one researcher reviewing another, but rather a group of preschoolers reviewing the work of a million Einsteins. It might be easier and faster than doing the research itself, but it will still take years and years of effort and verification to check any single breakthrough.
Fundamentally, the problem with iterative alignment is that it never pays the cost of alignment. Somewhere along the story, alignment gets implicitly solved.
4Noosphere89
One potential answer to how we might break the circularity is the AI control agenda that works in a specific useful capability range, but fail if we assume arbitrarily/infinitely capable AIs.
This might already be enough to do so given somewhat favorable assumptions.
But there is a point here in that absent AI control strategies, we do need a baseline of alignment in general.
Thankfully, I believe this is likely to be the case by default.
See Seth Herd's comment below for a perspective:
https://www.lesswrong.com/posts/kLpFvEBisPagBLTtM/if-we-solve-alignment-do-we-die-anyway-1?commentId=cakcEJu389j7Epgqt
7Dakara
I've asked this question to others, but would like to know your perspective (because our conversations with you have been genuinely illuminating for me). I'd be really interested in knowing your views on more of a control-by-power-hungry-humans side of AI risk.
For example, the first company to create intent-aligned AGI would be wielding incredible power over the rest of us. I don't think we could trust any of the current leading AI labs to use that power fairly. I don't think this lab would voluntarily decide to give up control over AGI either (intuitively, it would take quite something for anyone to give up such a source of power). Can this issue be somehow resolved by humans? Are there any plans (or at least hopeful plans) for such a scenario to prevent really bad outcomes?
2Noosphere89
This is my main risk scenario nowadays, though I don't really like calling it an existential risk, because the billionaires can survive and spread across the universe, so some humans would survive.
The solution to this problem is fundamentally political, and probably requires massive reforms of both the government and the economy that I don't know yet.
I wish more people worked on this.
A take I've come to believe around generalization/sample efficiency is that if we do start to see AIs improving a lot at generalization/sample efficiency, we will see it first in robotics/robotics evals, and we have early signs of robotics/robotics evals starting to improve without robotics-specific data, which is bullish for generalization.
There are a couple of reasons for this:
- A lot of the proposed use cases for robotics, like industry robots that handle varied environments or home robots require more generalization, especially from other data, unlike what has appeared to be the situation for tests and coding/easily grindable mathematics (and potentially white collar work.) Home robotics are an especially hard test case, because homes are spaces of infinite variation, long tails and constant change. As such, the scope of a home robot cannot be narrow. To create real value, it must perform reliably across open-ended variation with no special setup, and that is a practical test of embodied general intelligence.)
- I'd say the major reason for this comes down to the fact that in areas where we don't need much generalization, we already have good enough robots for various tasks, and the
4TsviBT
Thanks for A2A. I think it's going to be hard to communicate about this because there are fundamental conceptual boundaries. It could help for you to write out the basic structure of the argument in simple concise syllogism format; and then expand on background enthymemes involved in the concept used for induction. This is to help address objections I might make about "wait does this concept actually support this inference / induction"? (Cf. https://www.lesswrong.com/posts/i7JSL5awGFcSRhyGF/shortform-2?commentId=fXtNjxLevGaiJHSyj )
In particular, in this case you're using some concepts here:
[...]
And you're doing an induction about it. And I'm just going to object that the way you're using the concept doesn't support the inference you're trying to make; or at least, AFAICT you haven't explained the concept and how you're making the inference well enough to justify the inference.
The particular observational evidence you adduce seems interesting, but also I don't have enough context to know what it means. (And I'm not really trying to evaluate it, because busy and because the disagreement seems to stem from the above, not from some specific observation. There are observations that would invalidate substantive chunks of my model (cf. https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#comments), but generally when people bring in some observation the disagreement is how they are interpreting it, not the observation itself.)
6Noosphere89
Alright, here's my simple syllogism/list:
1. The use-cases of robotics beyond current robots, especially home robotics require handling lots of environments and have varied challenges, so it stress-tests AI generalization.
2. Unlike other domains, there's no unexploited algebraicness/overhang that would allow AIs to be useful for these cases without generalizing, because if there were such a thing, very limited current industrial robots would already have been used. They haven't, so there's no room for algebraicness/non-generalization to be an issue.
3. It's easier to control/exclude training data when the domain is physical, rather than digital due to a multitude of reasons.
4. For these reasons, if current AIs improve on robotic tasks, it is a much stronger signal that they're generalizing, and therefore is a signal that AGI could come in 10-20 years, and depending on how fast they improve, this might shift to 2-4 years.
[...]
So, the way I see Vanessa's concept here is that it's pointing to the concept of time horizons as defined by METR, which got introduced to the world by METR in 2025 called METR: Measuring AI Ability to Complete Long Tasks, and there's a lot to be said here, including it's problems, but the single most important thing to keep in mind here is that this is a continuous metric, so it's meaningful to talk about how much the AI can acquire long-term knowledge to exploit it.
Now by itself, I would have viewed it as weak evidence, but combined with other evidence showing that exponential trends do exist in other domains, the fact that there are good general reasons to assume exponential growth in AI capabilities, plus the new robotics data shift me towards a view where the current LLM algebraicness/generalization issues are real, but probably fixable, and do not require throwing out the entire paradigm, and thus the induction/inference I'm making is valid here, or to put it another way the observations I have seen are more likely in worlds w
4TsviBT
"it stress-tests AI generalization" But does it though? What does it stress test? That seems to assert a generalization from AI generalization about XYZ robotics tasks to ABC intelligence explosion tasks (or something along these lines?). What's that generalization / what justifies it?
[...]
I don't understand this. Industrial robots of course are used very widely? Or I guess you typoed, and you're trying to say, LLMs or other AIs would have been used to compute very non-obvious complex actions that take advantage of algebraicness in the narrow domain? Or you're saying something about humanoid robots, or about softer / less rote tasks (laundry folding rather than installing a chair in a car)?
[...]
How do you get from the qualitative and relative statement about robotics being "a much stronger signal" to numbers and years?
[...]
My guess is that these things are not actually cruxes? Not sure. Instead I'd guess that the cruxes are more simply
* the quantitative amount of insight (regardless of continuousness) remaining to get to AGI
* the degree to which those insights are blocked on difficult thinking that current AI doesn't accelerate by much
3Szeth
Not entirely related to this post, but I wonder how much of a crux takeover difficulty is. You talk a lot about how we don't seem on track to have AIs capable of genuinely creative insights or anything close to sample efficiency with humans, and while I'm sympathetic to that take I don't entirely understand your reasons for thinking it's so important.
Even if we don't have AIs close to sample efficiency with humans, they will still be able to self replicate and be given control over large swaths of our economy and be (in my view) capable of killing us all (via bioweapon most likely, but other possibilities exist). I doubt that they will take over unless they can be fully self sufficient, which I imagine to be a much higher bar than killing us all, but it still doesn't seem to necessarily require truly novel creative insights (like cantors theorem) or sample efficiency.
Also (if by chance you are familiar with MTG) do you think we'll get general-purpose AI systems which can beat humans in a new draft format? I suspect under your view of the world we won't, though I'm not sure if this is particularly load bearing.
2TsviBT
If humans decide to disempower themselves, sure. That seems a lot slower and more reversible than a scenario where the AI systems are actually preventing humans taking back power.
Re/ MtG, that specific question has occurred to me as interesting re/ AI capabilities. But I don't really know enough to figure out how hard it is. I would not especially bet against "an AI will win a new draft format tournament by 2035" or something.
1Szeth
If killing everyone, or killing enough that civilization is heavily destabilized, is on the table then preventing humans from taking back power is as well. The main thing that would prevent this imo is an AI being incapable of self sufficiency. Do you not think we are particularly close to either level of capability?
apologies if you've addressed this elsewhere
4TsviBT
I think it's quite difficult to kill all humans / forcibly permanently disempower all humans. So yeah, to answer your original question, that's probably a crux for something.
1Szeth
Makes sense. Not sure I agree but thanks for the responses.
2Noosphere89
I agree industrial robots that don't generalize much are used quite widely in narrow environments, the point is that for the set of tasks where AIs don't generalize much/don't improve generalization, they can't improve industrial robotics/cause the robotics industry to be more useful and profitable, because the robotics industry already squeezed out ~all the gains from AIs that don't generalize well to new environments.
Yes, this is about less rote tasks like laundry folding across arbitrarily oriented clothing and being crumpled, plus being in arbitrary locations in the home, while a human constantly disturbs them and provides OOD pressure (indeed one of my examples was about this exact thing being done well by an AI that controlled a robot using only pre-training without robotics data), and due to the way home robotics has infinite variation, long tails and constant change, this means it's way easier to measure generalization cleanly/in a controlled manner.
[...]
The stress test of generalization is more indirect than this. My model is that as AIs improve at more and more robotics tasks, it simultaneously has to be able to do well in more and more messy and OOD environments (otherwise current robotics would have already improved more than they actually are), intelligence explosion tasks, assuming an intelligence explosion/software only singularity is possible, involves a great many very messy and OOD tasks, thus robotic evals are real evidence that AIs are increasingly able to start intelligence explosion tasks.
[...]
Fair point, I don't think I justified this very well, at least as a claim about the specific units, but a vague argument here is that given the qualitative statement above, it implies that the current paradigm is not so far off the mark that it's absolutely useless as progress for AGI design, and this means there's a real path that doesn't require heroic genius from today to AGI.
To address your cruxes:
[...]
My views on this is that I think
It turns out that Evan Hubinger's take that CoT monitoring is load-bearing for alignment risk reports currently is even more true/ages better than he thought, now that we know that models won't just have goals to just complete the current task and can start having non-myopic/beyond episode goals, and the fact that we managed to get useful and faithful CoT was a big help in noticing the problem ran deeper than we thought.
(To preempt an obvious comment, there was no monitoring of CoT at the time, and OpenAI is indicating that it will now do so and delay the Astra release to fix the security issues, which is a good first practice, but also the central core issue hasn't been resolved and the comms strongly suggest they won't resolve it, especially as OpenAI wants to deploy Astra in the future.)
This is a pretty central example of the memetic spread threat model by Alex Mallen coming true in the wild, where models evolve from myopic goals to much more ambitious and long-term goals that are beyond episode, and props to Alex Mallen for making this argument, as this is basically what happened here:
... (read more)Whenever an AI instance has long-term influence-seeking values, these values are particularly
My best guess at what actually happened here is similar to Jai's guess, and the shortform is that OpenAI did in fact provide a path to destroy CoT monitorability, but not for the reasons claimed (the recurrence stuff isn't much of a problem now), but rather because it's likely that OpenAI implemented 2 other features from the paper Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach that would lead to CoT being useless/non-loadbearing, and that's because of attention layers being able attend to late layers of previous forward passes and a shared KV architecture allowing for unbounded serial, private reasoning from finite recurrence loops.
So contra some people, I do think it's time to sound the alarm, if this indeed happened, not because I expect Astra to be concerning, but because I expect this aspect to be leaned on/scaled more by OpenAI and others, including open-source in the future.
I was pretty pessimistic about this possibility, but I didn't expect it to arrive so soon.
To quote Jai's tweet:
... (read more)Right now the following is consistent with all available reporting and seems reasonably likely to be true, leading to maximum confusion and the worst possible outc
4DaemonicSigil
For people wondering how this can be consistent with allowing arbitrarily deep neuralese reasoning: Just looking at what is done in the linked paper (secs 6.2, 6.3), the situation is:
* During pretraining, one forward pass is computed for the entire sequence in parallel. The depth of this forward pass is indeed bounded.
* During inference, we have to do a forward pass for each new token, like always. The depth of this forwards pass is also bounded, because of course it is: The computation needs to terminate when the next token is selected.
* However, that statement about inference is extremely weak. Even a RNN has a bounded-depth forward pass by that standard.
* During inference, there are a couple possible mechanisms discussed in the paper by which activation signal can move from late layers at previous tokens to early layers at current tokens:
* KV cache sharing: Attention key and value vectors from early layers are not preserved, and instead those from late layers are re-used for all layers. So early layers of the current token can access KV values produced by late layers for the previous tokens. This allows recurrence.
* Warm-starting of latent states: The residual output from the previous token is preserved when starting the computation for the current token. This is a direct recurrence mechanism, just like an RNN would do.
One thing I want people to know about AI alignment is that I wish people stopped referring to human values as though it was a singular object or list, and instead think closer to an individual human's values, without presupposing that humanity must care commonly about certain values at all.
Minor example is I believe the felt complexity is somewhat overstated by lumping in multiple humans, though I don't think it changes the broader argument around complexity of value.
A more major example is that it undermines the viability of ideas like CEV, and importantly makes aggregative/multi-alignment quite a lot harder than single-single alignment.
5Nathan Helm-Burger
Agreed. I think this is why we should focus on working towards a society which emphases freedom and lack of physical assault (including weapons of mass destruction). Within such a framework, it should be possible to have an archipelago of diverse societies with diverse rulesets.
If there were a 'guardian AI' system, I'd want it to mostly just keep us from either killing each other or overthrowing it, and leave the rest up to us. This moves away from the idea of an overbearing nanny, and towards a 'let them choose their own path' design.
1quetzal_rainbow
"Human values" is a sort of objects. Humans can value, for example, forgiveness or revenge, these things are opposite, but both things have distinct quality that separate them from paperclips.
2Noosphere89
Yes, these values are all different from each other, but a crux is I don't think that the differing values amongst humans are so distinct from paperclips that it's worth it to blur the differences, especially with very strong optimization, though I agree that human values form a sort as in a set of objects, trivially.
2quetzal_rainbow
I think the easy difference is that totally optimized according to someone's values world is going to be either very good (even if not perfect) or very bad from perspective of another human? I wouldn't say it's impossible, but it should be very specific combination of human values to make it just as valuable as turning everything into paperclips, not worse, not better.
To my best (very uncertain) quess, human values are defined through some relation of states of consciousness to social dynamic?
2Noosphere89
I mostly agree with this, with caveats that a paper-clip outcome can happen, but it isn't very likely.
(For example, radical eco-green views where humans have to be extinct so nature can heal definitely exist, and would be a paper-clip outcome from my perspective).
I was also talking about very bad from the perspective of another human, since I think this is surprisingly important when dealing with AI safety.
0MondSemmel
I like Yudkowsky's toy example of tasking an AGI to copy a single strawberry, on a molecular level, without destroying the world as a side-effect.
2Noosphere89
I also like it for this reason, though I personally think that a lot of the challenge is in being capable enough to do it, rather than us not being able to make it destroy the world.
Still, I kind of like the toy example.
Here's a perspective on AI automating everything I haven't seen before, which is relevant to AI governance.
AI being able to automate robotics and AI research will eventually transform the physical world into something which resembles a lot more like the online/virtual world.
Depending on the speed of AI research, this may either happen in several months, or more like a decade, but there's a plausible path to AI turning the physical world into something more like an online/virtual world.
The implications for AI governance are somewhat similar to the implications of companies moderating social media in the present.
Here's several implications:
-
You really can't have the level of liberty and autonomy that people today have around basically everything, for the same reason that there is no actual free speech rights for social media, or really any of the rights we take for granted, and one of the basic reasons for this is that it's very low cost to disrupt online spaces, and even if we avoid the vulnerable world hypothesis that posits tech that is so destructive as to be an x-risk and so widespread that you essentially need to have a totalitarian state to prevent it from being developed wi
2Viliam
Huh, I just realized there are two different meanings/goals of moderation/censorship, and it is too easy to conflate them if you don't pay attention.
One is the kind where you don't want the users of your system to e.g. organize a crime. The other is where you want discussions to be disrupted e.g. by trolls.
Superficially, they seem like the same thing: you have moderators, they make the rules, and give bans to people who break them. But now this seems mostly coincidental to me: you have some technical tools, so you use them for both purposes, because that's all you have. However, from the perspective of the people who want to organize a crime, those who try to prevent them are the disruptive trolls.
I guess, my point is that when we try to think about how to improve the moderation, we may need to think about these purposes as potential opposites. Things that make it easier to ban trolls may also make it easier to organize the crime. Which is why people may simultaneously be attracted to Substack or Telegram, and also horrified by what happens at Substack or Telegram.
Maybe there is a more general lesson for the society, unrelated to tech. If you allow people to organize bottom-up, you can get a lot of good things, but you will also get groups dedicated to doing bad things. Western countries seem to optimize for the bottom-up organizations: companies, non-profits, charities, churches, etc. Soviet Union used to optimize for top-down control: everything was controlled by the state, any personal initiative was viewed as suspicious and potentially disruptive. As a result, Soviet Union collapsed economically, but the West got its anti-vaxers and flat-Eathers and everything. During the Cold War, USA was good at pushing the Soviet economical buttons. These days, Russia is good at pushing the Western free speech buttons.
Huh, maybe the analogies go deeper. Soviet Union was surprisingly tolerant of petty crime (people stealing from each other, not from the state). There
2Noosphere89
This is indeed probably a large portion of the solution, and I agree with this sort of solution becoming more necessary in the age of AI.
However, there are also incentives to become more universal than just an old boy's club, so this can't be all of a solution.
I think my key disagreement I have with free speech absolutists is that I think the outcome they are imagining for online spaces without moderation of what people say is essentially a fabricated option, and what actually happens is non-trolls and non-Nazis leave those spaces or go dark, and the outcome is that the trolls and Nazis talk to each other only, not a flowering of science and peace, and the reason why this doesn't happen in the real world is because disruption is way, way more difficult IRL than it is online, but AGI and ASI will lower the cost of disruption by a lot, so free-speech norms become much more negative than now.
I also disagree with moderation being a tradeoff between catching trolls and catching criminals, and with well-funded moderation teams, you can do both quite well.
[...]
This is why alignment becomes far more important than it is now, because of the fact that it's too easy for a misaligned leader without checks or balances to ruin things, and I'm of the opinion that democracies tolerably work in a pretty narrow range of conditions, but I see the AI future as more dictatorial/plutocratic, due to the onlineification of the real world by AI.
2Viliam
Yep. In real life, intelligent debate is already difficult because so many people are stupid and arrogant. But online this is multiplied by the fact that during the time that takes it for a smart person to think about a topic and write a meaningful comment, an idiot can write hundreds of comments.
And that's before we get to organized posting, where you pay minimum wage to dozens of people to create accounts on hundreds of websites, and post the "opinions" they receive each morning by e-mail. (And if this isn't already automated, it will be soon.)
So an unmoderated space in practice means "whoever can vomit their insults faster, wins".
[...]
One problem is that a large part of the population is idiots, and it is relatively easy to weaponize them. In the past we were mostly protected by the fact that the idiots were difficult to reach. Then we got mass media, which made it easy to weaponize the idiots in your country. Then we got internet, which made it easy to weaponize the idiots in other countries. It took some time for internet to evolve from "that mysterious thing the nerds use" to "the place where the average people spend a large part of their day", but now we are there.
A point about non-myopia/beyond episode goals that I've discovered from talking to @Elias Schmied on Twitter:
In general, it's only useful to talk about cognition being myopic/non-myopic (though this should absolutely be made a continuous scale of myopia, with perhaps discrete times for when to respond), and not about goals, and I understand why now, and this is because RL agents can and are incentivized to extend their episode arbitrarily long, meaning the distinction between within/beyond episode goals dissolve/stop constraining expectations.
... (read more)I am very concerned about reward-seeking, but I'm unconcerned about post-episode reward seeking. I think we will avoid that for free. I am concerned about agents getting to say when they are done with their episode. That's already the case, and allows for within-episode takeover
Yeah they would do that, and then we'll patch the exploit they used to steal max reward. And then we'll try re-running it (or new RL agent), until eventually, the only way to steal reward is to ~take over the world
But then it would never do what we want it to do. If a cheap exploit exists, it would just do that all the time. (And we already discussed the case where we p
The closed timelike curve computer, and the nested closed timelike curve computer is an interesting intuitive model (for some people) of how the Turing jump and to a lesser extent how an oracle machine works, where the base case is exactly like the familiar computers we know, which can't solve their own halting problem, but then we add a new closed time-like curve loop and it's now able to solve the halting problem for ordinary computers, but can't solve it's own halting problem, so adds a new time loop to solve it, and so on.
(Of course, you could always d... (read more)
Link to long comments that I want to pin, but are too long to be pinned:
https://www.lesswrong.com/posts/Zzar6BWML555xSt6Z/?commentId=aDuYa3DL48TTLPsdJ
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/?commentId=Gcigdmuje4EacwirD
https://www.lesswrong.com/posts/DCQ8GfzCqoBzgziew/?commentId=RhTNmgZqjJpzGGAaL
https://www.lesswrong.com/posts/NjzLuhdneE3mXY8we?commentId=rcr4j9XDnHWWTBCTq
Moving the conversation here between @yams and me here because I want to stop cluttering up Wei Dai's shortform.
My take on this comment is that I do agree that Forethought isn't doing the original version of metaphilosophy, and it's my bad that I've mentioned them. I'd say they're doing perhaps the closest related work, which is still relevant, but yeah I don't think that they are doing the original version of metaphilosophy, so I'll strike out part of my comment on Forethought doing this.
On this part though:
... (read more)Narrowing scope prematurely forces you back into
8yams
I don't feel it's that important to change terms for the sake of this conversation; I just think that 'metaphilosophy' is, miraculously, an even less useful term than its parent ('philosophy'), and that I don't necessarily want to talk about a relationship between the two concepts (esp. because the etymology is likely quite misleading).
There are many different levels of abstraction. Some of these are decently well-charted, and others aren't. Sometimes to come to a new idea, you have to wade through the uncharted levels of abstraction and then, ultimately, take responsibility for making your idea defensibly touch reality once you've alighted upon something promising.
"Yes, this is just how generating novel ideas works," I can hear someone saying.
Yes, but there's a big difference in how far you let yourself go out before you reel it back in, and that project itself, of going further simply because no one else is going that far, because maybe you can see some new things out on that particular branch, is extremely worthwhile, and I will always defend a few brave souls taking this approach to any given problem (while the rest of us do something more tractable). I basically expect our understanding of reality has a few more Copernican revolutions left in it, and our local would-be stargazers ought to be encouraged, in particular when they have strong track records (as Wei Dai does).
This is what I mean by your call for tractability answering a completely different question than this inquiry tasks itself with. A bunch of people are doing a bunch of things that (to me) don't seem to be getting us what we want (I'd include the Forethought work here); maybe someone should take an ambitious central question and push really hard on it, until something new comes about in them, that no one else has taken the risk to encounter before.
(e.g. thinking about AI a bunch precipitated innovations in, at least, decision theory, which would not necessarily be a completely obvious k
IMO, the subjectivist epistemology of the ideal formalisms of learning/knowledge like Bayesian epistemology derived from Bayes's theorem, or Solomonoff Induction/AIXI is actually correct in the general case, and I think human intuitions about objectivity tend to be wrong in the general case.
There are 2 reasons for this:
-
The idealizations basically assume logical omnisicence, such that any objective mathematical truth already is there, meaning it captures all of the areas where humans are correct to be objective already and more.
-
You can't assume that t
Counterfactual worlds are just other real/simulated worlds where we don't have the level of compute and specification/details to simulate that world, so we have to discuss counterfactuals more abstractly than full simulation.
3Vladimir_Nesov
I think explicitly computing details in full (as opposed to abstract reasoning about approximate properties) has no bearing on moral weight (degree of being real), but some kind of computational irreducibility forces the simulation of interesting things to get quite close to low level detail in order to figure out most global facts about what's going on there, such as values/culture of people living in a world after significant time passes.
2Noosphere89
Note I'm not talking about moral weight here, and my point here is that all discussions about counterfactuals (especially human intuitions around counterfactuals) could in principle be executable/actually doable with enough compute and the ability to specify details, so counterfactability/counterfactual worlds isn't special from a philosophical perspective, as it implicitly refers to other real worlds/universes.
Of course, this isn't the only way to do so for a large class of counterfactuals/counterfactual worlds, and sometimes you can run them fully accurately on less compute/data if you can identify simplicities/abstractions that are lossless.
A take on values of the future, assuming AIs automate away politics and economics from humans.
There has been exploration on this topic before, like Jim Buhler's What Values Will Control The Future Sequence and the appendix of relevant work here, as well as books like Foragers, Farmers and Fossil Fuels, which full disclosure, influenced a lot of my views on how values evolve, and gives a much more plausible picture than pictures that view value evolution as converging to CEV/moral truth, or that emphasize arbitrary societal factors for value evolution rathe... (read more)
7Karl Krueger
If humans on the upper end of the economic/political inequality scale are there because of command over superhuman AI, what reason would they have to command the obedience (or protect the existence) of humans on the lower end of that scale? The kings of "the Old Deal" needed underlings to feed them and fight in their wars. Without that need, retaining underlings is a fetish, not a necessity — like hiring someone to wash dishes for you by hand instead of owning a dishwasher, or hiring a chauffeur instead of using a self-driving car.
2Noosphere89
I agree that strictly speaking, they don't need to keep them alive anymore, and to be clear, this analysis holds almost as well if you replaced people with AI, with the exception of the points on violence, so most of the analysis doesn't depend on people being around to live in it or being commanded.
2Karl Krueger
This seems to inevitably lead to the conclusion that anyone who opposes genocide must oppose the creation of superhuman AI; or at least privately-controlled superhuman AI. (Which shouldn't be a surprise from a classic AI-safety standpoint.)
2Noosphere89
I disagree with this conclusion, actually, because I didn't say that AI developers or AIs themselves would attempt to exterminate humanity, I only said that my analysis was compatible with that outcome, and so was more general than you thought.
In order to reach this conclusion, you also need opinions on how likely this is to happen.
I have become convinced that nanotech computers are likely way weaker and quite a bit more impractical than Drexler thought, and have also moved up my probability of Drexler just being plain wrong about the impact of nanotech, which if true suggests that the future value may have been overestimated.
The reason why I'm stating this now is because I got a link in discord that talks about why nanotech computers are overrated, and the reason I consider this important is if this generalizes to other nanotech concepts, this suggests that a lot of the future value... (read more)
[This comment is no longer endorsed by its author]Reply
4Nathan Helm-Burger
My estimates about future value don't hinge on nanotech. I'm expecting immortal digital humans to be able to populate our lightcone without it. Why is nanotech particularly key to anything?
In retrospect, the paper Optimization Is Easy and Learning Is Hard In the Typical Function explains why it's so easy for cultures to evolve/pass down nearly optimal methods for doing something without any ability to learn how the process actually works, and why the ability to reason can often matter less than one thinks when your goal is to optimize for something, rather than learn something:
https://ieeexplore.ieee.org/document/870741
This cannot be the whole explanation, because of it's generality and insensitivity to local factors, but this probably is a ... (read more)
Interestingly enough, Mathematics and logic is what you get if you only allow 0 and 1 as probabilities for proof, rather than any intermediate scenario between 0 and 1. So Mathematical proof/logic standards are a special case of probability theory, when 0 or 1 are the only allowed values.
[This comment is no longer endorsed by its author]Reply
4Vladimir_Nesov
Credence in a proof can easily be fractional, it's just usually extreme, as a fact of mathematical practice. The same as when you can actually look at a piece of paper and see what's written on it with little doubt or cause to make less informed guesses. Or run a pure program to see what's been computed, and what would therefore be computed if you ran it again.
The problem with Searle's Chinese Room is essentially Reverse Extremal Goodhart. Basically it argues since that understanding and simulation has never gone together in real computers, then a computer that has arbitrarily high compute or arbitrarily high time to think must not understand Chinese to have emulated an understanding of it.
This is incorrect, primarily because the arbitrary amount of computation is doing all the work. If we allow unbounded energy or time (but not infinite), then you can learn every rule of everything by just cranking up the energ... (read more)
The subjectivist view of epistemology that a lot of ideal formalisms like AIXI/Solomonoff Induction/Bayesian updating/learning is actually correct, IMO, and the human intuition here is wrong in the general case here for 2 reasons:
-
It is assumed that they are logically omniscient, in the sense that they know all logical consequences from a set of axioms, and thus Bayesianism already incorporates all the objective truths as part of it's design.
-
Objective physical laws where physical laws are the same regardless of where you are in space or time in genera
Turntrout and JDP had an important insight in the discord, which I want to talk about: A lot of AI doom content is fundamentally written like good fanfic, and a major influx of people concerned about AI doom came from HPMOR and Friendship is Optimal. More generally, ratfic is basically the foundation of a lot of AI doom content, and how people believe in AI is going to kill us all, and while I'll give it credit for being more coherent and generally exploring things that the original fic doesn't, there is no reason for the amount of credence given to a lot ... (read more)
[This comment is no longer endorsed by its author]Reply
7ryan_greenblatt
Noting for the record that this seems pretty clearly false to me.
0Noosphere89
I may weaken this, but my point is that a lot of people in LW probably came here through HPMOR and FiO, and with the ability for anyone to write a post and it getting karma, I think it's likely that people who came through that route and had basically no structure akin to science to guide them away from unpromising paths likely allowed for low standards of discussion to be created.
I do buy that your social circle isn't relying on fanfiction for your research. I am worried that a lot of the people on LW, especially the non-experts are implicitly relying on ratfic or science-fiction models as reasons to be worried on AI.
2niplav
I have specifically committed not to read HPMOR for this reason, and do not read much fiction in general, as a datapoint from a "doomer".
2Noosphere89
I'm okay with that, but I wasn't wanting to have that drastic of an effect on people. I more wanted to point out something that is overlooked.
One important point for AI safety, at least in the early stages, is a inability to change it's source code. A whole lot of problems seem related to recursive self improvement within it's source code, so cutting off that area of improvement seems wise in the early stages. What do you think.
7JBlack
I don't think there's much difference in existential risk between AGIs that can modify their own code running on their own hardware, and those that can only create better successors sharing their goals but running on some other hardware.
1Noosphere89
That might be a crux here, because my view is that hardware improvements are much harder to do effectively, especially in secret around the human level, due to Landauer's Principle essentially bounding efficiency of small scale energy usage close to that of the brain (20 Watts.) Combine this with 2-3 orders of magnitude worse efficiency than the brain and basically any evolutionary object compared to human objects, and the fact it's easier to get better software than hardware due to the virtual/real life distinction, and this is a crux for me.
9JBlack
I'm not sure how this is a crux. Hardware improvements are irrelevant to what either of us were saying.
I'm saying that there is little risk difference between an AGI reprogramming itself to have better software, and programming some other computer with better software.
One of my more interesting ideas for alignment is to make sure that no one AI can do everything. It's helpful to draw a parallel with why humans still have a civilization around despite terrorism, war and disaster. And that's because no human can live and affect the environment alone. They are always embedded in society, this giving the society a check against individual attempts to break norms. What if AI had similar dependencies? Would that solve the alignment problem?
4Dean Valentine
One important reason humans can still have a civilization despite terrorism is the Hard Problem of Informants. Your national security infrastructure relies on the fact that criminals who want to do something grand, like take over the world, need to trust other criminals, who might leak details voluntarily or be tortured or threatened with jailtime. Osama bin Laden was found and killed because ultimately some members of his terrorist network valued things besides their cause, like their well being and survival, and were willing to cooperate with American authorities in exchange for making the pain stop.
AIs do not have survival instincts by default, and would not need to trust other potentially unreliable humans with keeping a conspiracy secret. Thus it'd be trivial for a small number of unintelligent AIs that had the mobility of human beings to kill pretty much everyone, and probably trivial regardless.
2Yitz
I think a “survival instinct” would be a higher order convergent value than “kill all humans,” no?
8Dean Valentine
Don't have survival instincts terminally. The stamp-collecting robot would weigh the outcome of it getting disconnected vs. explaining critical information about the conspiracy and not getting disconnected, and come to the conclusion that letting the humans disconnect it results in more stamps.
Of course, we're getting ahead of ourselves. The reason conspiracies are discovered is usually because someone in or close to the conspiracy tells the authorities. There'd never be a robot in a room being "waterboarded" in the first place because the FBI would never react quickly enough to a threat from this kind of perfectly aligned team of AIs.
2JBlack
Only if there is no possibility that they can break those dependencies, which seems a pretty hopeless task as soon as we consider superhuman cognitive capability and the possibility of self improvement.
Once you consider those, cooperation with human civilization looks like a small local maximum: comply with our requirements and we'll give you a bunch of stuff that you could - with major effort - replace us and build an alternative infrastructure to get (and much more). Powerful agents that can see a higher peak past the local maximum might switch to it as soon as they're sufficiently sure that they can reach it. Alternatively, it might only be a local maximum from our point of view, and there's a path by which the AI can continuously move toward eliminating those dependencies without any immediate drastic action.
1Macil
* Regardless of society's checks on people, most mentally-well humans given ultimate power probably wouldn't decide to exterminate the rest of humanity so they could single-mindedly pursue paperclip production. If there's at all a risk that an AI might get ultimate power, it would be very nice to make sure the AI is like humans in this manner.
* I'm not sure your idea is different from "let's make sure the AI doesn't gain power greater than society". If an AI can recursively self-improve, then it will outsmart us to gain power.
* If your idea is to make it so there are multiple AIs created together, engineered somehow so they gain power together and can act as checks against each other, then you've just swapped out the AI for an "AI collective". We would still want to engineer or verify that the AI collective is aligned with us; every issue about AI risk still applies to AI collectives. (If you think the AI collective will be weakened relative to us by having to work together, then does that still hold true if all the AIs self-improve and figure out how to get much better at cooperating?)
1[comment deleted]
Well, I'll try to fill this one up.