faul_sname's Shortform

faul_sname

This is a special post for quick takes by faul_sname. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

AI agents might not need to develop long-term drives of their own to perform well at long-term tasks, if they can use humans for that purpose. As a corollary, AI agents that do not have long-term drives of their own can still act in long-term coherent ways, if they inhabit a world with abundant, easy-to-hire human labor. If it's cheaper to hire a human to get a scaffolded AI system to stay on track towards achieving some goal than it is to automate that monitoring, then it probably makes sense to just use the human.

Concretely, consider an AI system tasked with "produce a plate with two strawberries on it which are identical down to the cellular level, then stop". This problem has a number of subproblems, and a number of ways you could approach each subproblem. More than literally zero percent of the problem is "decide which high-level approach to take" or "notice that an approach seems to have hit a wall and go back to the drawing board". But not much more than zero percent. Most of the work will look more like "evaluate the division plane of this newly-divided cell, and compare it against the predictive model, and iterate a few thousand times, and tweak the predictive model, and iterate that loop a few dozen times".

In such a scenario, going from "you need a human a couple times per trillion tokens" to "you never need a human" is just not a significant efficiency gain. The AI should just hire a human to watch, and, if it's worried about the loyalty of the human, build in checks to make sure that the human isn't sabotaging it in any obvious way. A human in that role would be important, but not at all powerful.

5Noosphere897mo

I think my main disagreement with this is that most of the value in AI ultimately comes down to the long-tail, and for a number of reasons, having the human as a monitor near-entirely removes the value prop of AI. The reasons that I expect to generalize for most jobs include: 1. Fast reaction times are necessary, like in this example of a mortgage job John Wentworth had down below, either due to the physical needs of a job (robotics is especially vulnerable to this, where latencies need to be optimized hard, and is part of the reason biological brains need their extra parameters to achieve the same capabilities as ANNs) [...] Which leads us to another issue, and that is context. One of the things you will see constantly in this job description is that in order to do even one piece of the job well, you need to know all of the context of the job to be productive, and a prior belief of mine is that this is often the normal state of jobs, not a weird exception, especially as the economy grows (this is part of the reason why long-term memory can't be fully substituted by large contexts). So what about the strategies to avoid them, like factorization, bureaucracies or making jobs more predictable? The short answer is that they do help, but are not one weird tricks to remove the bottleneck of humans being the main cost, at least without assuming something full automation of economics and politics already, and the cases where they do seem to be one weird tricks (like the bureaucractization of countries improving general economic productivity and health and welfare) are cases where the capabilities were already there, but required alignment to prevent selfish humans from stealing the wealth of their countries, making economies not grow. And due to the cost of hiring humans, AIs would prefer to hire other AIs to do the job of alignment. And edge cases/inherent complexity are quite prominent in IRL jobs, meaning that without the ability to fully simulate the job (whic

I find myself in agreement with basically everything in this comment, and yet I observe that most of the benefits of AI so far have looked like moving rarer and rarer classes of tasks from "the long tail" to "approximately solved". I suppose it's worth clarifying exactly what we mean by "the long tail" though.

I observe now that for most common low-level tasks (e.g. write a patch and apply it to a file) scaffolded LLMs can ever do, they can, given multiple attempts but no human feedback, do the task quite reliably. Likewise for most tasks which are simple c... (read more)

2faul_sname7mo

Yep. Probably not just cell division either, if you're attempting this it's probably also wise to use caged auxin for controlling growth, rather than leaving hormonal regulation up to the individual cells. And even with those tricks, I expect that this is a very very hard problem, and also that in the course of trying to solve it this way the researchers would discover reasons why the exact thing I spitballed wouldn't work (but would likely also discover other angles of attack that were more promising based on their observations). You probably need to be able to target cell growth even more finely than the cellular level, since in order to prevent small differences spiralling into large differences a few divisions down the road, you need to be able to make fine corrections to cell size/shape/orientation based on observations. [...] That's where I was going with the Ishida 1991 paper [...] You really don't want to be dealing with the parent plant at all. It looks to me like it's probably straightforward to handle this part (relative to the other parts of this project), but again that's the sort of thing that you have to actually go out and interact with the external world to determine for sure. FWIW I do expect that Yudkowsky expected that this problem would require the solver to solve bioprinting or something "sci-fi" like that rather than "just" absurd amounts of iterative work. But the problem as stated seems solvable through absurd amounts of iterative work, and it is worlds where solving problems involves lots of iterative work of the "reality has a surprising amount of detail" type that I expect humans remain relevant for a while.