The main concern around superintelligence is the recursive AI research.

Can someone taboo what "recursive AI research" is?

What kinds of experiments are we talking about here?

LLM agents are good at writing code. Presumably they can write code that can test various ML papers and such. If a subset of these experiments produces useful results (on benchmarks, or other tests), the findings will coalesce into the next generation that will then do the same things.

1) There is an expectation that there will definitely be useful findings. This is reasonable, because why wouldn't there be one.

2) There is an expectation that models writing code to run experiments will cause the entire process to go faster. Is the bottleneck models writing code, or is the bottleneck some other aspects that models cannot affect? (e.g. access to compute)

The main concern is that AI researchers will raise their productivity using AI tools (which then feeds back into AI development). But do AI tools actually increase productivity? Anecdotally, they do, for some capabilities (the "jagged frontier").

source on my blog: https://sundaystopwatch.eu/some-thoughts-in-traffic/

Some Thoughts in Traffic

(these are literally thoughts I had sitting in traffic and that relate to traffic, but you're free to extract generalized life lessons from them if you want)

1. When someone cuts you off in traffic, you might get angry, but their behavior, their cutting-off-other-people-in-traffic personality guarantees that you will never be stuck behind them, unable to go faster.
2. The way to minimize killing animals on the road is to drive in the middle of the road, straddling two lanes (if the oncoming lane is empty). That way, you are maximizing the space (and therefore reaction time) between the road and whatever it is on the side of the road. This predisposes that both sides of the road are equally likely to contain crossing animals - if there's a literal wall on one side, then it's less killing per kilometer (kpk - nice measure eh?) to drive near that wall.
3. Google Maps, on the surface, appears to be an Earth element (Fire, Water, Earth, Air, Void), but it's actually a Water element because when it detects congestion ahead, it reroutes you to the same destination, just like water finds another path. (I can't decide if this thought is high, pseudo-deep or plain schizo.)
4. There are defensive drivers and offensive drivers. Defensive drivers worry about bad things happening, and imagine them happening before they do. So the difference isn't "one speeds, the other one doesn't". In fact, defensive drivers also speed. The difference is what you're thinking about. If you're feeling really optimistic on your every drive, you're likely an offensive driver. If you are imagining all the ways something could go wrong as you drive, you're defensive. Thinking about the direction of oncoming animals is an example of pessimistic thinking, and therefore defensive driving.
5. Defensive driving is the obvious right choice.
6. Defensive driving has a bad effect though. Bad actors (offensive drivers) can do their thing unchallenged, such as cutting people off in traffic, because defensive drivers will have predicted their actions and already have adapted their maneuvers. Therefore the only thing that curtails offensive driving is offensivemaxxing, where an offensive driver is so offensive that nothing could be done to help them (e.g. driving at 100 km/h into a narrow corner and going offroad) OR other offensive drivers, which will cause crashes. It's a sort of predator-prey dynamic, but offensive drivers are both the predator and prey, and defensive drivers are the background foliage.
7. I've changed my mind about driving cars with manual shifting. I no longer think that makes any sense, unless you're racing or a hobbyist with retro cars. For everyday driving, automatic shifting is good, more relaxed, less cognitive overhead, and if traffic is safe enough, you can take a sip of water without worrying that you won't be able to downshift in time for a corner or something.
8. Despite my strong occasional bouts of hate towards automobile traffic and optimization for automobile infrastructure, overall I still think that cars did a whole lot of good, despite all the CO2 emissions, lead poisoning, community disruption - both on the level of making streets unsafe for children to play on, and disrupting public transport as a joining force of the community. If we can choose between a world where people have their own robocarriage, and a world where you can't go anywhere except on foot or by horse, even if all else isn't equal (!), you'll still probably choose to live in the robocarriage world... because people want to go places!
9. Computer networking and traffic engineering are essentially the same discipline, with differences in the protocol specifications (e.g. TCP vs. right-hand-side rule) and packet formats (e.g. datagrams vs. cars). Yet all the guys that I know that studied traffic engineering in college know nothing about protocols, or even that kind of thinking. You would expect that you'll learn about e.g. Dijkstra's pathfinding algorithm in computer science and in traffic engineering, but (from my experience with people in both disciplines), you only learn about it in CS.
10. The European version of the pickup truck is the cargo van with the pickup bed. It's almost the same thing, actually it literally is the same thing. Yet there's a sense in which a pickup truck is a car (a personal vehicle), and a pickup van is a van (a work vehicle), even when a pickup truck is used for work.