Yes mostly agree. Unless the providers themselves log all responses and expose some API to check for LLM generation, we're probably out of luck here, and incentives are strong to defect.

One thing I was thinking about (similar to i.e - speedrunners) is just making a self-recording or screenrecording of actually writing out the content / post? This probably can be verified by an AI or neutral third party. Something like a "proof of work" for writing your own content.

 

Yes, I agree with that. I'm not claiming that knowing about it stops you from wanting ice cream.

I'm claiming that if the concept was hardwired into our brains, evolution would have had an easy time optimizing us directly to want "inclusive genetic fitness" rather than wanting ice cream. 

i.e - we wouldn't want ice cream at all but reason from first principles what we should eat based on fitness.

Just finished reading "If Anyone Builds It, Everyone Dies". I had a question that seems like an obvious one, but one I didn't see addressed in the book, maybe someone can help:

The main argument in the book is the analogy to humans. Evolution "wanted" us to maximize genetic fitness, but it didn't get what it trained for. Instead, it created humans who love ice cream and condoms even though they reduce our genetic fitness.

With AGI, we're on track to do something similar - we won't get an AI aligned to human interests even though we do RLHF or any other such simple training or shaping to an AI, it'll end up wanting something weird and inhuman rather than maximizing human values.

But in my mind, this seems to miss a fairly important point: The fact that human brains don't come pre-wired with much knowledge. We have to learn it from scratch. We don't come out of the womb with concept of "inclusive genetic fitness". It took us culture and ~200,000 years to figure that out, and we still only learn it after about 15-20 years of existing.  So there's no way that evolution could have made us point our utility function to "inclusive genetic fitness" because that concept doesn't exist in our brains.

Modern AIs don't seem like that. They come with the sum of human knowledge baked in during pre-training. As they get smarter, the concept of "human values" or "friendly AI" is definitely something in it's existing mind. So it should be much easier for us to do alignement and test whether we can point it to that specific concept vs. what what evolution had.

Seems mostly true. There's also a group of people flailing around trying to fit it in their workflows because all the top tech companies are saying it's the next big thing. 

I notice lots of LARPing too with adding the word "AI" to everything hoping that will unlock some new avenues.

> The big struggle is to even start using AI coding assistant tools. Lots of teams just don’t use them at all, or use them in very limited ways. People leading these teams know they are going to lose if they don’t change but are struggling to get their orgs to let them.

It seems to me 25-50% of developers are using some form of AI-assisted coding. Did you notice that the beaureacracy of their companies was not allowing their developers to use coding assistants?

 

I think there are 3 ways to think about AI and lot of confusion seems to happen because the different paradigms are talking past each other. The 3 paradigms I see on the internet & when talking to people:

Paradigm A) AI is a new technology like the internet / smartphone / electricity - this seems to be mostly held by VC's / enterpreneurs / devs that think this will unlock a whole new set of apps like AI:new apps like smartphone:Uber or internet:Amazon


Paradigm B) AI is a step change in how humanity will work. Similarly to the agricultural revolution that led to the change in how large society could get and GDP growth, and the industrial revolution was a step-change in GDP growth from ~0% to 2-4% a year, and made things possible such as electricity and the internet and smartphones.

Paradigm C) AI is like the rise of humanity on this earth (the first general intelligences). The world changed completely with the rise of GI, and ASI/AGI will be a similar paradigm. We've been locked at humanity's level of intelligence for the past ~200k years, and getting ASI will be like unlocking multiple new revolutions all at the same time.


Most of the LW crowd is probably (C) or between (B) and (C)

When talking to the general population, I've found it to be very helpful to probe about where they are before talking about things like AI safety / how the world will change.
 

I made the same comment on the original post. I really think this is a blindspot for US-based AI analysis.

China has smart engineers, as much as DM, OpenAI etc. Even the talent in a lot of these labs is from China originally. With a) immigration going the way it is, b) the ability to coordinate massive resources as a state, subsidies, c) potentially invading Taiwan, d) how close DeepSeek / Qwen models seem to be and the rate of catchup, e) how uncertain we are about hardware overhand (again, see deepseek training costs) etc, I think we should put at least a 50% chance of China being ahead in the next year.

My initial reaction - A lot of AI related predictions are based on "follow the curve" predictions, and this is mostly doing that. With a lack of more deeper underlying theory on the nature of intelligence, I guess that's all we get.

If you look at the trend of how far behind China is to the US, that has gone from 5 years behind 2 years ago, to maybe 3 months behind now. If you follow that curve, it seems to me that China will be ahead of the US by 2026 (even with the chip controls, and export regulations etc - my take is you're not giving them enough agency). If you want to follow the curve, IMO  you can /s/USA/China after 2026 (i.e - China is ahead of the US), and I can imagine it being a better trend-following prediction. It's much less convincing to tell a story about Chinese AI labs being ahead given who we are, but I'd put at least a 50/50 chance on China being ahead vs. USA being ahead.

Other than that, thanks for putting something concrete out there. Even though it's less likely the more specific it is, I feel this will get a lot more talked about, and hopefully some people with power (i.e - governments) start paying some attention to disempowerment scenarios.

Coming from a somewhat similar space myself, I've also had the same thoughts. My current thinking is there is no straightforward answer on how to convert dollars to impact. 

I think the EA community did a really good job at that back in the day with a spreadsheet-based relatively easier way to measure impact per dollars or per life saved in the near-term future.

With AI safety / existential-risk - the space seems a lot more confused, and everyone has different models of the world, what will work, and what good ideas are. There are some people working directly on this space directly - like QURI, but IMO it's not anything close to a consensus for "where can I put my marginal dollar for AI safety". The really obvious / good ideas and people working on them don't seem funding-constrained.

There's in general (from my observation):

- Direct interpretability work on LLM
- Governance work (trying to convince regulators / goverments to put a stop to this)
- Explaining AI risk to the general public
- Direct alignment work on current-gen LLM (super-alignment type things in major labs)
- More theoretical work (like MIRI), but I don't know if anyone is doing this now.
- More weirder things like whole brain emulation, or gene-editing / making superbabies.

My guess is your best bet spending your money / time on the last one would be on the margin helpful, or just talk to people who are struggling for funding and otherwise seem like they have decent ideas that you can fund.

There's probably something other than those in the above list will actually work for reducing existential risk from AI, but no one knows what it it is.

I'm very dubious that we'll solve alignment in time, and it seems like my marginal dollar would do better in non-obvious causes for AI safety. So I'm very open to funding something like this in the hope we get a AI winter / regulatory pause etc.

I don't know if you or anyone else has thought about this, but what is your take on whether this or WBE is the more likely chance to getting done successfully? WBE seems a lot more funding intensive, but also possible to measure progress easier and potentially less regulatory burdens?