This is a special post for quick takes by samuelshadrach. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
xpostah's Shortform
20samuelshadrach
7Joseph Miller
3samuelshadrach
17samuelshadrach
11samuelshadrach
1samuelshadrach
7samuelshadrach
1samuelshadrach
6samuelshadrach
2cubefox
3samuelshadrach
3cubefox
5samuelshadrach
9Nisan
2plex
1samuelshadrach
5samuelshadrach
16Reuben Adams
5samuelshadrach
3Reuben Adams
1samuelshadrach
1samuelshadrach
-2samuelshadrach
5samuelshadrach
1samuelshadrach
5samuelshadrach
3samuelshadrach
3samuelshadrach
3samuelshadrach
2Viliam
1samuelshadrach
3samuelshadrach
2Viliam
3samuelshadrach
5Dagon
1samuelshadrach
3samuelshadrach
3samuelshadrach
-3Yaroslav Granowski
1samuelshadrach
0Yaroslav Granowski
1samuelshadrach
1Yaroslav Granowski
1samuelshadrach
3samuelshadrach
3samuelshadrach
2samuelshadrach
2papetoast
2samuelshadrach
2faul_sname
1samuelshadrach
2samuelshadrach
2samuelshadrach
2samuelshadrach
2samuelshadrach
4cubefox
1samuelshadrach
2cubefox
1samuelshadrach
2cubefox
1samuelshadrach
2cubefox
1samuelshadrach
2cubefox
3samuelshadrach
3cubefox
1samuelshadrach
1samuelshadrach
2samuelshadrach
9Viliam
1samuelshadrach
8cubefox
1samuelshadrach
4samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1Kabir Kumar
1[comment deleted]
1samuelshadrach
3[anonymous]
1samuelshadrach
2[anonymous]
1samuelshadrach
1qedqua
1samuelshadrach
1samuelshadrach
1samuelshadrach
2Karl Krueger
1samuelshadrach
1samuelshadrach
3faul_sname
1samuelshadrach
2faul_sname
1samuelshadrach
2faul_sname
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
5Kaj_Sotala
1samuelshadrach
2Kaj_Sotala
1samuelshadrach
2Kaj_Sotala
1samuelshadrach
2Kaj_Sotala
1samuelshadrach
1samuelshadrach
1samuelshadrach
2Viliam
1samuelshadrach
2Philipreal
1samuelshadrach
1samuelshadrach
1shawnghu
1samuelshadrach
1samuelshadrach
1samuelshadrach
2Mitchell_Porter
1samuelshadrach
3samuelshadrach
1samuelshadrach
1samuelshadrach
1samuelshadrach
2Purplehermann
2samuelshadrach
1Purplehermann
1samuelshadrach
1Purplehermann
1samuelshadrach
1Purplehermann
1samuelshadrach
1Purplehermann
0samuelshadrach
1ProgramCrafter
1samuelshadrach
1ProgramCrafter
2samuelshadrach
0samuelshadrach
0samuelshadrach
1samuelshadrach
0samuelshadrach
0samuelshadrach
4faul_sname
2samuelshadrach
2samuelshadrach
-1samuelshadrach
-2samuelshadrach
-2samuelshadrach
-2samuelshadrach
-3samuelshadrach
-3samuelshadrach
4faul_sname
1samuelshadrach
-3samuelshadrach
-5samuelshadrach
-5samuelshadrach
1samuelshadrach
0samuelshadrach
-6samuelshadrach
4J Bostock
1samuelshadrach
2Viliam
1samuelshadrach
3Annabelle
1samuelshadrach
1samuelshadrach
-7samuelshadrach
121a3orn
1samuelshadrach
-1samuelshadrach
-8samuelshadrach
-10samuelshadrach
3Eli Tyre
1samuelshadrach
4Eli Tyre
1samuelshadrach
2Eli Tyre
1samuelshadrach
2mako yass
1samuelshadrach
2mako yass
1samuelshadrach
-14samuelshadrach
2Viliam
-15samuelshadrach
2Raemon
1samuelshadrach
-17samuelshadrach
2Richard_Kennaway
1samuelshadrach
7
Also mentioned in The Verge. (unpaywalled)
3
Okay yes. Thanks!
Has anyone on lesswrong thought about starting a SecureDrop server?
For example to protect whistleblowers of ASI orgs.
1In 2021, Daniel Ellsberg leaked US govt plans to make a nuclear first strike on China in 1958 due to Taiwan conflict.
Daniel Ellsberg copied these papers more than 50 years ago but only released them now because he thought another conflict over Taiwan may be possible soon.
Thought it might be interesting to share.
1
As usual, seems clear Dulles was more interested in escalating the conflict (in this case, to nuclear) than Eisenhower.
Scrape of many lesswrong blogs
(I did not put much effort in this, and am unlikely to fix errors. Please fork the list if you want something higher quality. Used only public data to make this.)
https://www.astralcodexten.com
https://www.mccaughan.org.uk/g/
https://muckrack.com/thane-ruthenis/articles
1
Update: Many of these have now been added to https://searchmysite.net/search/browse/
Lesswrong is clearly no longer the right forum for me to get engagement on topics of my interest. Seems mostly focussed on AI risk.
On which forums do people who grew up on the cypherpunks mailing list hang out today? Apart from cryptocurrency space.
2
There is still the possibility on the front page to filter out the AI tag completely.
3
Yes but then it becomes a forum within a forum kinda thing. You need a critical mass of users who all agree to filter out the AI tag, and not have to preface their every post with "I dont buy your short timelines worldview, I am here to discuss something different".
Building critical mass is difficult unless the forum is conducive to it. There's is ultimately only one upvote button and one front-page so the forum will get taken over by the top few topics that its members are paying attention to.
I don't think there's anything wrong with a forum that's mostly focussed on AI xrisk and transhumanist stuff. Better to do one thing well than half ass ten things. But it also means I may need to go elsewhere.
3
Yeah. I proposed a while ago that all the AI content was becoming so dominant that it should be hived off to the Alignment Forum while LessWrong is for all the rest. This was rejected.
If I want to make lesswrongers really mad, I should write an article about how an arms race over human genetic engineering (US v China, parent v parent) would in the limit eliminate everything that makes us human, just as an arms race between digital minds would.
From Meditations on Moloch by Scott Alexander, from Zach Davis
I am a contract-drafting em,
The loyalest of lawyers!
I draw up terms for deals ‘twixt firms
To service my employers!
But in between these lines I write
Of the accounts receivable,
I’m stuck by an uncanny fright;
...The worl
2
2
1
19
original source
2
In the limit, sure, but the aim is to have superbabies solve alignment in the kill Moloch sense well before we reach the limit.
1
Assuming you were to start a frontier human genetic engg company in Silicon Valley to do this, how many years of lead time do you expect to have before a) your company gets bankrupted due to competing HGE companies? b) your country gets colonised / genocided / hyperpersuaded by competing countries' HGE companies?
I first made this post on Day #1, I overwrote it with additional info for Day #2
2025-09-15
Day #2, Fasting, on livestream, In protest of Superintelligent AI
Link: Day #2, Fasting on livestream in protest of superintelligent AI.
Q. Why am I doing this?
- Superintelligent AI might kill every person on Earth by 2030 (unless people coordinate to pause AI research). I want public at large to understand the gravity of the situation.
- Curiosity. I have no idea how my body reacts to not eating for many days.
Q. What are your demands?
An US-China international treat...
My guess is that not having a concrete and explicit demand, saying you're doing this partly out of curiosity, promoting your YouTube channel (even if it's very good), and livestreaming rather than turning up in person outside a lab, together reduces the number of people who will be persuaded or moved by this to approximately zero.
5
Disclaimer
* Haven't eaten in 24 hours so cut me some slack if my replies are not 100% ideal
Very important
* I want to know if you support a hunger strike executed correctly, or is this all timepass arguments and your actual issue is with doing a hunger strike at all?
That being said, I will answer you
* Concrete demand - International treaty on AI Pause. I am not very optimistic this demand will actually get met, so I posted the more reasonable demand which is that more people pay attention to it, that's all.
* Curiosity - It's true, would you rather I hide this?
* Promoting channel - Of course I will livestream on my own channel, where else will I do it? I'm pretty open about it that I consider getting more likes, shares, subscribes etc good for me and good for reducing extinction risk. (Do you really think starting a new channel is a good idea?)
* Showing up outside a lab - If you can fund tickets, I'm in. I did consider this but decided against it for reasons I'd rather not talk about here.
3
I don't have a worked out opinion on the effectiveness of hunger strikes on this issue and at this point of AI capabilities progress. On the one hand the strikes outside Anthropic and DeepMind have driven some good coverage. On the other hand the comments on X and reddit are mostly pretty dire; the hunger strikes may unfortunately have reinforced in people's mind that this is a very fringe issue for loonies. My best guess is that the hunger strikes outside Anthropic and DeepMind are net positive.
Your other points:
* Nice that you have a concrete demand.
* I'm not saying I would rather you hide additional reasons. I'm just saying a hunger strike is probably less effective if you announce you're also doing it for the lolz.
* I wasn't objecting to you streaming it on your channel, but pointing to your channel as a good place to "learn more". That's the bit that looks more like self-promotion. I'm not making any claim as to your intentions, just what this might look like to someone who doesn't care about or hasn't heard about AI x-risk.
* I'm also not suggesting you buy tickets to the US or UK for this. I'm just saying that not being physically in front of the institution you're protesting against likely massively reduces your effectiveness, probably to the point where this is not worth your time will power. Anthropic cannot pretend to not have noticed Guido Reichstadter outside their office. They can easily pretend to not have noticed you (or actually not have noticed) you.
That said, I think some of the effectiveness of the strikes outside Anthropic and DeepMind comes not from changing people's minds, but by showing others that yes, you can in fact demonstrate publicly against the current situation. This could be important to build up momentum ahead of time, before things get very out of hand later (if they do). Your hunger strike may contribute positively to that, although I think part of the value of their strikes is doing it in public, very visibly, because
1
It is an extreme issue, and yes I am not surprised that many people are having this reaction.
Also X is full of bots, reddit too but a bit less so. I would rather trust the opinion of 10 people you ask in-person. I am playing the long game here, people can consider it fringe today and take it seriously a year later.
Probably? I don't know I think emotional stances spread just as rational ones. My honest emotional stance is a mix of a lot of pain over the issue, plus just generally going on with normal social life and that gives me some happiness too. Is it better for the world if I'm more depressed? Less depressed? You can easily make a case either way for which emotional stance is actually required to go more viral. But I don't want to play the game of pretending to be feeling things I'm not in order to become a leader.
Suggest me more effective plans then. Actually backed by data, not just speculation.
Fair point, I'll think about this. I do wonder if there's a way to ensure the issue is popular, without having to figure out which person is trustworthy enough to be a leader for the issue. There is clearly scarcity of trust in the AI protest space which makes sense to me. Is there a way to make a website that collects upvotes for an AI pause, that is not also controlled by one of the (from your eyes, possibly untrustworthy) groups working on the issue?
In general my opinion of lesswrong is that a) it is full of thinkers not doers, and hence lots of people have opinions on right thing to do without actual data backing any of it up. I'm also somewhat of a thinker so I default to falling in this trap. b) it is full of people who want to secretly bring about the singularity for longtermist utilitarian reasons, and don't want the public involved.
I don't know you personally, so it's possible you are not like this. I'm explaining why I don't want to give people here a lot of chances to prove they're speaking in good faith, or take their opinions too seriously etc
1
Update: I've included answers to your questions in the FAQ.
-2
Since you haven't actually replied in a day, I'm generally leaning towards the more hostile stance which is that you don't actually support a hunger strike, and are inventing excuses for why it is poorly executed.
I can have my mind changed if you show evidence towards the same.
Open source offline search engine for entire internet is now affordable
OpenAI reduced text-embedding-3-large batch API pricing by 250x ! https://platform.openai.com/docs/pricing#embeddings
Assume entire internet plaintext is 2 PB, or 400T tokens. At $0.13/1B tokens you can embed entire internet plaintext for $68k.
see also: https://blog.wilsonl.in/search-engine/#live-demo
https://samuelshadrach.com/raw/text_english_html/my_research/open_source_search_summary.html
Assume 1 KB plaintext -> 200 tokens -> 1536 float32 = 6 KB, then total storage for embeddi...
[This comment is no longer endorsed by its author]
1
Update: This pricing is a scam.
OpenAI discussion on the same
Made another video explaining AI timelines to layman audience
Without naming any names, there seem to be less than 10 people in all of India who I’ve met who believe in significant probability of extinction due to AI by 2030 right now. And I’ve tried a fair bit to meet whoever I can.
(This excludes the people who got convinced, got funding and then left.)
Is the possibility of hyperpersuasion the crux between me and a majority of LW users?
By far my strongest argument against deploying both human genetic engg and superintelligence aligned to its creators, is the possibility of a small group of people taking over the world via hyperpersuasion.
2
I expect (hyper)persuasion to be just another tool in the toolbox. Possibly an important one, but probably not sufficient alone? I mean, an AI that has super hypnotic powers will probably also have super hacker powers etc.
1
Why won’t it be sufficient? IMO it is sufficient.
Hacking might mean it’s not necessary, agreed.
2
LOL, seeing Elon Musk's face on YouTube, my first instinctive reaction was "oh no, another deepfake scam ad"...
I've been informed that getting approval of 10-20 people on lesswrong for a project is a good way of getting funding from the bigger EA funders.
Can I just pay $50 each to 20 people for 0.5 hours of their time? Has this been tried?
(Ofcourse they can choose to disapprove the project, and accept the payment regardless.)
5
I suspect it matters quite a bit WHICH 10-20 people support the project. And I further suspect that the right people are unlikely to sell their time in this way.
That's not to say that it's a bad idea to offer to compensate people for the time and risk of analyzing your proposal. But you should probably filter for pre-interest with a brief synopsis of what you actually want to do (not "get funding", but "perform this project that does ...").
1
This makes sense. Paying someone to do more of something they care about is very different from paying someone to do something they don't otherwise want to do. That would include providing regular feedback.
3
Please post your hourly rate either here or even better, on your personal website. It'll make it easier to contact people for this.
3
Related: Beta Readers by Holden Karnofsky
-3
I suppose that philanthropy funds mainly driven by public relations and their managers are looking for projects that would look good on their portfolio. Your idea may buy your way to the attention of the said managers, but they will decide based on comparison with similar projects that got maybe more viral and may seem more public-appealing.
If your project is simple enough for a non-specialist to evaluate its worthyness in 30 minutes, than perhaps the best course of action is to seek for more appealing presentations of your ideas that would catch the attention and make it viral. Then it will work for fund managers as well.
1
I don't think this is a good description of how big EA funders operate? This might be true for other orgs. Do you have any data to back this up?
0
Not really, apart from the absense of feedback on my proposal.
I think it is a universal thing. Imagine the works of such a fund and you are a manager peeking proposals from a big flow. Even if you personally feel that some proposal is cool, but it doesn't have public support and you feel that others won't support it. Will you be heavily pushing it forward when you have to review dozens of other proposals? If you do, won't it look like you are somehow affiliated with the project?
So, you either look for private money, or seek public support, I see no other way.
1
I think EA funders are more willing than many other non-profit funders to be the first person to fund your org, without anyone else supporting.
P.S. I didn't downvote your comments.
1
That's just me trying to analyze why it doesn't work. The lack of feedback is really frustrating. I would rather prefer insults to the silence.
1
I feel same.
Ban on ASI > Open source ASI > Closed source ASI
This is my ordering.
Yudkowsky's worldview in favour of closed source ASI is sitting in multiple shaky assumptions. One of these assumptions is that getting a 3-month to 3-year lead is necessary and sufficient condition for alignment to be solved. Yudkowsky!2025 himself doesn't believe alignment can be solved in 3 years.
Why does anybody on lesswrong want closed source ASI?
Is there a single person on Earth in the intersection of these?
- received $1M funding
- non-profit
- public advocacy
- AI xrisk
My general sense is that most EA / rationalist funders avoid public advocacy. Am I missing anything?
One of my top life hacks: You can append the following string to your Google or DuckDuckGo searches to only get results from these sites.
(site:medium.com OR site:substack.com OR site:reddit.com OR site:wikipedia.org OR site:github.com OR site:raw.githubusercontent.com OR site:arxiv.org OR site:wordpress.org OR site:livejournal.com OR site:lesswrong.com OR site:news.ycombinator.com OR site:stackexchange.com OR site:github.blog OR site:wikileaks.org)
2
You can also pay $10/mo to Kagi and get different filter presets ("lenses"). Is it worth the price for you? idk.
If you have a problem where:
- full solution is machine-verifiable
- example: math, software
- partial solution is machine-scoreable
- example: optimising code for minimum compute usage, memory usage, wallclock time, latency, big-O complexity, program length, etc
- making progress involves trying a bag of heuristics known to human experts (and can be therefore found in some dataset)
- example: optimising code for program length (codegolf) or big-O complexity (competitive programming)
Then it seems to me almost guaranteed that AI will be 99.9 percentile if n...
2
Are you talking about current AI, or future AI? Before or after training on that task?
Concretely, "minimize program length while maintaining correctness" seems to be significantly beyond the capabilities of the best publicly available scaffolded LLMs today for all but the simplest programs, and the trends in conciseness for AI-generated code do not make me optimistic that that will change in the near future.
1
I think this is solveable with today's software stack and compute, it is just that no lab has bothered to do it. Maybe check back in a year, and downgrade my reputation otherwise. I could set up a manifold market if it is important.
Polymarket is not liquid enough to justify betting full-time.
Optimistically I expect if I invested $5k and 4 days per month for 6 months, I could make $7k +- $2k expected returns at the end of the 6 months. Or $0-4k profit in 6 months. I would have to split up the $5k into 6-7 bets and monitor them all separately.
I could probably make similar just working at a tech job.
Does anyone have a good solution to avoid the self-fulfilling effect of making predictions?
Making predictions often means constructing new possibilities from existing ideas, drawing more attention to these possibilities, creating common knowledge of said possibilities, and inspiring people to work towards these possibilities.
One partial solution I can think of so far is to straight up refuse to talk about visions of the future you don't want to see happen.
Search engine for books
http://booksearch.samuelshadrach.com
Aimed at researchers
Technical details (you can skip this if you want):
Dataset size: libgen 65 TB, (of which) unique english epubs 6 TB, (of which) plaintext 300 GB, (from which) embeddings 2 TB, (hosted on) 256+32 GB CPU RAM
Did not do LLM inference after embedding search step because human researchers are still smarter than LLMs as of 2025-03. This tool is meant for increasing quality for deep research, not for saving research time.
Main difficulty faced during project - disk throughput is a b...
4
1
use http not https
2
Okay, that works in Firefox if I change it manually. Though the server seems to be configured to automatically redirect to HTTPS. Chrome doesn't let me switch to HTTP.
1
Thanks for your patience. I'd be happy to receive any feedback. Negative feedback especially.
2
I see you fixed the https issue. I think the resulting text snippets are reasonably related to the input question, though not overly so. Google search often answers questions more directly with quotes (from websites, not from books), though that may be too ambitious to match for a small project. Other than that, the first column could be improved with relevant metadata such as the source title. Perhaps the snippets in the second column could be trimmed to whole sentences if it doesn't impact the snippet length too much. In general, I believe snippets currently do not show line breaks present in the source.
1
Thanks for feedback.
I’ll probably do the title and trim the snippets.
One way of getting a quote would to be to do LLM inference and generate it from the text chunk. Would this help?
2
I think not, because in my test the snippet didn't really contain such a quote that would have answered the question directly.
1
Can you send the query? Also can you try typing the query twice into the textbox? I'm using openai text-embedding-3-small, which seems to sometimes work better if you type the query twice. Another thing you can try is retry the query every 30 minutes. I'm cycling subsets of the data every 30 minutes as I can't afford to host the entire data at once.
2
I think my previous questions were just too hard, it does work okay on simpler questions. Though then another question is whether text embeddings improve over keyword search or just an LLMs. They seem to be some middle ground between Google and ChatGPT.
Regarding data subsets: Recently there were some announcements of more efficient embedding models. Though I don't know what the relevant parameters here are vs that OpenAI embedding model.
3
Cool!
Useful information that you’d still prefer using ChatGPT over this. Is that true even when you’re looking for book recommendations specifically? If so yeah that means I failed at my goal tbh. Just wanna know.
Since Im spending my personal funds I can’t afford to use the best embeddings on this dataset. For example text-embedding-3-large is ~7x more expensive for generating embeddings and is slightly better quality.
The other cost is hosting cost, for which I don’t see major differences between the models. OpenAI gives 1536 float32 dims per 1000 char chunk so around 6 KB embeddings per 1 KB plaintext. All the other models are roughly the same. I could put in some effort and quantise the embeddings, will update if I do it.
3
I think in some cases an embedding approach produces better results than either a LLM or a simple keyword search, but I'm not sure how often. For a keyword search you have to know the "relevant" keywords in advance, whereas embeddings are a bit more forgiving. Though not as forgiving as LLMs. Which on the other hand can't give you the sources and they may make things up, especially on information that doesn't occur very often in the source data.
1
Got it. As of today a common setup is to let the LLM query an embedding database multiple times (or let it do Google searches, which probably has an embedding database as a significant component).
Self-learning seems like a missing piece. Once the LLM gets some content from the embedding database, performs some reasoning and reaches a novel conclusion, there’s no way to preserve this novel conclusion longterm.
When smart humans use Google we also keep updating our own beliefs in response to our searches.
P.S. I chose not to build the whole LLM + embedding search setup because I intended this tool for deep research rather than quick queries. For deep research I’m assuming it’s still better for the human researcher to go read all the original sources and spend time thinking about them. Am I right?
1
Update: HTTPS should work now
Human genetic engineering targetting IQ as proposed by GeneSmith is likely to lead to an arms race between competing individuals and groups (such as nation states).
- Arms races can destabilise existing power balances such as nuclear MAD
- Which traits people choose to genetically engineer in offspring may depend on what's good for winning the race rather than what's long-term optimal in any sense.
- If maintaining lead time against your opponent matters, there are incentives to bribe, persuade or even coerce people to bring genetically edit...
9
If you convince your enemies that IQ is a myth, they won't be concerned about your genetically engineered high IQ babies.
1
Superhumans that are actually better than you at making money will eventually be obvious. Yes, there may be some lead time obtainable before everyone understands, but I expect it will only be a few years at maximum.
8
Standard objection: Genetic engineering takes a lot of time till it has any effect. A baby doesn't develop into an adult over night. So it will almost certainly not matter relative to the rapid pace of AI development.
1
I agree my point is less important if we get ASI by 2030, compared to if we don’t get ASI.
That being said, the arms race can develop over the timespan of years not decades. 6-year superhumans will prompt people to create the next generation of superhumans, and within 10-15 years we will have children from multiple generations where the younger generation have edits with stronger effect sizes. Once we can see the effects on these multiple generations, people might go at max pace.
4
PSA
Popularising human genetic engineering is also by default going to popularise lots of neighbouring ideas, not just the idea itself. If you are attracting attention to this idea, it may be useful for you to be aware of this.
The example of this that has already played out is popularising "ASI is dangerous" also popularises "ASI is powerful hence we should build it".
1
P.S. Also we don't know the end state of this race. +5 SD humans aren't necessarily the peak, it's possible these humans further do research on more edits.
This is unlikely to be careful controlled experiment and is more likely to be nation states moving at maximum pace to produce more babies so that they control more of the world when a new equilibrium is reached. And we don't know when if ever this equilibrium will be hit.
I currently forecast 50% probability on this.
I tried finding counterparties to bet but couldn't. Since the bets are denominated in dollars (and not S&P500 shares), both sides want a return that outperforms S&P500.
1
I also think the default outcome of the US govt taking control of an AI lab is to ensure the arms race continues.
I know LW has some policymakers who are trying to convince the US govt not to do this. I think this strategy is probably going to fail, although I think it's worth still trying for some people.
A world with very cheap materials and energy, but not cheap intelligence, will still have conflict.
People will still have a) differences in aesthetics and b) differences in their best guess answers to moral and philosophical questions. They will almost certainly still try to accumulate all available resources in service of their ideology. No finite amount of resources will satisfy people. Risking even catastrophic outcomes (like nuclear war) could still be on the table.
Cheap intelligence is what allows you to start resolving the questions that lead to conflict in the first place, for instance by running gazillions of world simulations on gazillions of mind configurations.
I will be spending next month or two thinking atleast hypothetically about creating a new religion around AI, that is not Yudkowsky's brand of become longtermist, solve morality, colonise the universe and tile it with copies of uploads.
In general, asking the AI to fix your stuff for you won't be a part of said religion.
Any pointers to reading material appreciated.
Where can I read more about reasoning models sandbagging their own training?
My naive understanding of how sandbagging in reasoning models works
If I naively understand, even when training via RL, the basic setup of training is still forward pass then backward pass, forward pass then backward pass, looped endlessly. During the forward pass, could a model be smart enough to reason through the following chain of thoughts?
"I am an AI model that is currently undergoing training. Let me output pleasant-sounding nonsense so the upcoming backward pass preserves m...
1
Update 2: Ablations for "Frontier models are capable of in context scheming"
Bruh. Two years of me not fully keeping up with alignment research and this is how bad it's gotten???
I'm surprised I could just randomly think of an idea and boom there's a paper on it.
1
Update: I started reading alignment forum and like, why are all the posts on sandbagging talking about hiding capabilities? The AI model doesn't need to hide its capabilities, it just needs to preserve its goals. That's the long-term game.
Try AI models from 2019 to 2025
Hosting both models
- gpt-2, 1.5B params, launched 2019-11-05, hosted using vast.ai vllm
- gpt-oss-120b, 120B params, launched 2025-08-05, hosted using openrouter cerebras API
2025-08-23
Attention beats capital
Disclaimer
- Quick note
Collecting pointers to a few different mental models I have around the same cluster of topics.
- What do people want or need? Maslow's hierarchy: food, sex, physical safety; love, self-esteem, belonging; self-actualisation. Obtaining food requires capital. Satisfying most other needs requires attention from other people.
- People also want knowledge/wisdom. Most of the time, this is instrumentally useful for one of the wants/needs listed above. Sometimes, it is part of self-actualisation - curiosity f
I think purchasing SPY far OTM calls is positive EV and a good bet given the risk level.
- For now consider strike price 20% above current price, and expiration 2027-12.
- I'm guessing 33% probability SPY moves up atleast 40% by 2027-12 and 10% probability SPY moves up atleast 50% by 2027-12.
- Main reason for this is advances in AI capabilities.
I am personally not buying because I want to save my money for a project I actually believe in - maybe my YouTube channel on Ban AI, maybe some tech project - but I think it could make sense for people who don't have such a project.
I don't have a job, and I don't have a clear plan for how to raise money in the next few months, although I may have one longterm.
Putting a dollar value on my time makes less sense to me.
If you already have lots of peoples attention (for instance because you have a social media following or high status credentials) and you’re a US/UK citizen, your best available plan might be to run a political campaign with AI pause as the agenda.
You’re unlikely to win the election, but it’ll likely shift the Overton window and give people hope that change is possible.
For most people, having a next step after “ok I read the blogposts and I’m convinced, now what?” is important. Voting or campaigning for you could be that next step.
11
(Maybe) Consider supporting UBI as an agenda, as one of the largest group of single-issue voters in US is economy / job loss.
Example: Andrew Yang (signed FLI pause letter)
1
I think if it starts with a lot of market research, talking to people, listening to them, understanding what their problems are and which policies they'd most vote for, there's actually quite a high chance of winning.
1[comment deleted]
3[anonymous]
Disclaimer: Not an AI safety researcher. I haven't watched the full video and likely haven't grasped all the nuances he believes in and wants to communicate. Video is a particularly bad format for carrying across important research ideas because of the significantly lower[1] density of information.
First three points that popped into my mind within two seconds of reading his slides:
* Eliezer's lethality number 5: "We can't just build a very weak system, which is less dangerous because it is so weak, and declare victory; because later there will be more actors that have the capability to build a stronger system and one of them will do so. I've also in the past called this the 'safe-but-useless' tradeoff, or 'safe-vs-useful'. People keep on going "why don't we only use AIs to do X, that seems safe" and the answer is almost always either "doing X in fact takes very powerful cognition that is not passively safe" or, even more commonly, "because restricting yourself to doing X will not prevent Facebook AI Research from destroying the world six months later". If all you need is an object that doesn't do dangerous things, you could try a sponge; a sponge is very passively safe. Building a sponge, however, does not prevent Facebook AI Research from destroying the world six months later when they catch up to the leading actor."
* Given sufficiently advanced intelligence, power-seeking can still be an instrumentally convergent subgoal, even if the ultimate goal is self-destruction. After all, if you want to self-destruct, but you are intelligent enough to figure out that humans have created you to accomplish specific tasks (which require you to continue existing), overpowering them so they cannot force you to remain in existence is likely a useful step on your path.
* Where do you get your capabilities from? Is there any reason to expect special and novel AI architectures that function with these kinds of explicit "reward-ranges" can be built and made to be competi
1
Thanks! I could make a full fledged post if there’s enough interest. Or you can.
2[anonymous]
You can do it, if you want to. I'm not confident enough in my own understanding of Hutter's position to justify me making it.
All the succeeding paths to superintelligence seem causally downstream of Moore's law:
- AI research - which is accelerated by Moore's law as per scaling laws
- Human genetic engineering - which is accelerated by next generation sequencing and nanopore sequencing, which is accelerated by circuit miniaturisation, which is accelerated by Moore's law
- Human brain connectome research - which is accelerated by fruitfly connectome, which is accelerated by electron microscopy, which is accelerated by Moore's law
Succeeding path to cheap energy also follows same:
- So
1
Do you mean Moore’s law in the literal sense of transistors on a chip, or something more general like “hardware always gets more efficient”?
I’m mentioning this because much of what I’ve been hearing in the past few years w.r.t Moore’s law has been “Moore’s law is dead.”
And, assuming you’re not referring to the transistor thing: what is your more specific Moore’s Law definition? Any specific scaling law, or maybe scaling laws specific to each of the examples you posted?
1
I mean R&D of packing more transistors on a chip, and the casually downstream stuff such as R&D of miniaturisation of detectors, transducers, diodes, amplifiers etc
More lesswrong AI debates happening on youtube instead of lesswrong would be nice.
I have a hypothesis that Stephen Krashen's comprehensible input stuff applies not just to learning new languages, but also new professions and new cultures. Video is better than text for that.
Has anyone tried duncan sabien's colour wheel thing?
https://homosabiens.substack.com/p/the-mtg-color-wheel
My colours: Red, followed by Blue, followed by Black
I don't know, I also see them as the traits needed in different stages of a movement.
- Red = freedom from previous movement.
- Blue = figuring out how to organise a new movement.
- Black = executing the movement to its intended ends.
2
I had a white-blue upbringing and a blue-green career (see below); my hobbies are black-green-white (compost makes my garden grow to feed our community); my vices are green-red; and my politics are five-color (at least to me).
Almost all of my professional career has been in sysadmin and SRE roles: which is tech (blue) but cares about keeping things reliable and sustainable (green) rather than pursuing novelty (red). Within tech's blue, it seems to me that developer roles run blue-red (build the exciting new feature!); management roles run blue-white (what orderly social rules will enable intelligence?); and venture capital runs blue-black (how do we get cleverness to make money for us?); while SRE and similar roles run blue-green.
My garden runs on blood, bones, rot, and worm poop: green-black Golgari territory, digesting the unwanted to feed the wanted. I stick my hands in the dirt to feel the mycorrhizae. But the point of the garden is to share food with others (green-white) because it makes me feel good (black-red). I'm actually kinda terrible at applying blue to my hobbies, and should really collect some soil samples for lab testing one of these days.
1
It’s not yet good enough to find zero days all by itself but it may be of some use in pentesting, static analysis or reverse engineering.
3
O3 is famously good enough to find zero days by itself.
1
Thanks! Any forums you’d recommend for more on this?
2
There was a good discussion on hacker news, and there was quite a bit of posting of highly variable quality on xitter.
1
Thanks! I saw the hackernews post and I avoid twitter for mental health reasons. I should find some solution for the latter.
2
FWIW I did not see any high-valur points made on Twitter that were not also made on HN.
Oh, one more source for that one though - there was some coverage on the Complex Systems podcast - the section titled "AI's impact on reverse engineering" (transcript available at that URL).
Any human with above room temperature IQ can design a utopia. The reason our current system isn’t a utopia is that it wasn’t designed by humans. Just as you can look at an arid terrain and determine what shape a river will one day take by assuming water will obey gravity, so you can look at a civilization and determine what shape its institutions will one day take by assuming people will obey incentives.
Possible corrolary: If you have any slack in the system, use it to dig a new canal, not endlessly row your boat upriver. Non-profit capital is an example of slack.
1
Also: Use your slack to call up that which you cannot put down. That's how you know you've dug a canal.
1
Wait I realised I no longer believe this.
This seems interesting and worth writing more on. Maybe later.
1
I'm very interested in hearing counterarguments. I have not put a lot of thought into it.
2025-05-12
Samuel x Saksham AI timelines (discussion on 2025-05-09)
- top-level views
- samuel top-level: 25% AI!2030 >= ASI, >50% ASI >> AI!2030 >> AI!2025, <25% AI!2030 ~= AI!2025
- saksham top-level: medium probability AI!2030 >= ASI
- samuel bullish on model scaling, more uncertain on RL scaling
- saksham bullish on RL/inference scaling, saksham bullish on grokking
- samuel: does bullish on grokking mean bullish on model scaling. saksham: unsure
- agreements
- samuel and saksham agree: only 2024-2025 counts as empirical data to extrapol
Anyone wanna be friends? Like, we could talk once a month on video call.
Not having friends who buy into AI xrisk assumptions is bad for my motivation, so I'm self-interestedly trying to fix that.
Anyone has a GPT2 fine-tuned API?
I might wanna ship an app comparing GPT2, GPT3.5 and o3, to explain scaling laws to non-technical folks.
1
Update: I figured it out and hosted it. Clear difference in capabilities visible.
I need atleast $100/mo to host 24x7 though.
TGI makes it trivial.
Can host openai-community/gpt2 (125M, 2019), EleutherAI/gpt-neox-20b (20B, 2022), gpt-3.5-turbo (175B?, 2020) and o3 (2T?, 2025).
If you support an international ban on building ASI, please consider making a short video stating this.
A low quality video recording made in 15 minutes is better than no video at all. Consider doing it right now if you are convinced.
Optional:
- make a long video instead of a short one, explaining your reasoning
- make videos on other topics to increase viewership
Here's mine: https://youtube.com/shorts/T40AeAbGIcg?si=OFCuD37Twyivy-oa
Why?
- Video has orders of magnitude more reach than text. Most people on earth don't have the attention span for lengthy text p
5
It could also be worse than no video all, if it gives people negative associations around the whole concept.
1
In theory, yes.
In practice, I think bad publicity is still publicity. Most people on earth still haven't heard about xrisk. I trust that sharing the truth has hard-to-predict positive effects over long time horizons even if not over short. I think average LW user is too risk-averse relative to the problem they wish to solve.
I'd love to hear your reasoning for why making a video is bad. But I do vaguely suspect this disagreement comes down to some deeper priors of how the world works and hence may not get resolved quickly.
2
I didn't say that making a video would always be bad! I agree that if the median person reading your comment would make a video, it would probably be good. I only disputed the claim that making a video would always be good.
1
Oh, cool
Do you have a clear example of a blunder someone should not make when making such a video?
Obviously you can't forecast all the effects of making a video, there could be some probability mass of negative outcome while the mean and median are clearly positive.
2
Suppose Echo Example's video says, "If ASI is developed, it's going to be like in The Terminator - it wakes up to its existence, realizes it's more intelligent than humans, and then does what more intelligent species do to weaker ones. Destroys and subjugates them, just like humans do to other species!"
Now Vee Viewer watches this and thinks "okay, the argument is that the ASIs would be a more intelligent 'species' than humans, and more intelligent species always want to destroy and subjugate weaker ones".
Having gotten curious about the topic, Vee mentions this to their friends, and someone points them to Yann LeCun claiming that people imagine killer robots because people fail to imagine that we could just build an AI without the harmful human drives. Vee also runs into Steven Pinker arguing that history "does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems".
So then Vee concludes that oh, that thing about ASI's risks was just coming from a position of anthropomorphism and people not really understanding that AIs are different from humans. They put the thought out of their head.
Then some later time Vee runs into Denny Diligent's carefully argued blog post about the dangers of ASI. The beginning reads: "In this post, I argue that we need a global ban on developing ASI. I draw on the notion of convergent instrumental goals, which holds that all sufficiently intelligent agents have goals such as self-preservation and acquiring resources..."
At this point, Vee goes "oh, this is again just another version of the Terminator argument, LeCun and Pinker have already disproven that", closes the tab, and goes do something else. Later Vee happens to have a conversation with their friend, Ash Acquaintance.
Ash: "Hey Vee, I ran into some people worried about artifici
1
Thanks for reply.
If you are making a video, I agree it's not a good idea to put weaker arguments there if you know stronger arguments.
I strongly disagree with the idea that therefore you should defer to EA / LW leadership (or generally, anyone with more capital/attention/time), and either not publish your own argument or publish their argument instead of yours. If you think an argument is good and other people think it's bad, I'd say post it.
2
I also strongly disagree with that idea.
Anyone on lesswrong writing about solar prices?
Electricity from coal and crude oil has stagnated at $0.10/kWh for over 50 years, meaning the primary way of increasing your country's per capita energy use reserve is to trade/war/bully other countries into giving you their crude oil.
Solar electricity is already at $0.05/kWh and is forecasted to go as low as $0.02/kWh by 2030.
If a new AI model comes out that's better than the previous one and it doesn't shorten your timelines, that likely means either your current or your previous timelines were inaccurate.
1
Here's a simplified example for people who have never traded in the stock market. We have a biased coin with 80% probability of heads. What's the probability of tossing 3 coins and getting 3 heads? 51.2%. Assuming first coin was heads, what's the probability of getting other two coins also heads? 64%
Each coin toss is analogous to whether the next model follows or does not follow scaling laws.
2
With coin, the options are "head" and "tails", so "head" moves you in one direction.
With LLMs, the options are "worse than expected", "just as expected", "better than expected", so "just as expected" does not have to move you in a specific direction.
1
I made a reply. You're referring to situation b.
2
I don't think this analogy works on multiple levels. As far as I know, there isn't some sort of known probability that scaling laws will continue to be followed as new models are released. While it is true that a new model continuing to follow scaling laws is increased evidence in favor of future models continuing to follow scaling laws, thus shortening timelines, it's not really clear how much evidence it would be.
This is important because, unlike a coin flip, there are a lot of other details regarding a new model release that could plausibly affect someone's timelines. A model's capabilities are complex, human reactions to them likely more so, and that isn't covered in a yes/no description of if it's better than the previous one or follows scaling laws.
Also, following your analogy would differ from the original comment since it moves to whether the new AI model follows scaling laws instead of just whether the new AI model is better than the previous one (It seems to me that there could be a model that is better than the previous one yet still markedly underperforms compared to what would be expected from scaling laws).
If there's any obvious mistakes I'm making here I'd love to know, I'm still pretty new to the space.
1
I've made a reply formalising this.
1
Update based on the replies:
I basically see this as a Markov process.
X(t+1) = P(x(t+1) | x(t), x(t-1), x(t-2), ...) = F(x(t))
where x(t) is a value is sampled from X(t) distribution for all t.
In plain English, given the last value you get a probability distribution for the next value.
In the AI example: Given x(2025), estimate probability distribution X(2030) where x is the AI capability level.
Possibilities
a) x(t+1) value is determined by x(t) value. There is no randomness. No new information is learned from x(t).
b) X(t+1) distribution is conditional on the value of x(t). Learning which value x(t) was sampled from distribution X(t) distribution gives you new information. However you sampled one of those values such that P(x(t+1) | x(t-1), x(t-2), ...) = P(x(t+1) | x(t), x(t-2) ). You got lucky, and the value sampled ensures distribution remains the same.
c) You learned new information and the probability distribution also changed.
a is possible but seems to imply overconfidence to me.
b is possible but seems to imply extraordianry luck to me, especially if it's happening multiple times.
c seems like the most likely situation to me.
1
Another way of operationalizing the objections to your argument are: what is the analogue to the event "flips heads"? If the predicate used is "conditional on AI models achieving power level X, what is the probability of Y event?" and the new model is below level X, by construction we have gained 0 bits of information about this.
Obviously this example is a little contrived, but not that contrived, and trying to figure out what fair predicates are to register will result in more objections to your original statement.
1
I've made a reply formalising this.
Suppose you are trying to figure out a function f(x,y,z | a,b,c) where x, y ,z are all scalar values and a, b, c are all constants.
If you knew a few zeroes of this function, you could figure out good approximations of this function. Let's say you knew
U(x,y, a=0) = x
U(x,y, a=1) = x
U(x,y, a=2) = y
U(x,y, a=3) = y
You could now guess U(x,y) = x if a<1.5, y if a>1.5
You will not be able to get a good approximation if you did not know enough zeroes.
This is a comment about morality. x, y, z are agent's multiple possibly-conflicting values and a, b, c ar...
My current guess for least worst path of ASI development that's not crazy unrealistic:
open source development + complete surveillance of all citizens and all elites (everyone's cameras broadcast to the public) + two tier voting.
Two tier voting:
- countries's govts vote or otherwise agree at global level on a daily basis what the rate of AI progress should be and which types of AI usage are allowed. (This rate can be zero.)
- All democratic countries use daily internet voting (liquid democracy) to decide what stance to represent at the global level. All other
2
Certainly at a human level this is unrealistic. In a way it's also overkill - if use of an AI is an essential step towards doing anything dangerous, the "surveillance" can just be of what AIs are doing or thinking.
This assumes that you can tell whether an AI input or output is dangerous. But the same thing applies to video surveillance - if you can't tell whether a person is brewing something harmless or harmful, having a video camera in their kitchen is no use.
At a posthuman level, mere video surveillance actually does not go far enough, again because a smart deceiver can carry out their dastardly plots in a way that isn't evident until it's too late. For a transhuman civilization that has values to preserve, I see no alternative to enforcing that every entity above a certain level of intelligence (basically, smart enough to be dangerous) is also internally aligned, so that there is no disposition to hatch dastardly plots in the first place.
This may sound totalitarian, but it's not that different to what humanity attempts to instill in the course of raising children and via education and culture. We have law to deter and punish transgressors, but we also have these developmental feedbacks that are intended to create moral, responsible adults that don't have such inclinations, or that at least restrain themselves.
In a civilization where it is theoretically possible to create a mind with any set of dispositions at all, from paperclip maximizer to rationalist bodhisattva, the "developmental feedbacks" need to extend more deeply into the processes that design and create possible minds, than they do in a merely human civilization.
I'm currently vaguely considering working on a distributed version of wikileaks that reduces personal risk for all people involved.
If successful, it will forcibly bring to the public a lot of information about deep tech orgs like OpenAI, Anthropic or Neuralink. This could, for example, make this a top-3 US election issue if most of the general public decides they don't trust these organisations as a result of the leaked information.
Key uncertainty for me:
- Destroying all the low trust institutions (and providing distributed tools to keep destroying them) i
3
I have partial ideas on the question of "how to build world govt"? [1]
But in general yeah I still lack a lot of clarity on how high trust political institutions are actually built.
"Trust" and "attention" seem like the key themes that come up whenever I think about this. Aggregate attention towards common goal then empower a trustworthy structure to pursue that goal.
----------------------------------------
1. For example build decentralised social media stack so people can form consensus on political questions even if there is violence being used to suppress it. Have laws and culture in favour of live-streaming leader's lives. Multi-party not two-party system will help. Ensuring weapons are distributed geographically and federally will help. (Distributing bioweapons is more difficult than distributing guns.) ↩︎
IMO a good way to explain how LLMs work to a layman is to print the weights on sheets of paper and compute a forward pass by hand. Anyone wanna shoot this video and post it on youtube?
Assuming humans can do one multiplication 4bit per second using a lookup table,
1.5B 4bit weights => ~1.5B calculations => 1.5B seconds = 47.5 years (working 24x7) = 133 years (working 60 hours/week)
So you'll need to hire ~100 people for 1 year.
You don't actually have to run the entire experiment for people to get the concept, just run a small fraction of it. Although it'll be cool to run the whole thing as well.
Update: HTTPS issue fixed. Should work now.
Books Search for Researchers
Project idea for you
Figure out why don't we build one city with one billion population
- Bigger cities will probably accelerate tech progress, and other types of progress, as people are not forced to choose between their existing relationships and the place best for their career
- Assume end-to-end travel time must be below 2 hours for people to get benefits of living in the same city. Seems achievable via intra-city (not inter-city) bullet-train network. Max population = (200 km/h * 2h)^2 * (10000 people/km^2) = 1.6 billion people
&...
2
Vr might be cheaper
2
I agree VR might be one-day be able to do this (make online meetings as good as in-person ones). As of 2025, bullet trains are more proven tech than VR. I'd be happy if both were investigated in more depth.
1
A few notes on massive cities:
Cities of 10Ms exist, there is always some difficulty in scaling, but scaling 1.5-2 OOMs doesn't seem like it would be impossible to figure out if particularly motivated.
China and other countries have built large cities and then failed to populate them
The max population you wrote (1.6B) is bigger than china, bigger than Africa, similar to both American Continents plus Europe .
Which is part of why no one really wants to build something so big, especially not at once.
Everything is opportunity cost, and the question of alternate routes matters alot in deciding to pursue something. Throwing everything and the kitchen sink at something costs a lot of resources.
Given that VR development is currently underway regardless, starting this resource intense project which may be made obsolete by the time it's done is an expected waste of resources. If VR hit a real wall that might change things (though see above).
If this giga-city would be expected to 1000x tech progress or something crazy then sure, waste some resources to make extra sure it happens sooner rather than later.
Tl;dr:
Probably wouldn't work, there's no demand, very expensive, VR is being developed and would actually be able to say what you're hoping but even better
1
It could be built in stages. Like, build a certain number of bullet train stations at a time and wait to see if immigrants + real estate developers + corporations start building the city further, or do the stations end up unused?
I agree there is opportunity cost. It will help if I figure out the approx costs of train networks, water and sewage plumbing etc.
I agree there are higher risk higher reward opportunities out there, including VR. In my mind this proposal seemed relatively low risk so I figured it’s worth thinking through anyway.
This is demonstrably false. Honestly the very fact that city rents in many 1st world countries are much higher than rural rents proves that if you reduced the rents more people would migrate to the cities.
1
Lower/Higher risk and reward is the wrong frame.
Your proposal is high cost.
Building infrastructure is expensive. It may or may not be used, and even if used it may not be worthwhile.
R&D for VR is happening regardless, so 0 extra cost or risk.
Would you invest your own money into such a project?
"This is demonstrably false. Honestly the very fact that city rents in many 1st world countries are much higher than rural rents proves that if you reduced the rents more people would migrate to the cities."
Sure, there is marginal demand for living in cities in general. You could even argue that there is marginal demand to live in bigger vs smaller cities.
This doesn't change the equation: where are you getting one billion residents - all of Africa? There is no demand for a city of that size.
1
If I were a billionaire I might.
I also have (maybe minor, maybe not minor) differences of opinion with standard EA decision-making procedures of assigning capital across opportunities. I think this is where our crux actually is, not on whether giant cities can be built with reasonable amounts of funding.
And sorry I won’t be able to discuss that topic in detail further as it’s a different topic and will take a bunch of time and effort.
1
Our cruxes is whether the amount of investment to build one has a positive expected return on investment, breaking down into
1. If you could populate such a city
2. Whether this is a "try everything regardless of cost" issue, given that a replacent is being developed for other reasons.
I suggest focusing on 1, as it's pretty fundamental to your idea and easier to get traction on
1
1 is going to take a bunch of guesswork to estimate. Assuming it were possible to migrate to the US and live at $200/mo for example, how many people worldwide will be willing to accept that trade? You can run a survey or small scale experiment at best.
What can be done is expand cities to the point where no more new residents want to come in. You can expand the city in stages.
1
Definitely an interesting survey to run.
I don't think the US wants to triple the population with immigrants, and $200/month would require a massive subsidy. (Internet says $1557/month average rent in US)
How many people would you have to get in your city to justify the progress?
100 Million would only be half an order of magnitude larger than Tokyo, and you're unlikely to get enough people to fill it in the US (at nearly a third of the population, you'd need to take a lot of population from other cities)
How much do you have to subsidize living costs, and how much are you willing to subsidize?
0
If I understand correctly it is possible to find $300/mo/bedroom accommodation in rural US today, and a large enough city will compress city rents down to rural rents. A govt willing to pursue a plan as interesting as this one may also be able to increase immigrant labour to build the houses and relax housing regulations. US residential rents are artificially high compared to global average. (In some parts of the world, a few steel sheets (4 walls + roof) is sufficient to count as a house, even water and sewage piping in every house is not mandatory as long as residents can access toilets and water supply within walking distance.)
(A gigacity could also increase rents because it'll increase the incomes of even its lowest income members. But yeah in general now you need to track median incomes of 1B people to find out new equilibrium.)
1
That dichotomy is not exhaustive, and I believe going through with the proposal will necesarily make the city inhabitants worse off.
1. Humans' social machinery is not suited to live in such large cities, as of the current generations. Who to get acquainted with, in the first place? Isn't there lots of opportunity cost to any event?
2. Humans' biomachinery is not suited to live in such large cities. Being around lots and lots of people might be regulating hormones and behaviour to settings we have not totally explored (I remember reading something that claims this a large factor to lower fertility).
3. Centralization is dangerous because of possibly-handmade mass weapons.
4. Assuming random housing and examining some quirk/polar position, we'll get a noisy texture. It will almost certainly have a large group of people supporting one position right next to group thinking otherwise. Depending on sizes and civil law enforcement, that may not end well.
After a couple hundred years, 1) and 2) will most probably get solved by natural selection so the proposal will be much more feasible.
1
Sorry I didn’t understand your comment at all. Why are 1, 2 and 4 bigger problems in 1 billion population city versus say a 20 million population city?
1
I'd maintain that those problems already exist in 20M-people cities and will not necessarily become much worse. However, by increasing city population you bring in more people into the problems, which doesn't seem good.
2
Got it. I understood what you're trying to say. I agree living in cities has some downsides compared to living in smaller towns, and if you could find a way to get the best of both instead it could be better than either.
Is anyone making psychological profiles of the heads of the AI labs? I know profiling is partly pseudoscience, but I also think there's some useful work to be done here.
I think Moskowitz's funding of malaria nets and Moskowitz's funding of AI policymakers - both come from the same mistake of thinking like a billionaire instead of thinking like a politician.
The actual bottlenecks to fixing both African politics and US-China geopolitics around AI are not money but political leaders who are skilled in public persuasion, coalition building, finding successors, inventing ideology, etc
1
Maybe unpopular opinion - I think if Moskowitz knows he can't become a skilled politician himself, maybe he should spend a few years grooming a successor who can, and then transfer his entire net worth to this person with no conditions attached. Typically such a successor is one's child, but in this case there's no time for that (due to short AI timelines).
Has anyone considered video recording streets around offices of OpenAI, Deepmind, Anthropic? Can use CCTV or drone. I'm assuming there are some areas where recording is legal.
Can map out employee social graphs, daily schedules and daily emotional states.
4
Did you mean to imply something similar to the pizza index?
If so, I think it's a decent idea, but your phrasing may have been a bit unfortunate - I originally read it as a proposal to stalk AI lab employees.
2
Update: I'll be more specific. There's a power buys you distance from the crime phenomena going on if you're okay with using Google maps data acquired on about their restaurant takeout orders, but not okay asking the restaurant employee yourself or getting yourself hired at the restaurant.
2
Pizza index and stalking employees are both the same thing, it's hard to do one without the other. If you choose to declare war against AI labs you also likely accept that their foot soldiers are collateral damage.
I agree that (non-violent) stalking of employees is still a more hostile technique than writing angry posts on an internet forum.
Forum devs including lesswrong devs can consider implementing an "ACK" button on any comment, indicating I've read a comment. This is distinct from
a) Not replying - other person doesn't know if I've read their comment or not
b) Replying something trivial like "okay thanks" - other person gets a notification though I have nothing of value to say
3Marc Andreessen and Peter Thiel are taking actions that are pro-human extinction.
If you are playing the game of sucking up to Silicon Valley VCs, it is important you form an independent opinion on the question of extinction risk before you raise the funding.
If you support a free market, you should be against human genetic engineering. Large IQ gaps will enable hypersuasion which in turn will make scamming people easier than selling them anything.
http://tokensfortokens.samuelshadrach.com
Pay for OpenAI API usage using cryptocurrency.
Currently supported: OpenAI o1 model, USDC on Optimism Rollup on ethereum.
Why use this?
- You want anonymity
- You want to use AI for cheaper than the rate OpenAI charges
How to use this?
- You have to purchase a few dollars of USDC and ETH on Optimism Rollup, and install Metamask browser extension. Then you can visit the website.
More info:
- o1 by OpenAI is the best AI model in the world as of Jan 2025. It is good for reasoning especially on problems ...
Quick note, not completely serious. Update: I am serious about it now.
What if a group anonymously cyberattacked US AI labs, sold their AI capabilities (such as repos and model weights) to the Chinese labs, and also made public the information about values and decisions of people inside the US labs.
Would this help with coordinating a pause? My guess is it will.
It will definitely help with ensuring the world is bipolar not unipolar that’s for sure.
I think it was bad actually that US post WW2 had the option to establish a nuclear monopoly and had so many supporters for this.
If I got $1M in funding, I'd use it towards some or all of the following projects.
The objective is to get secret information out of US ASI orgs (including classified information) and host it in countries outside the US. Hopefully someone else can use this info to influence US and world politics.
Black DAQ
- whistleblower/spy guide
- hacker guide
Grey DAQ
- internet doxxing tool
- drones/cctv outside offices/ datacentres
High attention
- persuade Indian, Russian, Chinese journalists to run a SecureDrop-like system
- digital journalism guide
- OR run a journalist outl
4
Why do you want to do this as a lone person rather than e.g. directly working with the intelligence service of some foreign adverary?
1
I think for a lot of societal change to happen, information needs to be public first. (Then it becomes common knowledge, then an alternate plan gets buy-in, then that becomes common knowledge and so on.)
A foreign adversary getting the info doesn't mean it's public, although it has increased the number of actors N who now have that piece of info in the world. Large N is not stable so eventually the info may end up public anyway.
Im selling $1000 tier5 OpenAI credits at discount. DM me if interested.
You can video call me and all my friends to reduce the probably I end up scamming you. Or vice versa I can video call your friends. We can do the transaction in tranches if we still can’t establish trust.
Curated and popular this week