So if alignment is as hard as it looks, desperately scrabbling to prevent recursive superintelligence should be an extremely attractive instrumental subgoal. Do we just lean into that?

Is anyone else on this website making YouTube videos? Less Wrong is great, but if you want to broadcast to a larger audience, video seems like the place to be. I know Rational Animations makes videos. Is there anyone else? Are you, personally, making any?

I'd like there to be a reaction of "Not Exhaustive", meant for a step where comment (or top-level post, for that matter) missed an important case - how a particular situation could play out, perhaps, or an essential system's component is not listed. An example use: on statement "to prevent any data leaks, one must protect how their systems transfer data and how they process it" with the missed component being protection of storage as well.

I recall wishing for it like three times since the New Year, with the current trigger being this comment:

Elon already has all of the money in the world. I think he and his employs are ideologically driven, and as far as I can tell they're making sensible decisions given their stated goals of reducing unnecessary spend/sprawl. I seriously doubt they're going to use this access to either raid the treasury or turn it into a personal fiefdom. <...>

which misses a specific case (which I'll name under the original comment if there is any interest).

Hi! I'm Embee but you can call me Max.

I'm a mathematics for quantum physics graduate student considering redirecting my focus toward AI alignment research. My background includes:
- Graduate-level mathematics
- Focus on quantum physics
- Programming experience with Python
- Interest in type theory and formal systems

I'm particularly drawn to MIRI-style approaches and interested in:
- Formal verification methods
- Decision theory implementation
- Logical induction
- Mathematical bounds on AI systems

My current program feels too theoretical and disconnected from urgent needs. I'm looking to:
- Connect with alignment researchers
- Find concrete projects to contribute to
- Apply mathematical rigor to safety problems
- Work on practical implementations

Regarding timelines: I have significant concerns about rapid capability advances, particularly given recent developments (o3). I'm prioritizing work that could contribute meaningfully in a compressed timeframe.

Looking for guidance on:
- Most neglected mathematical approaches to alignment
- Collaboration opportunities
- Where to start contributing effectively
- Balance between theory and implementation

If you've been collecting data on yourself, it's now moderately easy to let an LLM write code that'll tease out causal relationships between those variables, e.g. with a time series causal inference library like tigramite.

I've done that with some of my data, and surprisingly it both fails to back out results from RCTs I've run, and also doesn't find many (to me) interesting causal relationships.

The most surprising one is that creatine influences creativity. Meditating more doesn't influence anything (e.g. has no influence on mood or productivity), as far as could be determined. More mindful meditations cause happiness.

Because of the resampling/interpolation necessary, everything has a large autocorrelation.

Mainly posting this here so that people know it's now easy to do this, and shill tigramite. I hope I can figure out a way of dealing with rare interventions (e.g. MDMA, CBD, ketamine), and incorporate more collected data. Code here, written by Claude 3.5.1 Sonnet.

[CLAIMED!] ODD JOB OFFER: I think I want to cross-post Intro to Brain-Like-AGI Safety as a giant 200-page PDF on arxiv (it’s about 80,000 words), mostly to make it easier to cite (as is happening sporadically, e.g. here, here). I am willing to pay fair market price for whatever reformatting work is necessary to make that happen, which I don’t really know (make me an offer). I guess I’m imagining that the easiest plan would be to copy everything into Word (or LibreOffice Writer), clean up whatever formatting weirdness comes from that, and convert to PDF. LaTeX conversion is also acceptable but I imagine that would be much more work for no benefit.

I think inline clickable links are fine on arxiv (e.g. page 2 here), but I do have a few references to actual papers, and I assume those should probably be turned into a proper reference section at the end of each post / “chapter”. Within-series links (e.g. a link from Post 6 to a certain section of Post 4) should probably be converted to internal links within the PDF, rather than going out to the lesswrong / alignmentforum version. There are lots of textbooks / lecture notes on arxiv which can serve as models; I don’t really know the details myself. The original images are all in Powerpoint, if that’s relevant. The end product should ideally be easy for me to edit if I find things I want to update. (Arxiv makes updates very easy, one of my old arxiv papers is up to version 5.)

…Or maybe this whole thing is stupid. If I want it to be easier to cite, I could just add in a “citation information” note, like they did here? I dunno.

Does anyone here ever think to themselves, or out loud, "Here I am in the 21st Century. Sure, all the old scifi stories told me I'd have a shiny flying car but I'm really more interested in where my 21st Century government is?"

For me that is premised on the view that pretty much all existing governments are based on theory and structures that date at least back to the 18th Century in the West. The East might say they "modernized" a bit with the move from dynasties (China, Korea, Japan) to democratic forms but when I look at the way those governments and polities actually work seems more like a wrapper around the prior dynastic structures.

But I also find it challenging to think just what might be the differentiating change that would distinguish a "21st Century" government from existing ones. The best I've come up with is that I don't see it as some type of privatization divestiture of existing government activities (even though I do think some should be) but more of a shift from government being the acting agent it is now and more like a markets in terms of mediating and coordinating individual and group actions via mechanisms other than voting for representation or direct voting on actions.

I'm trying to think clearly about my theory of change and I want to bump my thoughts against the community:

• AGI/TAI is going to be created at one of the major labs.
• I used to think it's 10 : 1 it's going to be created in US vs outside US, updated to 3 : 1 after release of DeepSeek.
• It's going to be one of the major labs.
• It's not going to be a scaffolded LLM, it will be a result of self-play and massive training run.
• My odds are equal between all major labs. 

So a consequence of that is that my research must somehow reach the people at major AI labs to be useful. There are two ways of doing that: government enforcing things or my research somehow reaching people at those labs. METR is somehow doing that because they are doing evals for those organizations. Other orgs like METR are also probably able to do that (tell me which) 

So I think that one of the best things to help with AI alignment is to: do safety research that people at Antrophic or OpenAI or other major labs find helpful, work with METR or try to contribute to thier open source repos or focus on work that is "requested" by governance. 

Hi, I'm Alistair – am mostly new to LessWrong and rationality, but have been interested in and to differing extents involved in effective altruism for about five years now.

Some beliefs I hold:

• Veganism is a moral obligation, i.e. not being vegan is morally unjustifiable
  • Defining veganism not as a 100% plant-based diet, but rather as the ethical stance that commits one to avoid causing the exploitation (non-consensual use) and suffering of sentient non-humans, as far as practicable
    • This definition is inspired by but in my view better than the Vegan Society's definition
    • "As far as practicable" is ambiguous – this is deliberate
    • And there may be some trade-offs between exploitation and suffering when you take into account e.g. crop deaths and wild animal suffering
• Transformative AI is probably coming, and it's probably coming soon (maybe 70% chance we start seeing a likely irreversible paradigm shift in the world by 2027)
• If the way we treat members of other species relative to whom we are superintelligent is anything to go by, AGI/ASI/TAI could go really badly for us and all sentients
• As far as I can tell, the only solution we currently have is not building AGI until we know it won't e.g. invasively experiment on us (i.e. PauseAI)

I'm currently lead organiser of the AI, Animals, & Digital Minds conference in London in June 2025, and would love to speak to people who are interested in the intersection of those three things, especially if they're in London.

I'll be co-working in the LEAH Coworking Space and the Ambitious Impact office in London in 2025, and will be in the Bay Area in California in Feb-Mar for EAG Bay Area (if accepted) and the AI for Animals conference there.

Please reach out by DM!

Interested in:

• Sentience- & suffering-focused ethics; sentientism; painism; s-risks
• Animal ethics & abolitionism
• AI safety & governance
• Activism, direct action & social change
• Trying to make transformative AI go less badly for sentient beings, regardless of species and substrate

Bio:

• From London
• BA in linguistics at the University of Cambridge
• Almost five years in the British Army as an officer
• MSc in global governance and ethics at University College London
• One year working full time in environmental campaigning and animal rights activism at Plant-Based Universities / Animal Rising
• Now pivoting to the (future) impact of AI on biologically and artifically sentient beings

Update: Confusion of the Inverse

Is there an aphorism regarding the mistake P(E|H) = P(H|E) ?

Suggestions:

1. Thou shalt not reverse thy probabilities!
2. Thou shalt not mix up thy probabilities!
3. Thou shalt not invert thy probabilities! -- Based on Confusion of the Inverse

Arbital conditional probability

Example 2

Suppose you're Sherlock Holmes investigating a case in which a red hair was left at the scene of the crime.

The Scotland Yard detective says, "Aha! Then it's Miss Scarlet. She has red hair, so if she was the murderer she almost certainly would have left a red hair there. P(redhair∣Scarlet)=99%, let's say, which is a near-certain conviction, so we're done."

"But no," replies Sherlock Holmes. "You see, but you do not correctly track the meaning of the conditional probabilities, detective. The knowledge we require for a conviction is not P(redhair∣Scarlet), the chance that Miss Scarlet would leave a red hair, but rather P(Scarlet∣redhair), the chance that this red hair was left by Scarlet. There are other people in this city who have red hair."

"So you're saying..." the detective said slowly, "that P(redhair∣Scarlet) is actually much lower than 1?"

"No, detective. I am saying that just because P(redhair∣Scarlet) is high does not imply that P(Scarlet∣redhair) is high. It is the latter probability in which we are interested - the degree to which, knowing that a red hair was left at the scene, we infer that Miss Scarlet was the murderer. This is not the same quantity as the degree to which, assuming Miss Scarlet was the murderer, we would guess that she might leave a red hair."

"But surely," said the detective, "these two probabilities cannot be entirely unrelated?"

"Ah, well, for that, you must read up on Bayes' rule."

Hey, can anyone help me find this LW (likely but could be diaspora) article, especially if you might have read it too?

My vague memory: It was talking about (among other things?) some potential ways of extending point estimate probability predictions and calibration curves. I.e. in a situation where making a prediction in one way affects what the outcome will be, i.e. if there is a mind-reader/accurate-simulator involved that bases its actions on your prediction. And in this case, a two dimensional probability estimate might be more appropriate: If 40% is predicted for event A, event B will have a probability of 60%. If 70% for event A, then 80% for event B, and so on, a mapping potentially continuously defined for the whole range. (event A and event B might be the same.) IIRC the article contained 2D charts where curves and rectangles were drawn for illustration.

IIRC it didn't have too many upvotes, more like around low-dozen, or at most low-hundred.

Searches I've tried so far: Google, Exa, Gemini 1.5 with Deep Research, Perplexity, OpenAI GPT-4o with Search.

p.s. if you are also unable to put enough time into finding it, do you have any ideas how it could be found?  

Now seems like a good time to have a New Year's Predictions post/thread? 

Hello

I've been lurking on this website for a few months now. I'm interested in logic, computer science, ai, probability, information... I think I'll fit here. I speak French in every language I know.

I hope I'll be able to publish/discuss/coauthor on lesswrong, or somewhere else.

Biting my nails writing this as the welcome and FAQ made this site sound quite intimidating... I hope this is the right place for this...

Hi, I'm MPG. I'm a little younger than the main demographic for this forum skews, but I figured now was as good a time as any to get into rationality and the like. As a concept, rationality has interested me for a while, even before I knew it had a name. I first discovered LessWrong whilst reading a Gwern article and decided to create an account out of curiosity.

As the new user guide suggested, I will lurk for a while (whilst slowly, slowly trudging through The Sequences) before actually raising any points or engaging in any discussion, unless something specifically strikes my fancy.

I am new here and very excited to have discovered this community, as I rarely encounter anyone else who strongly embodies the qualities and values that I share with you. I am eager to engage, and expect that it will be mutually beneficial. 

At the same time, I have great admiration and respect for the people here and the community you have created, so I would like to ensure the impact of the ways I am engaging match my intent. Ideally I would love to fully absorb all the perspectives offered here already, before attempting to offer bits of my own. However, that's just not practical, so I will do my best to engage in ways that are mindful of the limits of my knowledge and contribute positively to discussions and to the community. If (and when) I fall short of those goals, I would really appreciate and welcome your feedback.

To that end, I have several questions already, which may or may not require a little back and forth. Is this the best place to ask those, or should I use intercom instead, or perhaps even discord? (I really appreciate the very helpful welcome guide and for this thread, and will try to avoid asking questions that it already answers. There's an awful lot to process though.)

Thanks so much for your assistance, and your dedication to maintaining this community.

Did the Ask Question type post go away? I don't see it any more. So I will ask here since it certainly is not worthy of a post (I have no good input or thoughts or even approaches to make some sense of it). Somewhat prompting the question was the report today about MS just revealing it's first quantum chip, and the recent news about Google's advancement in its quantum program (a month or two back).

Two branches of technology have been seen as game, or at least potential game changers: AI/AGI and quantum computing. The former often a topic here and certainly worth calling "mainstream" technology at this point. Quantum computing has been percolating just under the surface for quite a while as well. There have been a couple of recent announcements related to quantum chips/computing suggesting progress is being made.

I'm wondering if anyone has thought about the intersection between these two areas of development. Is a blending of quantum computing and AI a really scary combination for those with relatively high p(doom) views of existing AGI trajectories? Does quantum computing, in the hands of humans perhaps level the playing field for human v. AGI? Does quantum computing offer any potential gains in alignment or corrigibility?

I realize that current state quantum computing is not really in the game now, and I'm not sure if those working in that area have any overlap with those working in the AI fields. But from the outside, like my perspective is, the two would seem to offer large complementarities - for both good and bad I suppose, like most technologies.