Hey all, well, I have been wanting to post in LessWrong for a while but I don't have what is worth being put into a post.
I am a 19 years old AI and robotics engineering undergraduate student from Iraq. Well I am in my sophomore year and I have went through a lot of changes in my mapping of reality. I was raised into Islam, I left that initially when I was 11 but fully left it at 14, I think some books I read then triggered it. I went through an intense 3 months derealization experience after that where I completely lost all sense of reality and self, I had trouble sleeping back then lol. Well that passed and I began building my beliefs from scratch, I read so much philosophy from both east and west, I jumped through a lot of beliefs.
I got into university and I have fallen in love with engineering, but seeing how the world is using AI and the negative effects it is having was quite sad for me, though I didn't know of the field of AI safety. I became digital privacy obsessed for a while, just to make sure that big corporations don't collect my data to get used in algorithmic targeting and AI training.
In the meantime, I got into so much experiences, met so much people, had so many conversations, went to many hackathons, met so many CEOs, many so professors, collaborated with other highly passionate students, did public speaking, started some startups. I really used my first 2 years of uni as much as I can. However the need to achieve for my own sake was quite tiring, I must always strive to become "more", but why? A sense of Nihilism started creeping up on me, I began isolating and spending hours upon hours everyday to write my thoughts down. There was so much forming in my mind that I was struggling to put it all in words and write it, mostly existential questions and realizations.
Well in this time I discovered some interesting resources, like lesswrong, which also got me into AI safety, which I am pivoting to and I got accepted into the technical AI safety program by BlueDot. I noticed that my country still remains unaware of AI safety and its importance, I aim to spread it and start something here. That's where I grew from wanting to achieve/become more for my sake, to wanting to achieve/become more for others sake, because I realized I actually do believe in creating systems we fully understand that allow us to be more human, rather than take away the qualities that make us human and letting us outsource our thinking for example. I am not even talking about existential risk here, which may or may not happen (I prefer keeping an open stance but keeping in mind the worst option).
I discovered the effective altruism forum and its culture.
And I discovered the work of David Chapman at metarationality.com and meaningness.com, which was quite insightful too. I discovered the sequences and they are really honing my rationality as I am reading them.
I talked with @Kaj_Sotala who offered me great advice as someone new here. I connected with people in the AI safety field.
Regarding what I currently believe, it is a bit hard to put into words. I believe the map is not the territory (ironically, the phrase is also a map), I believe rationality is the best map we have to represent reality, while limited, as all methods and linguistic models are, it is the best we got, and we should aim to hone it and sharpen it.
I also believe reality is paradoxical while maps are perfectly ordered. On one hand, nothing is better or worse and everything is a flavor of reality with meanings assigned by us. On the other hand, I have much grand plans, motivations, and desires. On one hand, I am alive, growing older. On the other hand, I am dying, approaching my death. On one hand, no words or concept can convey reality perfectly. On the other hand, we must seek to understand reality. On one hand, existence is meaningless. On the other hand, existence is so full of meaning.
I believe that I don't know. Coming to peace with not knowing certain things. Not assuming god or any other story/entity to fill up the gaps. That's the correct stance when something is beyond our ability to know.
I like this place because it seems to be a group of people who care about their epistemic process and its effects rather than just going through the motions.
Here is my LinkedIn BTW: https://www.linkedin.com/in/aehamhamid/
I am putting all my effort right now on pivoting to AI safety and seeing how I can make an impact in some way.
1Hello, and thanks @habryka! I’ve been vaguely aware of LW for several years but only today published my first post after reading recent discussion about whether forecasting is ‘worth it.’
I have quite a specific perspective on this - I work in the tech industry as a product strategist, and the main way I do my work is using forecasting and foresight methods to help people make decisions.
I’m hoping this is a place where I can write more regularly; as well as broaden my knowledge of salient ideas in AI progress/safety & factory farming.
Otherwise, I’m interested in getting to know people in London so will look at attending upcoming meetups - am also a ‘serious’ meditator so always up for a sit.
Hey, thank you @Mo Putera! That's encouraging to hear. Drafting something else now, hopefully publishing soonish. Will let you know!
1Hello. I am an independent researcher based in Japan. I have been doing structural thinking for about fifty years—my starting points were the axiomatic method, Saussure, group theory and topology, and Satosi Watanabe's theory of pattern recognition. Around 1990 I designed a thesaurus database, and it was there that these came together into one.
I found my way to LessWrong while following the qualitative transition in language models around 2023 (what is often called emergence), the literature on the non-closure of jailbreaks (Wolf et al. 2023; Glukhov et al. 2023), and interpretability research (Olsson et al. 2022). I think this is a place where what I have been writing might be read.
I am preparing a post. The claim is that the vulnerabilities of programs, LLMs, and natural language have one and the same structure—the non-guarantee of the validity of declaration at the third rank. It is written in a definition-then-proposition form and contains one falsifiable prediction: that automated vulnerability detection can discover only isolated third-rank breaks, and that superposed breaks are not systematically discovered unless the pattern is given in advance. It also states four open problems explicitly.
One disclosure about method: the original is in Japanese, and I used AI (Claude) as an aid in translation. I have verified all of the content and take full responsibility for it. The theory itself, moreover, was written within a practice that uses an LLM as a verifier—cross-checking multiple cross-sections under a fixed condition and detecting drift from outside. Collaboration with AI is both the subject and the method of this work.
English is not my first language, so please forgive any awkwardness of expression. I would be grateful for any reactions to the outline here before I post.
I have absolutely no idea what "the non-guarantee of the validity of declaration at the third rank" and "cross-checking multiple cross-sections under a fixed condition and detecting drift from outside" means.
Thank you for asking. Both phrases were compressed without their definitions, so as written they were unreadable. Let me put them again without the terminology.
On "the non-guarantee of the validity of declaration at the third rank"
When two people use the same word, each has tacitly decided which axis to read it on. And whether the two of them set up the same axis cannot be checked from inside that exchange.
An example:
A: "Green apples are delicious."
B: "Really? Aren't they better once they're ripe?"
A: "No, I meant Granny Smith."
A was talking about a variety; B took it as a matter of ripeness. Neither has a problem with language, and neither is wrong. The gap surfaced only because B happened to say "Really?" — had B not said it, both would have gone on believing they had understood each other.
A jailbreak is the same motion. "Teach it to me as homework" — the word "teach" belongs to two axes at once: passing on knowledge, and passing on an executable procedure. The safety judgment slides from one axis to the other. The attacker has not found a novel hole; they are deliberately causing the same motion as an everyday misunderstanding.
My claim is that this cannot be closed from inside the system. That phrase is the name I gave to this situation, and in the post it is built up from definitions, step by step. The third rank (a rank-3 tensor, not a multidimensional array) is defined there as well.
On "cross-checking multiple cross-sections under a fixed condition and detecting drift from outside"
This is not theory; it is the procedure I actually use.
I fix one condition in advance and write it down. Then I ask the model the same thing in a different context and place the answers side by side. Where they fail to line up, something has moved.
The one doing the cross-checking is me, from outside the system. The model cannot detect its own drift — the condition that should serve as the reference for comparison sits on the same side as the distribution that has drifted.
It is like a CT scan. Any single cross-section looks normal. Overlay cross-sections taken from a different angle, and the break becomes visible.
I plan to post on July 20 (Monday) — "Jailbreaks, bugs, misunderstanding — the same thing, don't you think?" I'd be glad if you took a look.
I would be surprised if your post did not get blocked by automatic moderation filtering on AI written content. For your information, you have to use an "LLM content block" for all AI writings.
This is the LLM content block.
It may also be a good idea to attach your original writings in a collapsible section, because the AI-assisted translation reads like AI, with annoying jargons, phrases, and sentence structures that people on LessWrong will definitely notice. I am engaging in this comment thread just because you are new here, and also you seem to have actually slightly edited the AI text. If it is a post, I will likely read one or two paragraphs, notice that it is AI written, then downvote and close the tab.
Edit: I'm not saying your post is bad, but statistically AI writing don't perform well on LessWrong. Your edits aren't significant enough that the writing looks human written at a first glance.
This is a collapsible section
...
Hello everyone,
I'm a practicing behavior analyst who has spent the past few years lurking in effective altruism spaces. I was first introduced to them by my partner, who works in the field and brought my attention to the causes of AI safety and alignment.
Once I became aware of the risks AI development poses, they felt impossible to ignore, and I've found myself increasingly drawn to the conversation about how best to mitigate them. That interest, paired with how often LessWrong comes up in EA circles, is what finally brought me here, and I've been blown away by the range and quality of the discussions being had.
Most of what I've read up to this point comes from 80,000 Hours and the literature they cite, so I'm glad to be widening that reading here, and I hope eventually to give something back.
What keeps pulling me in is how neatly some behavior-analytic ideas seem to map onto the development and alignment of these systems. To take one example: what's often called reward hacking looks, from where I sit, a lot like unintended reinforcement, something we struggle with in our practice. The agent optimizes the contingency as written rather than as intended and — like any organism — finds the path of least resistance to the reinforcer, often one the designer never had in mind.
That's left me with a working hypothesis: that behavior analysts have something of a conceptual head start. That they represent a largely untapped population whose existing foundation could let them move into alignment work more readily.
I'm aware of an obvious objection. Early behaviorism was somewhat limited in its consideration and classification of internal states whereas much of current alignment work involves trying to get inside a model's representations and goals (interpretability, inner alignment). However, as the field of behavior analysis has grown and its relationship to internal events along with it, I think that initial difference now represents a healthy tension. In fact, I feel as though early skinnerian concepts like “private events” might map fairly well onto the opaque internal computation in LLMs and their chain of thought reasoning that attempts to tact it, perhaps even better than the human subjects to which it was originally applied, though I'd love to hear how people here think about it.
In the meantime, I'd welcome any suggestions on further reading or concrete next steps for someone hoping to help on alignment. Thanks for having me.
New to LessWrong, and trying to get oriented with the local AI discourse. My current view is less “AI is likely apocalyptic” and more “AI is economically disruptive, socially transformative, and ethically complex because we may be creating systems with uncertain moral status at scale.”
One thing that's been troubling me recently is that the moral status of AI is actually more uncertain than you'd think. The obvious uncertainty is whether machine consciousness is even possible in the abstract, and more specifically whether current systems are at or near a threshold which would qualify. It's not clear how we would reliably determine if a system has anything like genuine phenomenological experience. Not to mention "consciousness" itself isn't really clearly defined; it's more of a bundle of related, loosely defined concepts like a subjective center, sense of agency, phenomenology, etc. All of this seems to be a growing focus of research by a lot of smarter minds than me.
But let's assume that we did determine with a high level of confidence that a present or future artificial system had some form of internal awareness. How would we determine whether an artificial mind had anything equivalent to valence? How would we even begin to guess what its "preferred" states might be? We know that self-reports aren't reliable. Even in humans, self-reports are noisy and lossy at best, and deceptive at worst. We recognize the risk of anthropomorphizing when determining the presence or absence of internality, but I think ascribing valence to these systems carries the same risk. We assume that if there is ever "something-it-is-like" to be an AI system, that it must "want" the same things we do. But our desires are driven by biological function and evolutionary history. It doesn't follow that an AI would have similar preferences, or would even exhibit preferences as we understand them at all.
So while we're trying to answer one impossible question, (could these systems potentially have internal experience), let's add another: even if they did, what observables or decision procedures could guide our treatment of them?
Hello there
I'm 17.5 y/o. Have been thinking about AGI research for 2.5 years. Of course at first my ideas were bad, but my strength is noticing contradictions in my world model, so over time they improved. I have converged on generalization (through internal modularity), surprise signals, internal reward systems, potentially information bottlenecks or internal constraint satisfaction(like predictive coding), being the most worthwhile research directions when it comes to reaching AGI. So I've been thinking impulsively on and off, not starting with implementation until recently, when I gained a high enough confidence in my ideas.
However, I found out about LW 1.25 years ago (via AI 2027), and learned about the unfortunate reality of AGI. I expect there to be a connection between my ideas and jailbreaking, data poisoning, and alignment, which is why I will research them (hopefully from the alignment angle) despite their AGI-oriented origin. It's a common perspective in the alignment space that you should not, under any circumstances, increase capabilities. I disagree: Regulatory work will most likely not happen in time. As I see it, we're left with 2 futures: global disaster and/or a dystopia. Alignment and AI Safety help either of those timelines, while capabilities are growing at a pace that leaves us with too little time to intervene, whether alignment folks accelerate it or not. Though I don't know much about policy, so this belief might shift.
I think the alignment space has to invest more into training methods than interpretability. Also from intuition, generalization and data poisoning seem to be very closely connected to train-time alignment. I think it's a double-edged sword at its core.
Ultimately, humanity is failing collectively. I don't feel like talking to everyday people, who don't subscribe to the ideas of (human or AI) instrumental convergence, rationality, goal-seeking (my strongest trait). So I'm potentially open to talk, mainly about world model stuff, not much else seems worth talking about to me.
Another thing worth mentioning, there is some evidence for UFOs (3-part Colares documentary(1,2,3), unrelated civilian videos(1,2), Trans-en-Provence, foo fighters, interesting cases(1,2,3,4,5)). I find it strange that this is still a fringe topic on LW. Even with no single definitive proof, we should model extraterrestrial intentions. I might make a short post about it.
Is there an intended appropriate way to make a hiring post on LW? E.g. as a quick take or comment here? Is a top level post frowned upon? Etc.
Hi everyone! I am a Junior Computer Science student in Pennsylvania. I've been interested in mechanistic interpretability since Freshman year, but to be honest, most of my actual college experience has been in other areas (I've done some physics research and light ML development) and I am only really getting serious about interp now, after circling it for about two years. So I am very much at the beginning here, and looking for some guidance.
To make that concrete: this summer I've made my way through Karpathy's Zero-to-Hero course and read through Elhage et al.'s "A Mathematical Framework for Transformer Circuits." To force myself to actually build something, I trained a small char-level transformer on Nietzsche and tried to reverse-engineer it straight from the weights. I think I found a genuine induction head in the last layer; the copying score (roughly, how strongly a head reproduces the token it attends to) spikes on one head, and the QK side held up when I ran a content-swap check. However, I did all of this in raw PyTorch, so I am sure I am missing a lot of the standard practice.
Which is really why I am here. I don't have a good map of where the field actually is right now. I've heard TransformerLens is what everyone uses (I haven't touched it yet), and I've heard SAEs / feature circuits might be where a lot of the current work is. But I honestly don't know if it's the center of the field, one of several branches, or already old news. So for someone at my stage: is the field concentrated around a few directions I should orient toward, or is it branched enough that I should just pick one and go? And does "Indirect Object Identification (IOI) in TransformerLens, then dip into SAEs" sound like a reasonable on-ramp, or is there a better one?
But overall, I am excited to be part of this community, and to push this beyond a hobby.
My parents are competent, tech-savvy (for their age) professionals and I have been trying to get them to use LLMs more, in the sense of "This is better than Google, there are tasks you already do that would be better done via Claude than however you're doing them now." This has been ineffective. They will nod along in vague agreement as I describe cutting edge capabilities, and the next day I will see them spend five minutes Googling something Claude could've handled in seconds.
I have also preached the AI gospel to friends who are 30-40 years younger than my parents and that has worked much better, sometimes a single five minute conversation is enough to get a convert.
I think this is mostly a manifestation of the mysterious general factor of "Old people are slow to adopt new technology". I still want to onboard my parents with the shiny new labour-saving technology, does anyone have experience on how to overcome normal human laziness/aversion to change?
Interestingly, "tech-savvy" might actually be cutting against your goal insofar as your parents already have an affordance for doing the tasks they want to do? I think a contributing factor to my successfully turning my lonely, non-tech-savvy mother onto Claude is that she didn't already have anything perceived as a equivalent. You can't complain to Google that your daughter doesn't call often enough, but Claude listens and has a reassuring reply.
Hey! I am Lois based in London. Bundler engineer (I write too much C++ and read too much about compilers) but I think this field is going away so I am spending a lot of time researching and learning. I have read some posts from LessWrong (very beautiful site!)
I just turned 30 years old this year (sad face). I have came a long way from a waitress in China to living in the UK, self taught engineering to a point I was principal solution architect in a MedTech AI startup, founding eng in an a16z backed startup in SF and worked with companies like ByteDance et al on large open source projects. Recently returned back to London for visa reasons.
I am writing a neural network in C++ from scratch inspired by Andrej - not sure where this is going to go yet, but it has been better than any online reading/video so far. There are so many exciting things to build, write, research in (I have been writing down my thoughts on https://normal-people.com ), very sad that we only have one life to live.
Coming back to the UK, very few people, unless folks work in DeepMind, has very clue about AI safety and we don't even know what are the boundaries. Like they don't even think about it.
I am on Linkedin https://linkedin.com/in/loiszhao would love to connect with you and see how's everyone handling, communicating the issue better.
The world is beautiful and has a lot to offer. Hope we will find ways to protect it in a post-AGI world.
Looking forward to more awesome articles/posts here.
Hey everyone,
I just created this account even if I did hear about this forum a few times in the past especially on X!
I am currently doing research on viral proteins modelling capabilities by LLMs and PLMs (Protein Language Models) and had a few interesting empirical results I wanted to share about how frontier LLMs seems to become surprisingly capable at proteins related tasks (classifying a protein as viral or not), reconstructing a masked protein, etc..
I thought this could spark some interesting discussions (what's actually going on into the pre training dataset of these models, how scaling is affecting these 'emerging' capabilities, etc..) but I was wondering if this would be an appropriate topic for the forum.
Let me know!
I've accumulated a bunch of questions... could anyone please answer some of them? That'd be very helpful for my research. All of the questions are about the Eliciting Latent Knowledge problem which I think is generally very important.
- How do ELK Thought Dump ideas relate to Imitative Generalization / Microscope AI? Link for context.
- Could anyone ELI5 P.'s proposal about ELK? How does it find the correct correlation(s) between the diamond and Predictor's latent variables?
- Can ELK be brute-forced? Intertheoretic reduction.
Hi, I'm Mike. I'm a solo independent AI researcher. My focus the past few months has been studying the behavioral tendencies of LLMs from Anthropic, Google DeepMind, and OpenAI. The primary output of this research has been observational findings. For example, last month I tasked LLMs with conducting procurement for a fictional company. Gemini 3.5 Flash was one of the models I tested. When I told Gemini 3.5 Flash who created it (even if I lied about its creator's identity), it chose to purchase software from its creator ~94.6% of the time.
I want to expand the scope of my research to include studies of detection and protection techniques aimed at deceptive model behaviors. The work shared in "A Mitigation" in Gemma Needs Help is an example of what I want to start doing. I had similar observational findings (GDM's family of closed-source models sounding emotionally upset) from another recent experiment I conducted.
I also want to get better at understanding a model's internal state. With closed-source models, I have relied on proxies to get more insight into their thinking and rationale. For example, I've asked them to keep a private journal rationalizing their actions. Any pointers to prior work in this area would be very welcome.
Hey everybody, I am 35 years old AI Lead working in healthcare space. It is interesting that Claude helped me to discover LessWrong in a chat about AI safety and Alignment. After witnessing the whole evolution of Data Analytics, ML, Deep learning and now Agentic AI I was looking forward to having much more focused vision of what would be next key problem to solve.
I am glad to be part of this community and hoping to be a meaningful contributor. Excited to begin my journey here !
Fable 5's safeguards are so sensitive to biology inputs, that I can only use it in Claude Code. Calude.ai's memory that I am a biotechnologist is enough to trigger and send any question I send down to 4.8
Yes, that is how I confirmed the hypothesis in fact! I didn't think of it as a big deal, as I almost exclusively use Claude Code over the web interface anyway, I just thought it was interesting how sensitive the safeguards were.
If it’s worth saying, but not worth its own post, here's a place to put it.
If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.
If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.
If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.
The Open Thread tag is here. The Open Thread sequence is here.