Open Thread Winter 2025/26

kave

LESSWRONG
LW

Open Thread Winter 2025/26 — LessWrong

23 Open Thread Winter 2025/26

by kave

2nd Dec 2025

1 min read

23

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.

Open Threads

Personal Blog

23

New Comment

71 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:55 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]AugBro3mo200

Hello All,
New to LW and still reading through the intro material and getting a hang of the place. I am ashamed to admit I found this place through Reddit - ashamed because I despise Reddit and other social media.

I came here because I cannot find a place to engage in long form discussions about ideas contrary to my own. I dream of a free speech platform where only form is policed, not content. Allowing any idea to be voiced no matter how fringe as long as it adheres to agreed upon epistemic standards.

Anyways, I know LW probably is not that place but it is adjacent. It seems most people here want to discuss AI research but hoping to find some communities outside of that topic.

6Screwtape2mo

Hello and welcome! There's a few of us around who discuss things other than AI research, myself among them. I suggest looking at the filtering options for the front page; it's the gear next to Latest, Enriched, Recommended, and Bookmarks. I filter the AI tag pretty heavily down. If you want to lean into voicing fringe ideas around here, I'd suggest reading the LessWrong Political Prerequisites and maybe Basics of Rationalist Discourse. They're not universally agreed upon, but I think they do make for a decent pointer to the local standards.

4Viliam2mo

Are you familiar with Astral Codex Ten (also called ACX)? The people there are also mostly smart, the rules of discussion in Open Threads are more relaxed... which can be a good thing or a bad thing, depends.

[-]MissReaper1mo120

Hello, all!
Long-time lurker here. I'm a recent psychology undergraduate who cares about kelp forests and people. I'm currently exploring the viability of blue carbon sequestration and alt protein projects(which involve kelp). This is part of my broader investigation into climate change risks and adaptation strategies.

I'm trying to find the best ways to use my limited time and resources to choose an effective career path and test my fit for different self-employment options. I am currently testing my fit for coaching and coding. I've also restarted my exploration into philosophy, specifically Stoicism, with Marcus Aurelius' Meditations.

I'm also a yoga practitioner and teacher. I'm interested in learning how yogic philosophy and rationality align (I began thinking of this after living with someone who is hyper-rational but is passionately indifferent towards yoga).

I'm also a nut for life optimisation, but struggle with executing optimisation strategies.

Being a lurker is fairly low effort, but I wanted to begin interacting more with the cool people here. I'm still quite intimidated by the whole karma system for posting, but I think I'll find my way around it fairly quickly.

Any tips and guidance are much appreciated! I'm a lifelong learner and hyper-curious, so please throw any amount of information at me. Thank you for being part of such a unique community!

[-]Screwtape2mo111

In case folks missed it, the Unofficial LessWrong Community Census is underway. I'd appreciate if you'd click through, perhaps take a survey, and help my quest for truth- specifically, truth about what the demographics of the website userbase looks like, what rationality skills people have, whether Zvi or Gwern would win in a fight, and many other questions! Possibly too many questions, but don't worry, there's a question about whether there's too many questions. Sadly there's not a question about whether there's too many questions about whether there's too many questions (yet, growth mindset) so those of you looking to maximize your recursion points will have to find other surveys.

If you're wondering what happens to the data, I use it for results posts like this one.

2Screwtape1mo

Heads up, I'm planning to close the census sometime tomorrow. You can take it here if you haven't yet!

[-]Alex_Altair2mo102

Hi all! I'm a long-time LWer, but I'm making a comment thread here so that my research fellows can introduce themselves under it!

For the past year or so I've been running the Dovetail research fellowship in agent foundations with @Alfred Harwood. We like to have our fellows make LW posts about what they worked on during the fellowship, and everyone needs a bit of karma to get started. Here's a place to do that!

[-]Santiago Cifuentes1mo110

Hi everyone!

I'm Santiago Cifuentes, and I've been a Dovetail Fellow since November 2025 working on Agentic Foundations. My current research project consists in extending previous results that aim to characterize which agents contain world models (such as https://arxiv.org/pdf/2506.01622). On a similar line, I would like to provide a more general definition of what a world model is!

I've been silently lurking LessWrong since 2023, and I came across the forum while looking for rationality content (and in particular I found The Sequences quite revealing). I am looking forward to contribute to the discussion!

6Margot1mo

Hi everyone! I am Margot Stakenborg, and I have worked with Dovetail in this winter fellowship cohort. I have a background in theoretical physics and philosophy of physics, and now making a switch into conceptual mechinterp, after having been interested in it and learning about it for some years. I have been working with Dovetail on formalising world models, I am writing up a sequence of posts on the philosophical and mathematical prerequisites for proper world models, and which tools from physics can help us understand and analyse different world models, and I will dive into the different definitions of "world model" that float around in mechinterp and AI safety literature. Things I will discuss are: * How is the concept "world model" used in different areas of ML literature * Concept representation in the brain: new frontiers from neuroscience * Tools from physics: renormalisation and coarse-graining * What are "natural features"? * When can networks find similar representations of the world as we do? * Can NNs discover new natural kinds? * Theoretical equivalence and intertheoretic reduction * Bayesian experimental design * And probably more.. I hope to build this out into a quite comprehensive and complete sequence. Do let me know if there are other questions or subjects you would be interested in to read about!

5GuilleMarSan1mo

Hello! I'm Guillermo, a fellow in the Winter25 cohort. I have a background in mathematics, computer science and particularly computational neuroscience. In my project I am looking at the Reward Hypothesis in decision theory and reinforcement learning theory and would like to write a digest of what are the main results that connect a preference order from order-preserving functions, to expected utility maximization and reward functions (with discount factors). I furthermore would like to formalize some of the key results in Lean. Overall I am interested in topics that connect rationality and decision theory all the way to practical aspects of machine learning and reinforcement learning, to try to bridge these topics for AI Safety. Nice to meet you all!

4Leo Cymbalista1mo

Hello everyone! I am Léo Cymbalista, one of the Dovetail fellows since November 2025. I’m a physics undergraduate in the process of switching to theoretical AI safety research. My current research project is writing an explainer for computational mechanics, which is almost finished. I hope to use the knowledge I acquired while researching for it to answer questions such as “given 2 coupled stochastic processes, when can we say that one is modeling the other?”, which could be useful for investigating the presence of world models in agents. I have only known about LW (and AI safety) for about a year, so I’m still not very familiar with it, but it seems very interesting so far!

4Vardhan1mo

Hi, I’m Vardhan, one of the Dovetail fellows this winter. Thanks Alex & Alfred for running this! Background: I study mathematics and computer science (probability, algorithms, game theory) and I’m interested in formal models of agents and multi-agent interaction. For the fellowship, I looked at the question: Which agents can be faithfully described by finite automata / finite transducers, and which structural properties make that more or less likely? In other words, when can an agent’s externally observable behavior be captured by a finite (possibly stochastic) automaton, and what observable signatures indicate that a finite-state model is impossible or misleading? I’ve written a brief report summarizing definitions, toy examples, and some light lemmas. I’m planning a longer post with formal definitions, more examples, and proofs. I’d really appreciate recommendations on literature I may have missed (especially anything linking automata/dynamical-systems perspectives to algorithmic information theory, ergodic theory, or learning theory). Comments, questions, and pointers very welcome!

3Robert Adragna1d

Hi everyone! My name is Robert Adragna, and I’ve been working with Dovetail this winter fellowship cohort on Agent Foundations. Specifically, I’ve been trying to better understand what background assumptions the Natural Abstractions Hypothesis (NAH) makes about the world, and whether they might be learned in existing LLM systems. Specific questions that I’m exploring include: 1. Is the Platonic Representation Hypothesis from deep learning evidence for the Natural Abstractions Hypothesis? 2. Is it possible to construct a dataset which represents the world in a completely unbiased way? 3. How can Natural Abstractions be both universal & observer/goal dependant? 4. What would it take to empirically test the NAH? I’ve been lurking on LessWrong since 2024, when I got interested in AI Safety, and am very excited to spend more time engaging with the community.

[-]Ninety-Three23d91

Is there a way to make the list of posts shown on lesswrong.com use the advanced filters I have set up at lesswrong.com/allPosts? I hate hate hate all of Recent, Enriched and Recommended (give me chronological or give me death) but given that I already have a set of satisfactory filters set up, rendering them on the main page seems like a feature that should exist, if only I can find it.

[-]Oliviero2mo70

Hi! I'm very new to LW.

I found this website while searching up philosophy websites that are useful. I've been looking around LW for about a week now, just reading and learning peoples takes. There's a lot of it and it's great if you ask me.

I'm still learning the guidelines and the karma system, which had been a little intimidating, but I'm getting the hang of it now. I do recognise that LW is more professional than I originally thought, especially professional for my age, but it's not like I'm applying to work for Nasa or anything.

That's just me, though. I would greatly appreciate any tips for navigating, filtering content, etc.

5Viliam2mo

A good resource to get familiar with the basics of LW approach to life, universe, and everything is https://www.readthesequences.com/

[-]Sodium2mo6-2

I feel like the react buttons are cluttering up the UI and distracting. Maybe they should be e.g., restricted to users with 100+ karma and everyone gets only one react a day or something?

Like they are really annoying when reading articles like this one.

[-]habryka2mo130

Yeah, I agree with this. I think they are generally decent on comments, but some users really spam them on posts. It’s on my list to improve the UI for that.

4gilch2mo

Seems like a signal-to-noise problem. Some amount seems like a useful signal, but too much is too hard to digest. Privileges based on karma make some sense but restricting it based on time (1/day/user or something) seems pretty crude, so I don't like that idea. Not sure if this is a good idea either, but the number of reacts allowed per post could be based on the amount of karma that user generated on comments on that post. That way, a user who's doing too many reacts would be encouraged to just write a comment instead. That still doesn't seem like exactly the right incentive, but I'm also not sure how I want it to work. Maybe the ability to filter out reacts from a particular (prolific) user would suffice?

1Sodium2mo

Yeah I would like to mute some users site-wide so that I never see reacts from them & their comments are hidden by default....

1TristanTrim2mo

Do you have any thoughts on those UI improvement written down anywhere? I'll admit to being one of the users that really spams reactions on posts. I like them as a form of highlighting for review and as a form of backchannel communication. I would be much happier if people would use more reacts towards me. So I would be upset with UI modifications to restrict reacts, but fully support updates to make the UI around viewing reacts cleaner and more useful. I wrote a longer comment with some feature suggestions. If you have time it would be nice to hear your thoughts.

5Steven Byrnes2mo

Part of it is the “vulnerability” where any one user can create arbitrary amounts of reacts, which I agree is cluttering and distracting. Limiting reacts per day seems reasonable (I don’t know if 1 is the right number, but it might be, I don’t recall ever react-ing more than once a day myself). Another option (more labor-intensive) would be for mods to check the statistics and talk to outliers (like @TristanTrim) who use way way more reacts than average.

2TristanTrim2mo

[EDIT: I think issues stem from different people using reacts in different ways and having different assumptions about their use. I think I am probably using them in a less common way than other people, but I also find myself believing I am using them in a better way than other people. As such, I am trying to put in effort to communicate my POV. I would appreciate if anyone who disagrees with me would do so with a higher bandwidth signal than just pressing the "Agreement: Downvote" button. Perhaps by using some inline reacts on my comment?] Haha! Sorry if I'm bothering anyone! ☺♡ I really like reacts and am bothered in essentially the opposite direction as Sodium in that I think reacts are a very useful backchannel communication, and see it as a minor moral failing that most users do not use them more. I think it's great that many reacts are based on LW ideals for discourse. I don't know exactly how they are managed, but I think they could be even more valuable if there was some team that reviewed how people are currently using them and then improved and updated react descriptions and usage guides based on that. A descriptivist approach. I also think a prescriptive approach would also be good. People should be suggesting concepts for reacts that they think would be valuable for communication, and people should be figuring out how to promote proper use of reacts. I do agree that relevance may be an issue. I would like it if everyone would drop ~10 reacts while reading a post, but then, if all of them showed in the UI, it would be too noisy to make sense of easily. I think there are a few ways around this: * [EDIT: I've discovered on the triple dot menu it is already possible to select for inline reactions to hide all, hide downvoted, or show all. I think a plausible sane modification of this would be to make the default to hide under 2 or 3 votes and always show reactions to highlighted text. However I think some kind of more complicated scheme could be better

4Screwtape2mo

My two cents, I'm happy with the amount of reacts I usually see and would probably enjoy about 20% more. Thank you for chipping in your two cents!

3Karl Krueger2mo

I use the "typo" reaction and hope it is useful for authors, but I don't ever go back to remove it if the typo has been corrected. I'm not even sure what happens in that case.

[-]habryka2mo100

We recently made it so that authors can remove typo reacts themselves. It’s still a bit annoying, but it’s less annoying than before!

[-]Richard Amerman2mo60

Hello,

I'm very happy to be here!

Unfortunately I'm only just bringing LessWrong into my life and I do consider that a missed opportunity. I wish I had found this site many years ago though that could have been dangerous as this could be a rabbit hole I might have found challenging to escape, but how bad would that have actually been? I'm sure my wife would not have been thrilled. My reason for coming here now unfortunately, especially at this point in time, is very unoriginal. In the last eight months I've taken what was a technology career possibly in its ... (read more)

1TristanTrim2mo

Seems like JustDone gives abnormally high AI content estimations. Plausibly this is to scare you into using their "text humanizer" in which an AI re-writes what you wrote to make it seem less like an AI to an AI... I weep for humanity. I'd recommend reading and commenting until you have enough karma to submit your post to the LW editor who can more straightforwardly tell you why your post would or wouldn't be rejected. PS: I would like to encourage you, like everyone, to stop focusing on AI capabilities and instead focus on AI interpretability and preference encoding.

[-]Carlo Martinucci1mo50

Hello! Long time reader, I regularly run a local ACX meetup in Padova, Italy. My entry points for the rationalist community were ACX and HPMOR, but I also loved The Story of Us blogpost series by Tim Urban (now collected into a book).

At the beginning of 2025 I left my job at Bending Spoons to study AI alignment (I took the https://www.aisafetybook.com/virtual-course, much recommended), and finally decided to tackle the other problem I'm most interested in, which is social polarization.

With an ex colleague I founded https://unbubble.news, a tool that uses L... (read more)

[-]Sodium1mo51

I'm curious: what percent of upvotes are strong upvotes? What percent of karma comes from strong upvotes?

5kave1mo

Here's the tally of each kind of vote: Weak Upvote 3834911 Strong Upvote 369901 Weak Downvote 426706 Strong Downvote 43683 And here's my estimate of the total karma moved for each type: Weak Upvote 5350471 Strong Upvote 1581885 Weak Downvote 641568 Strong Downvote 206491

2gilch1mo

The mods may have better overall data, but personally, I weak vote a lot more than I strong vote, and I don't vote on everything I read either.

[-]derfriede2mo50

Hello! I chose the name “derfriede” for LW. This is my first post here, which I am happy about. I have read some of the introductory materials and am very interested.

What interests me? First of all, I want to explore the topic of AI and photography. I study the theory and philosophy of photography, look for new approaches, and try to apply a wide variety of perspectives. I think it's useful to address the question of what AI cannot do. It's very similar to researching glitch culture. Okay, I'll stop here for now, because I just want to get acquainted.

Have a nice day, wherever you are!

1CstineSublime2mo

I'm sure that hobbyists on Civitai or TensorArt have some thoughts on it. Many LoRAs are made to evoke antiquated camera technologies, digital and analog (although they often incorporate elements of what we may call 'art direction' like costume and furnishing of spaces to match the formats). I think most people aren't aware of how much AI there already is, and has been, in their smartphone and the influence that has on their photos.

1Roman Malov2mo

Welcome! The only thing I can think of on the intersection of AI and photography (besides IG filters) is this weird "camera", which uses AI to turn a little bit of geographical information to create images. Do you know of any other interesting intersections?

[-]Ninety-Three8d40

Lately I've seen several front page posts that read as obvious slop and Pangram reports as 100% AI-generated. I assume that this is frowned upon here, so I suggest that LW add in an automatic Pangram API call (cost: 5 ¢/1000 words) at some point before a post gets frontpaged.

3habryka8d

We already have this for admins! I am planning to make it visible to everyone sometime in the next week or two, and also add some triggers that even for established users, we want to do a manual review pass if Pangram thinks it's AI written.

1Ninety-Three8d

Oh neat, great minds think alike. But I would have assumed that as soon as you had Pangram hooked up you would make sure 100% AI works don't get front-paged, and you've only mentioned it as a visibility feature here. Is there going to be an algo change as well when it goes public?

3Ben Pace8d

~90% of our daily mod effort goes into new users, where we are actively tracking and rejecting content on the basis of being LLM written. Been a bit sad to find that people who've been around for many years have been submitting LLM-written content, but yeah, I had just brought up internally that we'll have to start doing this for all content.

2kave8d

I think in most cases that a >5k karma user posts something that's 100% AI, it's better to let it through (though I expect I would strong downvote it).

4Ben Pace8d

Why's that? Sounds like you agree it's a strong signal of low-quality / spammy content.

5kave6d

Folks with 5k+ karma often have pretty interesting ideas, and I want to hear more of them. I am pretty into them trying to lower the activation energy required for them to post. Also, they’re unusually likely to develop ways of making non-slop AI writing There’s also a matter of “standing”; I think that users who have contributed that much to the site should take some risky bets that cost LessWrong something and might payoff. To expand my model here: one of the moderators’ jobs, IMO, is to spare LW the cost of having to read bad stuff and downvote it to invisibility. If LW had to do all the filtering that moderators do, it would make LW much noisier and more unpleasant to use. But users who’ve contributed a bunch should be able to ask LW to make that judgement directly. That said, I do expect I’d strong downvote. LLM text often contains propositions no human mind believes, and I’m happy to triage to avoid reading a bunch of sentences no one believes. But I could be wrong and if there’s a strong enough quality signal, I’d be happy to see that. For example, consider Christian homeschoolers in the year 3000. I’ve not read it; I bounced off of it. Based on Buck’s description of his writing process, I think it’s quite likely it would have been automatically rejected. (Pangram currently only gives it an LLM score of 0.1, though). I think writers like Buck might like to try more experiments like that in the future, with even more LLM prose. My guess is that LW is better off for having that post on it than not.

2ChristianKl8d

I think the idea is that >5k karma users have karma to lose to punish them for posting low-quality content and it's better to have humans make the judgement about what's low-quality than AI.

1Ninety-Three7d

I saw another Pangram 100% on the front page, this one from a 1 day old account that somehow slipped through the cracks. I guess you'd know firsthand at this point if there's a false positive rate to worry about, but from the user side it feels like it'd be a strict improvement if LW was configured so that 100% cases never get frontpaged.

2Ben Pace7d

Plz DM that to me? We do have auto rejection for 100% pangram for new users, so that sounds like there was a human error involved.

[-]TrE2mo41

I have some time on my hands and would be interested in doing something meaningful with it. Ideally learn / research about AI alignment or related topics. Dunno where to start though, beyond just reading posts. Anyone got pointers? Got a background in theoretical / computational physics, and I know my way around the scientific Python stack.

3TristanTrim2mo

AI alignment has been getting so much bigger as a field! It's encouraging, but we still have a long way to go imo. Did you see Shallow review of technical AI safety, 2025? I'd recomment looking through that post or their shallow review website and finding something that seems interesting and starting there. Each sub-domain has its own set of jargon and assumptions so I wouldn't worry too much about trying to learn the foundations since we don't have a common set of foundations yet. Just reading posts isn't bad, but since there isn't that common set of foundations, it could be confusing when you're just starting out (or even when you're quite experienced). Good luck and glad to have you!

[-]AlexandraChirila1mo30

Hello everyone! I'm very new to the LW community and I'm still trying to understand how this platform works, but I'm glad to have found a space where people can engage in meaningful conversations. I am a philosophy PhD (defence scheduled next month, wish me luck!) and my thesis is about the philosophy of mind and AI. I'll be spending the next hours (days) reading and I hope to post some of my slightly less formal writing once I get the hang of this platform. I can't wait to explore!

[-]daphnejarjoura1mo30

Hi everyone!

New to LW. Recently I've been interested in AI research, especially mech interp, and this seems to be the place that people go to discuss this. I studied philosophy in undergrad and while since then I've gotten interested in CS and math, my predilections still tend toward the humanities side of things. Will mostly be lurking at first as I read through The Sequences and get used to the community norms here, but hope to share some of my independent research soon!

[-]RichardT1mo30

Hello everyone,
Just a quick "Hi" and figured I'd intro myself as I'm new to this space.
As part of my new year's resolution to "do something different" this year (beyond the yearly failed attempt to exercise more, and eat/drink less) I thought that this is something I can achieve - and enjoy doing.
So let's see where to start?
I live in Canada, in my 5th decade, am a family man and work in computing. I in fact enjoy being proven wrong - as it helps to show I am still learning.
I enjoy long walks on the beach, and am at equally at home at the opera as I am at a baseball stadium .. wait .. sorry that was for the dating site ... don't tell my wife ;)
Jokes aside, looking forward to being a lurker!
Richard

[-]Noosphere892mo*30

Now that it is the New Year, I made a massive thread on twitter concerning a lot of my own opinionated takes on AI, which to summarize are my lengthening timelines, which correlates to my view that new paradigms for AI are likelier than they used to be and more necessary, which reduces AI safety from our vantage point in expectation, AI will be a bigger political issue than I used to think and depending on how robotics ends up, it might be the case that by 2030 LLMs are just good enough to control robots even if their time horizon for physical tasks is pre... (read more)

[-]ConformalInfinity3mo30

Hello, I am an entity interested in mathematics! I'm interested in many of the topics common to LessWrong, like AI and decision theory. I would be interested in discussing these things in the anomalously civil environment which is LessWrong, and I am curious to find out how they might interface with the more continuous areas of mathematics I find familiar. I am also interested in how to correctly understand reality and rationality.

2TristanTrim2mo

Hi! What sorts of mathematics are you interested in? I'm interested in topology and manifolds which I hope to apply to understanding the semantics of latent spaces within neural networks, especially the residual stream of transformers. I'm also interested in linear algebra for the same reason. I would like to learn more about category theory, because it seems interesting. Finally, I like probability theory and statistics because, like you, I'd like to "correctly understand reality and rationality".

2ConformalInfinity2mo

Hi. I am interested in much of the mathematics which underlies theories of physics, such as complex analysis, as well as most of mathematics, although I sadly do not have the capacity to learn about the majority of it. Your interests seem interesting to me, but I do not understand enough about AI to know exactly what you mean. What is the residual stream of a transformer?

4TristanTrim2mo

Sadly, it's a problem you share with me and most humans, I think, with possible rare exceptions like Paul Erdős. I'll try to build up a quick sketch of what the residual stream is, forgive me if I say things that are basic, obtuse, or slightly wrong for brevity. All neural networks (NN) are built using linear transformations/maps which in NN jargon are called "weights" and non-linear maps called "activation functions". The output the activation functions are called "activations". There are also special kinds of maps and operations depending on the "architecture" of the NN (eg: convNet, resNet, LSTM, Transformer). A vanilla NN is just a series of "layers" consisting of a linear map and then an activation function. The activation functions are not complicated nonlinear maps, but quite simple to understand. One of the most common, ReLu, can be understood as "for all vectors, leave positive components alone, set negative components to 0" or "project all negative orthants onto the 0 hyperplane". So, since most of the complex behaviour of NNs is coming from the interplay of the linear maps and these simple nonlinear maps, so linear algebra is a very foundational tool for understanding them. The transformer architecture is the fanciest new architecture that forms the foundation of modern LLMs which act as the "general pretrained network" for products such as chat-GPT. The architecture is set up with a series of "transformer blocks" each of which has a stack of "attention heads" which is still matrix transformations but set up in a special way, and then a vanilla NN. The output of each transformer block is summed with the input to use as the input for the next transformer block. The input is called a "residual" from the terminology of resNets. So the transformer block can be thought of as "reading from" and "writing to" a "stream" of residuals passed along from one transformer block to the next like widgets on a conveyor belt, each worker doing their one operation and

2ConformalInfinity2mo

Thanks for your clear explanation, understanding the topology of the space seems fascinating. If it's a vector space, I would assume its topology is simple, but I can see why you would be interested in the subspaces of it where meaningful information might actually be stored. I imagine that since topology is the most abstract form of geometry, the topological structure would represent some of the most abstract and general ideas the neural network thinks about.

2TristanTrim2mo

Yeah! I think distance, direction, and position (not topology) are at least locally important in semantic spaces, if not globally important, but continuity and connectedness (yes topology) are probably important for understanding the different semantic regions, especially since so much of what neural nets seem to do is warping the spaces in a way that wouldn't change anything about them from a topological perspective! At least for vanilla networks, the input can be embedded into higher dimensions or projected into lower dimensions, so you're only ever really throwing away information, which I think is an interesting perspective for when thinking about the idea that meaningful information would be stored in different subspaces. It feels to me more like specific kinds of data points (inputs) which had specific locations in the input distribution would, if you projected their activation for some layer into some subspace, tell you something about that input. But whatever it tells you was in the semantic topology of the input distribution, it just needed to be transformed geometrically before you could do a simple projection to a subspace to see it.

[-]Brian McCallion13d20

Hi all,
Despite occasional fits of lurking over many years, I'd never actually created a LW account. Sometimes it feels easier, or more appropriate, to peer over the garden wall than to climb in and start gardening. Or at least glance in to see what you might apply to your own small patch of earth.
Lately I've come to realise that approach was more grounded in protection of a shaky personal identity, than dislike of building engagement within an established group. This became especially apparent with recent research, paper & project builds I'd taken on, ... (read more)

[-]Brimstone1mo20

Greetings, Claude sent me here! My goals are primarily self-improvement- I will appreciate engaging with individuals that are able and willing to inform me of weaknesses in my lines of thinking, whatever the topic. Lucky that this place exists. I miss the old internet when authentic honest material was more commonly found rather than ideologically skewed, bait, or persuasion, especially well-disguised persuasion. Basically, just a guy that feels half the internet is attempting to hijack my thoughts rather than present good faith information. Lucky to be here!

[-]inimino1mo20

Hi everyone,

I've read many of the posts here over the years. A lot of the ideas I first met here seem to be coming up again in my work now. I think the most important work in the world today is figuring out how to make sure AI continues to be something we control, and I find most of the people I meet in SF still think AI safety means not having a model say something in publc that harms a corporate brand.

I'm here to learn and bounce some ideas off of people who are comfortable with Bayesian reasoning and rational discussion, and interested in similar topics... (read more)

[-]Sherrinford1mo20

I'm a bit confused about forecasting tournaments and would appreciate any comments:

Suppose you take part in such a tournament.

You could predict as accurately as you can and get a good score. But let's say there are some other equally good forecasters in the tournament and it becomes a random draw who wins. On expectation, all forecasters of the same quality have the same forecasts. If there are many good forecasters, your chances of winning become very low.

However, you could include some outlier predictions in your predictions. Then you lower your ex... (read more)

3papetoast1mo

My knowledge level: I read the metaculus FAQ a couple days ago At least on metaculus the prize pool is distributed among everyone with good enough accuracy, rather than winner-takes-all. So it shouldn't be affected by the (real) phenomenon that you are describing.

3Sherrinford1mo

Thanks, good to know. So I assume there is an incentive difference between monetary incentives that can be distributed in such a way, and the incentive of being able to say that you won a tournament (maybe also as a job qualification).

[-]papetoast2mo28

It would be nice to have a post time-sorted quick takes feed. https://www.lesswrong.com/quicktakes seems to be latest comment-sorted or magic sorted

[-]papetoast4d10

I want to be able to change the editor inside the "New Quick Take" popup

[-]papetoast1mo10

(Reposted from my shortform)

What coding prompt do you guys use? It seems exceedingly difficult to find good ones. GitHub is full of unmaintained & garbage awesome-prompts-123 repos. I would like to learn from other people's prompt to see what things AIs keep getting wrong and what tricks people use.

Here are mine for my specific Python FastAPI SQLAlchemy project. Some parts are AI generated, some are handwritten, should be pretty obvious. This is built iteratively whenever the AI repeated failed a type of task.

AGENTS.md

# Repository Guidelines

## Project

... (read more)

[-]kivispace2mo10

I'm starting to explore AI alignment, and this seemed like a good forum to start reading and thinking more about it. The site still feels a little daunting, but I'm sure I'll get the hang of it eventually. Let me know if there are any posts you love and I'll check them out!

[-][anonymous]2mo-1-1

Hello.

My interests are transformer architecture and where it breaks.
Extending transformers toward System-2 behavior.
Context primacy over semantics.

I’m focused on the return to symbolics.
On the manifold hypothesis, and how real systems falsify it.
Inference, finite precision, discrete hardware.
Broken latent space, not smooth geometry.

I’m interested in mechanistic interpretability after the manifold assumption fails.
What survives when geometry doesn’t.
What replaces it.

I’m also seeking advice on intellectual property.

I’m here to find others thinking along these lines.

1TristanTrim2mo

Interesting. Some thoughts: * I think focusing on increasing transformer capability is bad because we haven't solved the alignment problem. * What do you mean by the "manifold hypothesis"? Can you share links? * Latent space could be both broken and smooth, both broken in terms of subspaces and in terms of different regions of space having different semantics. I think all of this can still be understood in terms of manifolds though. * Transformers are based on matrix transformations. No matter what there is a geometric interpretation, so I'm not sure if you are thinking about non geometric interpretations or future systems that don't have geometric interpretations. * "advice on intellectual property" -- This is very vague, but in general I recommend supporting an overhaul of IP law and calling for large AI and software companies to respect the IP of individual, independent citizens.

-6[anonymous]2mo

Moderation Log