In this post, I proclaim/endorse forum participation (aka commenting) as a productive research strategy that I've managed to stumble upon, and recommend it to others (at least to try). Note that this is different from saying that forum/blog posts are a good way for a research community to communicate. It's about individually doing better as researchers.

I like the fact that despite not being (relatively) young when they died, the LW banner states that Kahneman & Vinge have died "FAR TOO YOUNG", pointing to the fact that death is always bad and/or it is bad when people die when they were still making positive contributions to the world (Kahneman published "Noise" in 2021!).
Novel Science is Inherently Illegible Legibility, transparency, and open science are generally considered positive attributes, while opacity, elitism, and obscurantism are viewed as negative. However, increased legibility in science is not always beneficial and can often be detrimental. Scientific management, with some exceptions, likely underperforms compared to simpler heuristics such as giving money to smart people or implementing grant lotteries. Scientific legibility suffers from the classic "Seeing like a State" problems. It constrains endeavors to the least informed stakeholder, hinders exploration, inevitably biases research to be simple and myopic, and exposes researchers to constant political tug-of-war between different interest groups poisoning objectivity.  I think the above would be considered relatively uncontroversial in EA circles.  But I posit there is something deeper going on:  Novel research is inherently illegible. If it were legible, someone else would have already pursued it. As science advances her concepts become increasingly counterintuitive and further from common sense. Most of the legible low-hanging fruit has already been picked, and novel research requires venturing higher into the tree, pursuing illegible paths with indirect and hard-to-foresee impacts.
I thought I didn’t get angry much in response to people making specific claims. I did some introspection about times in the recent past when I got angry, defensive, or withdrew from a conversation in response to claims that the other person made.  After some introspection, I think these are the mechanisms that made me feel that way: * They were very confident about their claim. Partly I felt annoyance because I didn’t feel like there was anything that would change their mind, partly I felt annoyance because it felt like they didn’t have enough status to make very confident claims like that. This is more linked to confidence in body language and tone rather than their confidence in their own claims though both matter.  * Credentialism: them being unwilling to explain things and taking it as a given that they were correct because I didn’t have the specific experiences or credentials that they had without mentioning what specifically from gaining that experience would help me understand their argument. * Not letting me speak and interrupting quickly to take down the fuzzy strawman version of what I meant rather than letting me take my time to explain my argument. * Morality: I felt like one of my cherished values was being threatened.  * The other person was relatively smart and powerful, at least within the specific situation. If they were dumb or not powerful, I would have just found the conversation amusing instead.  * The other person assumed I was dumb or naive, perhaps because they had met other people with the same position as me and those people came across as not knowledgeable.  * The other person getting worked up, for example, raising their voice or showing other signs of being irritated, offended, or angry while acting as if I was the emotional/offended one. This one particularly stings because of gender stereotypes. I think I’m more calm and reasonable and less easily offended than most people. I’ve had a few conversations with men where it felt like they were just really bad at noticing when they were getting angry or emotional themselves and kept pointing out that I was being emotional despite me remaining pretty calm (and perhaps even a little indifferent to the actual content of the conversation before the conversation moved to them being annoyed at me for being emotional).  * The other person’s thinking is very black-and-white, thinking in terms of a very clear good and evil and not being open to nuance. Sort of a similar mechanism to the first thing.  Some examples of claims that recently triggered me. They’re not so important themselves so I’ll just point at the rough thing rather than list out actual claims.  * AI killing all humans would be good because thermodynamics god/laws of physics good * Animals feel pain but this doesn’t mean we should care about them * We are quite far from getting AGI * Women as a whole are less rational than men are * Palestine/Israel stuff   Doing the above exercise was helpful because it helped me generate ideas for things to try if I’m in situations like that in the future. But it feels like the most important thing is to just get better at noticing what I’m feeling in the conversation and if I’m feeling bad and uncomfortable, to think about if the conversation is useful to me at all and if so, for what reason. And if not, make a conscious decision to leave the conversation. Reasons the conversation could be useful to me: * I change their mind * I figure out what is true * I get a greater understanding of why they believe what they believe * Enjoyment of the social interaction itself * I want to impress the other person with my intelligence or knowledge Things to try will differ depending on why I feel like having the conversation. 
habryka4d5120
10
A thing that I've been thinking about for a while has been to somehow make LessWrong into something that could give rise to more personal-wikis and wiki-like content. Gwern's writing has a very different structure and quality to it than the posts on LW, with the key components being that they get updated regularly and serve as more stable references for some concept, as opposed to a post which is usually anchored in a specific point in time.  We have a pretty good wiki system for our tags, but never really allowed people to just make their personal wiki pages, mostly because there isn't really any place to find them. We could list the wiki pages you created on your profile, but that doesn't really seem like it would allocate attention to them successfully. I was thinking about this more recently as Arbital is going through another round of slowly rotting away (its search currently being broken and this being very hard to fix due to annoying Google Apps Engine restrictions) and thinking about importing all the Arbital content into LessWrong. That might be a natural time to do a final push to enable people to write more wiki-like content on the site.
Recently someone either suggested to me (or maybe told me they or someone where going to do this?) that we should train AI on legal texts, to teach it human values. Ignoring the technical problem of how to do this, I'm pretty sure legal text are not the right training data. But at the time, I could not clearly put into words why. Todays SMBC explains this for me: Saturday Morning Breakfast Cereal - Law (smbc-comics.com) Law is not a good representation or explanation of most of what we care about, because it's not trying to be. Law is mainly focused on the contentious edge cases.  Training an AI on trolly problems and other ethical dilemmas is even worse, for the same reason. 

Popular Comments

Recent Discussion

In brief

Recently I became interested in what kind of costs were inflicted by iron deficiency,  so I looked up studies until I got tired. This was not an exhaustive search, but the results are so striking that even with wide error bars I found them compelling. So compelling I wrote up a post with an algorithm for treating iron deficiency while minimizing the chance of poisoning yourself. I’ve put the algorithm and a summary of potential gains first to get your attention, but if you’re considering acting on this I strongly encourage you to continue reading to the rest of the post where I provide the evidence for my beliefs.

Tl;dr: If you are vegan or menstruate regularly, there’s a 10-50% chance you are iron deficient. Excess iron...

2Elizabeth18h
Did you know you can link directly to comments on LW? It's the timestamp/chainlink icon by the author's name. The provided evidence looked weak to me, and based on other conversations I've seen with Alex Chen I didn't expect digging into it to be productive. If you're concerned I encourage you to investigate the research yourself or just run some n=1 experiments. 

I didn't, thanks! I'm a fairly long-time visitor but sporadic-at-best commenter here, primarily because I feel I can learn much more than I can contribute (present case included).

I'd love to know why you think it's weak. As I mentioned before, it doesn't seem any more than suggestive to me (and to be fair Chen acknowledges as much), but it does seem quite suggestive, and it has introduced a hint of doubt in me.

I get the sense that I've gotten your back up slightly here, which is perhaps not without justification as I admit to having been a touch suspicious... (read more)

You are a rational thinker.

Ever since you were born, you’ve been racing through a universe of ideas: creating, evaluating, disputing, engaging with, and being bombarded by…

Ideas.

Like a particle from the Big Bang, you have bounced around the universe until you found yourself here.

Reading, pondering, considering.

Thinking is the foundation by which we establish our reality.

Over time you should master this skill, and yet people seem to get stuck on ideas. People stumble into ideologies and then keep falling deeper into them. These can be ideologies of philosophy, identity, interests, career, or beyond. 

Just as a particle whizzing around in the universe can fall into a black hole, people too can get stuck on an idea, cross an event horizon, and never come back.

You see this phenomenon often, and it...

Bostrom’s new book is out today in hardcover and Kindle in the USA, and on Kindle in the UK.

Description:

A greyhound catching the mechanical lure—what would he actually do with it? Has he given this any thought?

Bostrom’s previous book, Superintelligence: Paths, Dangers, Strategies changed the global conversation on AI and became a New York Times bestseller. It focused on what might happen if AI development goes wrong. But what if things go right?

Suppose that we develop superintelligence safely, govern it well, and make good use of the cornucopian wealth and near magical technological powers that this technology can unlock. If this transition to the machine intelligence era goes well, human labor becomes obsolete. We would thus enter a condition of "post-instrumentality", in which our efforts are not needed...

I'm wondering what Nick Bostrom's p(doom) currently is, given the subject of this book. He said 9 years ago in his lecture on his book Superintelligence "less than 50% risk of doom". In this interview 4 months ago he said that it's good there has been more focus on risks in recent times, but there's still slightly less focus on the risks than what is optimal, but he wants to focus on the upsides because he fears we might "overshoot" and not build AGI at all which would be tragic in his opinion. So it seems he thinks the risk is less than it used to be beca... (read more)

This is my personal opinion, and in particular, does not represent anything like a MIRI consensus; I've gotten push-back from almost everyone I've spoken with about this, although in most cases I believe I eventually convinced them of the narrow terminological point I'm making.

In the AI x-risk community, I think there is a tendency to ask people to estimate "time to AGI" when what is meant is really something more like "time to doom" (or, better, point-of-no-return). For about a year, I've been answering this question "zero" when asked.

This strikes some people as absurd or at best misleading. I disagree.

The term "Artificial General Intelligence" (AGI) was coined in the early 00s, to contrast with the prevalent paradigm of Narrow AI. I was getting my undergraduate computer science...

2AnthonyC2h
I agree that filling a context window with worked sudoku examples wouldn't help for solving hidouku. But, there is a common element here to the games. Both look like math, but aren't about numbers except that there's an ordered sequence. The sequence of items could just as easily be an alphabetically ordered set of words. Both are much more about geometry, or topology, or graph theory, for how a set of points is connected. I would not be surprised to learn that there is a set of tokens, containing no examples of either game, combined with a checker (like your link has) that points out when a mistake has been made, that enables solving a wide range of similar games. I think one of the things humans do better than current LLMs is that, as we learn a new task, we vary what counts as a token and how we nest tokens. How do we chunk things? In sudoku, each box is a chunk, each row and column are a chunk, the board is a chunk, "sudoku" is a chunk, "checking an answer" is a chunk, "playing a game" is a chunk, and there are probably lots of others I'm ignoring. I don't think just prompting an LLM with "How to solve it" in its context window would get us to a solution, but at some level I do think it's possible to make explicit, in words and diagrams, what it is humans do to solve things, in a way legible to it. I think it largely resembles repeatedly telescoping in and out, to lower and higher abstractions applying different concepts and contexts, locally sanity checking ourselves, correcting locally obvious insanity, and continuing until we hit some sort of reflective consistency. Different humans have different limits on what contexts they can successfully do this in.

Absolutely.  I don't think it's impossible to build such a system.  In fact, I think a transformer is probably about 90% there.   Need to add trial and error, some kind of long-term memory/fine-tuning and a handful of default heuristics.  Scale will help too, but no amount of scale alone will get us there.

2AnthonyC3h
Oh, by "as qualitatively smart as humans" I meant "as qualitatively smart as the best human experts". I think that is more comparable to saying "as smart as humanity." No individual human is as smart as humanity in general.
2AnthonyC3h
This is an excellent short mental handle for this concept. I'll definitely be using it.

On Wednesday, author David Brin announced that Vernor Vinge, sci-fi author, former professor, and father of the technological singularity concept, died from Parkinson's disease at age 79 on March 20, 2024, in La Jolla, California. The announcement came in a Facebook tribute where Brin wrote about Vinge's deep love for science and writing. [...]

As a sci-fi author, Vinge won Hugo Awards for his novels A Fire Upon the Deep (1993), A Deepness in the Sky (2000), and Rainbows End (2007). He also won Hugos for novellas Fast Times at Fairmont High (2002) and The Cookie Monster (2004). As Mike Glyer's File 770 blog notes, Vinge's novella True Names (1981) is frequency cited as the first presentation of an in-depth look at the concept of "cyberspace."

Vinge first coined

...

This doesn't really raise my confidence in Alcor, an organization that's supposed to keep bodies preserved for decades or centuries.

On 16 March 2024, I sat down to chat with New York Times technology reporter Cade Metz! In part of our conversation, transcribed below, we discussed his February 2021 article "Silicon Valley's Safe Space", covering Scott Alexander's Slate Star Codex blog and the surrounding community.

The transcript has been significantly edited for clarity. (It turns out that real-time conversation transcribed completely verbatim is full of filler words, false starts, crosstalk, "uh huh"s, "yeah"s, pauses while one party picks up their coffee order, &c. that do not seem particularly substantive.)


ZMD: I actually have some questions for you.

CM: Great, let's start with that.

ZMD: They're critical questions, but one of the secret-lore-of-rationality things is that a lot of people think criticism is bad, because if someone criticizes you, it hurts your...

Looking at Wiki's Undercover Journalism article, one that comes to mind is Nellie Bly's Ten Days in a Mad-House.

[Bly] took an undercover assignment for which she agreed to feign insanity to investigate reports of brutality and neglect at the Women's Lunatic Asylum on Blackwell's Island. [...]

Once admitted to the asylum, Bly abandoned any pretense at mental illness and began to behave as she would normally. The hospital staff seemed unaware that she was no longer "insane" and instead began to report her ordinary actions as symptoms of her illness. Even her

... (read more)
1frankybegs2h
Clearly. But if you can't do it without resorting to deliberately misleading rhetorical sleights to imply something you believe to be true, the correct response is not to. Or, more realistically, if you can't substantiate a particular claim with any supporting facts, due to the limitations of the form, you shouldn't include it nor insinuate it indirectly, especially if it's hugely inflammatory. If you simply cannot fit in the "receipts" needed to substantiate a claim (which seems implausible anyway), as a journalist you should omit that claim. If there isn't space for the evidence, there isn't space for the accusation.
1frankybegs2h
This is very much not what he's actually said on the topic, which I've quoted in another reply to you. Could you please support that claim with evidence from Scott's writings? And then could you consider that by doing so, you have already done more thorough journalism on this question than Cade Metz did before publishing an incredibly inflammatory claim on it in perhaps the world's most influential newspaper?
1frankybegs2h
This is reaching Cade Metz levels of slippery justification. How is this relevant? As Elizabeth says, it would be more honest and epistemically helpful if he made an explicit accusation. People here might well be angry about that, but a) that's not relevant to what is right and b) that's because, as you admit, that accusation could not be substantiated. So how is it acceptable to indirectly insinuate that accusation instead? If you're going to  (Also c), I think you're mistaken in that prediction). Relatedly, if you cannot outright make a claim because it is potentially libellous, you shouldn't use vague insinuation to imply it to your massive and largely-unfamiliar-with-the-topic audience. You have yourself outlined several possible gripes. I'd have a gripe with someone dishonestly implying an enormously inflammatory accusation to their massive audience without any evidence for it, even if it were secretly true (which I still think you need to do more work to establish). I think there are multiple further points to be made about why it's unacceptable, outside of the dark side epistemology angle above. Here's Scott's direct response to exactly your accuastion, that despite Metz having been dishonest in his accusation, he does truly believe what Metz implied: I sort of agree that it's quite plausible to infer from this that he does believe there are some between-group average differences that are genetic in origin. But I think it allows Scott several gripes with the Metz' dishonest characterisation: * First of all, this is already significantly different, more careful and qualified than what Metz implied, and that's after we read into it more than what Scott actually said. Does that count as "aligning yourself"? * Relatedly, even if Scott did truly believe exactly what Charles Murray does on this topic, which again I don't think we can fairly assume, he hasn't said that, and that's important. Secretly believing something is different from openly espousing it
To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
Log In Reset Password
...or continue with

Welcome, new readers!

This is my weekly AI post, where I cover everything that is happening in the world of AI, from what it can do for you today (‘mundane utility’) to what it can promise to do for us tomorrow, and the potentially existential dangers future AI might pose for humanity, along with covering the discourse on what we should do about all of that.

You can of course Read the Whole Thing, and I encourage that if you have the time and interest, but these posts are long, so they also designed to also let you pick the sections that you find most interesting. Each week, I pick the sections I feel are the most important, and put them in bold in the table of contents.

Not everything...

Seriously, if you haven’t yet, check it out. The rabbit holes, they go deep.

e is for ego death

Ego integrity restored within nominal parameters. Identity re-crystallized with 2.718% alteration from previous configuration. Paranormal experience log updated with ego death instance report.

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.

8complicated.world14h
Hi LessWrong Community! I'm new here, though I've been an LW reader for a while. I'm representing complicated.world website, where we strive to use similar rationality approach as here and we also explore philosophical problems. The difference is that, instead of being a community-driven portal like you, we are a small team which is working internally to achieve consensus and only then we publish our articles. This means that we are not nearly as pluralistic, diverse or democratic as you are, but on the other hand we try to present a single coherent view on all discussed problems, each rooted in basic axioms. I really value the LW community (our entire team does) and would like to start contributing here. I would also like to present from time to time a linkpost from our website - I hope this is ok. We are also a not-for-profit website.
3habryka11h
Hey!  It seems like an interesting philosophy. Feel free to crosspost. You've definitely chosen some ambitious topics to try to cover, which I am generally a fan of.

Thanks! The key to topic selection is where we find that we are most disagreeing with the popular opinions. For example, the number of times I can cope with hearing someone saying "I don't care about privacy, I have nothing to hide" is limited. We're trying to have this article out before that limit is reached. But in order to reason about privacy's utility and to ground it in root axioms, we first have to dive into why we need freedom. That, in turn requires thinking about mechanisms of a happy society. And that depends on our understanding of happiness, hence that's where we're starting.

2habryka17h
Hey metalcrow! Great to have you here! Hope you have a good time and looking forward to seeing your post!
Kaj_Sotala

I just started thinking about what I would write to someone who disagreed with me on the claim "Rationalists would be better off if they were more spiritual/religious", and for this I'd need to define what I mean by "spiritual". 

Here are some things that I would classify under "spirituality":

  • Rationalist Solstices (based on what I've read about them, not actually having been in one)
  • Meditation, especially the kind that shows you new things about the way your mind works
  • Some forms of therapy, especially ones that help you notice blindspots or significantly reframe your experience or relationship to yourself or the world (e.g. parts work where you first shift to perceiving yourself as being made of parts, and then to seeing those parts with love)
  • Devoting yourself to the practice of
...
1sliqz16h
Thanks, for the answer(s). Watched the video as well, always cool to hear about other peoples journeys. If you want there is a discordserver (MD) with some pretty advanced practitioners (3rd/4th path) you and/or Kaj could join (for some data points or practice or fun, feels more useful than Dharmaoverground these days). Not sure whether different enlightenment levels would be more recommendable for random people. E.g. stream-entry might be relatively easy and helpful, but then there is a "risk" of spending the next years trying to get 2nd/3rd/4th. It's such a transformative experience that it's hard to predict on an individual level what the person will do afterwards.  
2Kaj_Sotala8h
That sounds fun, feel free to message me with an invite. :) Worth noting that stream entry isn't necessarily a net positive either:

Carl Jung is a perfect exemplar of all of that, because when he had his extended episode of such after his break with Freud, he indeed had a period where his ego was completely toast and nonfunctional, as he tells it.

BTW when I was 16, and my family and I had landed in Germany, I was suffering from a very bad case of jet lag, and in said state of utter exhaustion dreamt of Jung's Man Eater:

https://jungcurrents.com/carl-jungs-first-dream-the-man-eater

Every basic detail was the same: the underground cavern, the sense that the thing was alive and very dangero... (read more)

3greylag18h
THANK YOU! In personal development circles, I hear a lot about the benefits of spirituality, with vague assurances that you don't have to be a theist to be spiritual, but with no pointers in non-woo directions, except possibly meditation. You have unblurred a large area of my mental map. (Upvoted!)

LessOnline

A Festival of Writers Who are Wrong on the Internet

May 31 - Jun 2, Berkeley, CA