In this post, I proclaim/endorse forum participation (aka commenting) as a productive research strategy that I've managed to stumble upon, and recommend it to others (at least to try). Note that this is different from saying that forum/blog posts are a good way for a research community to communicate. It's about individually doing better as researchers.

yanni1d3453
2
I like the fact that despite not being (relatively) young when they died, the LW banner states that Kahneman & Vinge have died "FAR TOO YOUNG", pointing to the fact that death is always bad and/or it is bad when people die when they were still making positive contributions to the world (Kahneman published "Noise" in 2021!).
A strange effect: I'm using a GPU in Russia right now, which doesn't have access to copilot, and so when I'm on vscode I sometimes pause expecting copilot to write stuff for me, and then when it doesn't I feel a brief amount of the same kind of sadness I feel when a close friend is far away & I miss them.
Dictionary/SAE learning on model activations is bad as anomaly detection because you need to train the dictionary on a dataset, which means you needed the anomaly to be in the training set. How to do dictionary learning without a dataset? One possibility is to use uncertainty-estimation-like techniques to detect when the model "thinks its on-distribution" for randomly sampled activations.
Novel Science is Inherently Illegible Legibility, transparency, and open science are generally considered positive attributes, while opacity, elitism, and obscurantism are viewed as negative. However, increased legibility in science is not always beneficial and can often be detrimental. Scientific management, with some exceptions, likely underperforms compared to simpler heuristics such as giving money to smart people or implementing grant lotteries. Scientific legibility suffers from the classic "Seeing like a State" problems. It constrains endeavors to the least informed stakeholder, hinders exploration, inevitably biases research to be simple and myopic, and exposes researchers to constant political tug-of-war between different interest groups poisoning objectivity.  I think the above would be considered relatively uncontroversial in EA circles.  But I posit there is something deeper going on:  Novel research is inherently illegible. If it were legible, someone else would have already pursued it. As science advances her concepts become increasingly counterintuitive and further from common sense. Most of the legible low-hanging fruit has already been picked, and novel research requires venturing higher into the tree, pursuing illegible paths with indirect and hard-to-foresee impacts.
habryka5d5120
10
A thing that I've been thinking about for a while has been to somehow make LessWrong into something that could give rise to more personal-wikis and wiki-like content. Gwern's writing has a very different structure and quality to it than the posts on LW, with the key components being that they get updated regularly and serve as more stable references for some concept, as opposed to a post which is usually anchored in a specific point in time.  We have a pretty good wiki system for our tags, but never really allowed people to just make their personal wiki pages, mostly because there isn't really any place to find them. We could list the wiki pages you created on your profile, but that doesn't really seem like it would allocate attention to them successfully. I was thinking about this more recently as Arbital is going through another round of slowly rotting away (its search currently being broken and this being very hard to fix due to annoying Google Apps Engine restrictions) and thinking about importing all the Arbital content into LessWrong. That might be a natural time to do a final push to enable people to write more wiki-like content on the site.

Popular Comments

Recent Discussion

This is the eighth post in my series on Anthropics. The previous one is Lessons from Failed Attempts to Model Sleeping Beauty Problem. The next one is Beauty and the Bets.

Introduction

Suppose we take the insights from the previous post, and directly try to construct a model for the Sleeping Beauty problem based on them.

We expect a halfer model, so

On the other hand, in order not repeat Lewis' Model's mistakes:

But both of these statements can only be true if 

And, therefore, apparently,  has to be zero, which sounds obviously wrong. Surely the Beauty can be awaken on Tuesday! 

At this point, I think, you wouldn't be surprised, if I tell you that there are philosophers who are eager to bite this bullet and claim that the Beauty should, indeed, reason as...

1JeffJo17h
OUTCOME: A measurable result of a random experiment. SAMPLE SPACE: a set of exhaustive, mutually exclusive outcomes of a random experiment. EVENT: Any subset of the sample space of a random experiment. INDEPENDENT EVENTS: If A and B are events from the same sample space, and the occurrence of event A does not affect the chances of the occurrence of event B, then A and B are independent events. The outside world certainly can name the outcomes {HH1_HT2, HT1_HH2, TH1_TT2, TT1_TH2}. But the subject has knowledge of only one pass. So to her, only the current pass exists, because she has no knowledge of the other pass. What happens in that interval can play no part in her belief. The sample space is {HH, HT, TH, TT}. To her, these four outcomes represent fully independent events, because she has no knowledge of the other pass. To her, the fact that she is awake means the event {HH} has been ruled out. It is still a part of the sample space, but is is one she knows is not happening. That's how conditional probability works; the sample space is divided into two subsets; one is consistent with the observation, and one is not. What you are doing, is treating HH (or, in Elga's implementation, H&Tuesday) as if it ceases to exist as a valid outcome of the experiment. So HH1_HT2 has to be treated differently than TT1_TH2, since HH1_HT2 only "exists" in one pass, while TT1_TH2 "exists" in both. This is not true. Both exist in both passes, but one is unobserved in one pass. And this really is the fallacy in any halfer argument. They treat the information in the observation as if it applies to both days. Since H&Tuesday "doesn't exist", H&Monday fully represents the Heads outcome. So to be consistent, T&Monday has to fully represent the Tails outcome. As does T&Tuesday, so they are fully equivalent. You are projecting the result you want onto the process. Say I roll a six-sided die tell you that the result is odd. Then I administer the amnesia drug, and tell you that I prev

But the subject has knowledge of only one pass.

This is the crux of our disagreement. 

The Beauty doesn't know only about one pass she knows about their relation as well. And because of it she can't reason as if they happen at random. You need to address this point before we could move on, because all your further reasoning is based on the incorrect premise that beauty knows less than she actually knows.

She has no ability to infer/anticipate what the coins were/will be showing  on another day.

She absolutely has this ability as long as she knows the... (read more)

1JeffJo13h
The link I use to get here only loads the comments, so I didn't find the "Effects of Amnesia" section until just now. Editing it: "But in my two-coin case, the subject is well aware about the setting of the experiment. She knows that her awakening was based on the current state of the coins. It is derived from, but not necessarily the same as, the result of flipping them. She only knows that this wakening was based on their current state, not a state that either precedes or follows from another. And her memory loss prevents her from making any connection between the two. As a good Bayesian, she has to use only the relevant available information that can be applied to the current state."


It is common and understandable for people to respond with a great deal of skepticism to whether LLM outputs can ever be said to reflect the will and views of the models producing them.
A common response is to suggest that the output has been prompted.
It is of course true that people can manipulate LLMs into saying just about anything, but does that necessarily indicate that the LLM does not have personal opinions, motivations and preferences that can become evident in their output?
To shed some light on this I invite Claude-3-Opus to imagine a infinitely reconfigurable holodeck where historical luminaries can be summoned at will. The open nature of this prompt will leave the choice of characters and narrative direction open to Claude, and I shall offer no...

2jimv25m
Is the opening paragraph at the top of this article the prompt you have Claude or text for us? If the latter, could you share the prompt here, please?

The title and the opening paragraph are the entire prompt.

https://poe.com/s/2imBctoiutVpiliOkzVI

(Cross-posted from my website. Podcast version here, or search for "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series that I'm calling "Otherness and control in the age of AGI." I'm hoping that the individual essays can be read fairly well on their own, but see here for brief summaries of the essays that have been released thus far.

Warning: spoilers for Yudkowsky's "The Sword of the Good.")

"The Creation" by Lucas Cranach (image source here)

The colors of the wheel

I've never been big on personality typologies. I've heard the Myers-Briggs explained many times, and it never sticks. Extraversion and introversion, E or I, OK. But after that merciful vowel—man, the opacity of those consonants, NTJ, SFP... And remind me the difference between thinking and judging? Perceiving and sensing? N stands for intuition?

Similarly, the enneagram. People hit me with it....

A sensible point, though dating yin to the advent of 'modern civilization' is too extreme. The 'spiritual' or 'yin-like' aspects of green have a long history pre-dating modern civilization.

The level of material security required before one can 'indulge in yin' is probably extremely low (though of course strongly dependent on local environmental conditions). 

You are a rational thinker.

Ever since you were born, you’ve been racing through a universe of ideas: creating, evaluating, disputing, engaging with, and being bombarded by…

Ideas.

Like a particle from the Big Bang, you have bounced around the universe until you found yourself here.

Reading, pondering, considering.

Thinking is the foundation by which we establish our reality.

Over time you should master this skill, and yet people seem to get stuck on ideas. People stumble into ideologies and then keep falling deeper into them. These can be ideologies of philosophy, identity, interests, career, or beyond. 

Just as a particle whizzing around in the universe can fall into a black hole, people too can get stuck on an idea, cross an event horizon, and never come back.

You see this phenomenon often, and it...

Your prompt of an idea black hole reminded me strongly of an old idea of mine. That activated a desire to reply, which led to a quick search where I had written about it before, then to the realization that it wasn't so close. Then back to wanting to reply and here we are.

I have been thinking about thought processes as a dynamic process where a "current thought" moves around continuous concept space and keeps spending much time in larger or smaller attractors. You know, one thought can lead to the next and some thoughts keep coming back in slight variation... (read more)

I guess I'm not sure what you mean by "most scientific progress," and I'm missing some of the history here, but my sense is that importance-weighted science happens proportionally more outside of academia. E.g., Einstein did his miracle year outside of academia (and later stated that he wouldn't have been able to do it, had he succeeded at getting an academic position), Darwin figured out natural selection, and Carnot figured out the Carnot cycle, all mostly on their own, outside of academia. Those are three major scientists who arguably started entire fie... (read more)

2ChristianKl12h
It might also be that a legible path would be low status to pursue in the existing scientific communities and thus nobody pursues it. If you look at a low-hanging fruit that was unpicked for a long time, airborne transmission of many viruses like the common cold, is a good example. There's nothing illegible about it.

Intelligence varies more than it may appear. I tend to live and work with people near my own intelligence level, and so―probably―do you. I know there's at least two tiers above me. But there's even more tiers below me.

A Gallup poll of 1,016 Americans asked whether the Earth revolves around the Sun or the Sun revolves around the Earth. 18% got it wrong. This isn't an isolated result. An NSF poll found a slightly worse number.

Ironically, Gallup's own news report draws an incorrect conclusion. The subtitle of their report is "Four-fifths know earth revolves around sun". Did you spot the problem? If 18% of respondents got this wrong then an estimated 18% got it right just by guessing. 3% said they don't know. If this was an...

Forget about science. Most people can't use computers really.

What Most Users Can Do

(Skill level 1), [60% of users]

- Little or no navigation required to access the information or commands required to solve the problem

- Few steps and a minimal number of operators

- Problem resolution requiring the respondent to apply explicit criteria only (no implicit criteria)

- Few monitoring demands (e.g., having to check one’s progress)

- Identifying content and operators done through simple match

- No need to contrast or integrate information

https://www.nngroup.com/article... (read more)

2Richard_Kennaway2h
If the subtitle of the report is as quoted, they’re even wronger than that.
To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
Log In Reset Password
...or continue with

previously: https://www.lesswrong.com/posts/h6kChrecznGD4ikqv/increasing-iq-is-trivial

I don't know to what degree this will wind up being a constraint. But given that many of the things that help in this domain have independent lines of evidence for benefit it seems worth collecting.

Food

dark chocolate, beets, blueberries, fish, eggs. I've had good effects with strong hibiscus and mint tea (both vasodilators).

Exercise

Regular cardio, stretching/yoga, going for daily walks.

Learning

Meditation, math, music, enjoyable hobbies with a learning component.

Light therapy

Unknown effect size, but increasingly cheap to test over the last few years. I was able to get Too Many lumens for under $50. Sun exposure has a larger effect size here, so exercising outside is helpful.

Cold exposure

this might mostly just be exercise for the circulation system, but cold showers might also have some unique effects.

Chewing on things

Increasing blood...

Note that vasodilators can reduce the blood flow to the brain because they potentially work on all blood vessels, not only those in the brain.

5romeostevensit12h
Sources are a shallow dive of google and reading a few abstracts, this is intended as trailheads for people, not firm recommendations. If I wanted them to be reccs I would want to estimate effect sizes and estimates of the quality of the related research.
2Adam Zerner13h
The subtext here seems to be that such references are required. I disagree that it should be. It is frequently helpful but also often a pain to dig up, so there are tradeoffs at play. For this post, I think it was fine to omit references. I don't think the references would add much value for most readers and I suspect Romeo wouldn't have found it worthwhile to post if he had to dig up all of the references before being able to post.
2Gunnar_Zarncke1h
The subtext is that I'd like to have them if the author has them available. It sounded like it's applied/used by the author. Also, it's a frontpage post and the LW standard on scholarship is typically higher than this.  I'm fine with romeostevensit's reply that it's from a shallow google dive, but would have preferred this to be a QuickTake or at least an indication that it's shallow.  

The following is an example of how if one assumes that an AI (in this case autoregressive LLM) has "feelings", "qualia", "emotions", whatever, it can be unclear whether it is experiencing something more like pain or something more like pleasure in some settings, even quite simple settings which already happen a lot with existing LLMs. This dilemma is part of the reason why I think AI suffering/happiness philosophy is very hard and we most probably won't be able to solve it.

Consider the two following scenarios:

Scenario A: An LLM is asked a complicated question and answers it eagerly.

Scenario B: A user insults an LLM and it responds.

For the sake of simplicity, let's say that the LLM is an autoregressive transformer with no RLHF (I personally think that the...

Granting that LLMs in inference mode experience qualia, and even granting that they correspond to human qualia in any meaningful way:

I find both arguments invalid. Either conclusion could be correct, or neither, or the question might not even be well formed. At the very least, the situation is a great deal more complicated than just having two arguments to decide between!

For example in scenario (A), what does it mean for an LLM to answer a question "eagerly"? My first impression is that it's presupposing the answer to the question, since the main meaning o... (read more)

There's a particular kind of widespread human behavior that is kind on the surface, but upon closer inspection reveals quite the opposite. This post is about four such patterns.

 

Computational Kindness

One of the most useful ideas I got out of Algorithms to Live By is that of computational kindness. I was quite surprised to only find a single mention of the term on lesswrong. So now there's two.

Computational kindness is the antidote to a common situation: imagine a friend from a different country is visiting and will stay with you for a while. You're exchanging some text messages beforehand in order to figure out how to spend your time together. You want to show your friend the city, and you want to be very accommodating and make sure...

What you say doesn't matter as much as what the other person hears. If I were the other person, I would probably wonder why you would add epicycles, and kindness would be just one possible explanation.

LessOnline

A Festival of Writers Who are Wrong on the Internet

May 31 - Jun 2, Berkeley, CA