Recent Discussion

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version here (may not be up yet).

Highlights

Self-training with Noisy Student improves ImageNet classification (Qizhe Xie et al) (summarized by Dan H): Instead of summarizing this paper, I'll provide an opinion describing the implications of this and other recent papers.

Dan H's opinion: Some in the safety community have speculate... (Read more)

1SoerenMind9hPotential paper from DM/Stanford for a future newsletter: https://arxiv.org/pdf/1911.00459.pdf [https://arxiv.org/pdf/1911.00459.pdf] It addresses the problem that an RL agent will delude itself by finding loopholes in a learned reward function.

In coherence arguments do not imply goal directed behavior Rohin Shah argues that a system's merely being at all model-able as an EU maximizer does not imply that it has "goal directed behavior". The argument as I understand it runs something like this:

1: Any behavior whatsoever maximizes some utility function.

2: Not all behaviors are goal directed.

Conclusion: A system's behavior maximizing some utility function does not imply that its behavior is goal directed.

I think this argument is technically sound, but misses an important connection between VNM coherence and goal dir... (Read more)

I think this formulation of goal-directedness is pretty similar to one I suggested in the post before the coherence arguments post (Intuitions about goal-directed behavior, section "Our understanding of the behavior"). I do think this is an important concept to explain our conception of goal-directedness, but I don't think it can be used as an argument for AI risk, because it proves too much. For example, for many people without technical expertise, the best model they have for a laptop is that it is pursuing some goal (at least, many of my ... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post

1Daniel Kokotajlo12hThis theory of goal-directedness has the virtue of being closely tied to what we care about: --If a system is goal-directed according to this definition, then (probably) it is the sort of thing that might behave as if it has convergent instrumental goals. It might, for example, deceive us and then turn on us later. Whereas if a system is not goal-directed according to this definition, then absent further information we have no reason to expect those behaviors. --Obviously we want to model things efficiently. So we are independently interested in what the most efficient way to model something is. So this definition doesn't make us go that far out of our way to compute, so to speak. On the other hand, I think this definition is not completely satisfying, because it doesn't help much with the most important questions: --Given a proposal for an AGI architecture, is it the sort of thing that might deceive us and then turn on us later? Your definition answers: "Well, is it the sort of thing that can be most efficiently modelled as an EU-maximizer? If yes, then yes, if no, then no." The problem with this answer is that trying to see whether or not we can model the system as an EU-maximizer involves calculating out the system's behavior and comparing it to what an EU-maximizer (worse, to a range of EU-maximizers with various relatively simple or salient utility and credence functions) would do, and if we are doing that we can probably just answer the will-it-deceive-us question directly. Alternatively perhaps we could look at the structure of the system--the architecture--and say "see, this here is similar to the EU-max algorithm." But if we are doing that, then again, maybe we don't need this extra step in the middle; maybe we can jump straight from looking at the structure of the system to inferring whether or not it will act like it has convergent instrumental goals.
[Link]Drowning children are rare
156mo5 min readShow Highlight

Stories such as Peter Singer's "drowning child" hypothetical frequently imply that there is a major funding gap for health interventions in poor countries, such that there is a moral imperative for people in rich-countries to give a large portion of their income to charity. There are simply not enough excess deaths for these claims to be plausible.

Much of this is a restatement of part of my series on GiveWell and the problem of partial funding, so if you read that carefully and in detail, this may not be new to you, but it's important enough to have its own concise post. Th... (Read more)

It was originally marketed as a health tonic, but its apparent curative properties were due to the powerful stimulant and analgesic cocaine, not any health-enhancing ingredients. Later the cocaine was taken out (but the “Coca” in the name retained), so now it fools the subconscious into thinking it’s healthful with - on different timescales - mass media advertising, caffeine, and refined sugar.

It’s less overtly a scam now, in large part because it has the endowment necessary to manipulate impressions more subtly at scale.

How To Change a Dance
118d2 min readShow Highlight

Let's say you don't like something about your local dance. Perhaps you'd like to see gender free calling, a different approach to booking bands, a new kind of special event, or something else. How can you make this happen?

The best case is that you talk to the organizers, they say "what a great idea!" and handle the rest, but it rarely works this way. [1] Maybe the organizers have a different background and don't understand why you think your ideas would be an improvement. Maybe your ideas would be more work, at least at first, and they're feeling overworked with what they're doing a... (Read more)

If this is a post about strategy then strategy can be discussed. It's not a vanity post from my perspective, but even if it is I'm not married to authorial intent.

As for any group being too small to infiltrate for gain, that hasn't been my experience. It only takes 3 members for entryism to occur, as only one needs to defect from the established order. You see this in cases of adultery within a social group all the time. Lots of people lose their partner and their 'best friend' at the same time.

Affordance Widths
1472y2 min readShow Highlight

This article was originally a post on my tumblr. I'm in the process of moving most of these kinds of thoughts and discussions here.


Okay. There’s a social interaction concept that I’ve tried to convey multiple times in multiple conversations, so I’m going to just go ahead and make a graph.

I’m calling this concept “Affordance Widths”.

Let’s say there’s some behavior {B} that people can do more of, or less of. And everyone agrees that if you don’t do enough of the behavior, bad thing {X} happens; but if you do too much of the behavior, bad thing {Y} happens.

Now, let’s say we have five differ... (Read more)

Since others have done a contextualized review, I'll aim to do a decoupled review, with a caveat that I think the contextual elements are important for consideration with inclusion into the compendium.

Okay. There’s a social interaction concept that I’ve tried to convey multiple times in multiple conversations, so I’m going to just go ahead and make a graph.
I’m calling this concept “Affordance Widths”.

I'd like to see a clear definition here before launching into an example. In fact, there's no clear ... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post

7leggi18hCaught by comments ... I've had a read of this post. It seems rather whiny. I'm struggling to see the value. edited to add - Imagine if this was an English compression test and the question was "with which character does the author most identify with"

One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy. There must be some other strategy that never does any worse than your strategy, but does strictly better than your strategy with certainty in at least one situation. There’s a good explanation of these arguments here.

We shouldn’t expect mere humans to be able to notice any failures of coherence in a superintelligent agent,... (Read more)

2rohinmshah1hI pretty strongly agree with this review (and jtbc it was written without any input from me, even though Daniel and I are both at CHAI). Yeah, maybe I should say "coherence theorems" to be clearer about this? (Like, it isn't a theorem that I shouldn't give you limitless number of dollars in return for nothing; maybe I think that you are more capable than me and fully aligned with me, and so you'd do a better job with my money. Or maybe I value your happiness, and the best way to purchase it is to give you money no strings attached.) Fwiw, I do in fact worry about goal-directedness, but (I think) I know what you mean. (For others, I think Daniel is referring to something like "the MIRI camp", though that is also not an accurate pointer, and it is true that I am outside that camp.) My responses to the questions: 1. The ones in Will humans build goal-directed agents? [https://www.lesswrong.com/posts/9zpT9dikrrebdq3Jf/will-humans-build-goal-directed-agents] , but if you want arguments that aren't about humans, then I don't know. 2. Depends on the distribution over utility functions, the action space, etc, but e.g. if it uniformly selects a numeric reward value for each possible trajectory (state-action sequence) where the actions are low-level (e.g. human muscle control), astronomically low. 3. That will probably be a good model for some (many?) powerful AI systems that humans build. 4. I don't know. (I think it depends quite strongly on the way in which we train powerful AI systems.) 5. Not likely at low levels of intelligence, plausible at higher levels of intelligence, but really the question is not specified enough.

it was written without any input from me

Well, I didn't consult you in the process of writing the review, but we've had many conversations on the topic which presumably have influenced how I think about the topic and what I ended up writing in the review.

2DanielFilan1hSorry, I meant theorems taking 'no limitless dollar sink' as an axiom and deriving something interesting from that.
8DanielFilan5hPutting my cards on the table, this is my guess at the answers to the questions that I raise: 1. I don't know. 2. Low. 3. Frequent if it's an 'intelligent' one. 4. Relatively. You probably don't end up with systems that resist literally all changes to their goals, but you probably do end up with systems that resist most changes to their goals, barring specific effort to prevent that. 5. Probably. That being said, I think that a better definition of 'goal-directedness' would go a long way in making me less confused by the topic.
On the Chatham House Rule
581y4 min readShow Highlight

I have gone to several events operating under the Chatham House Rule, and have overall found it more annoying than useful. In this post, I share why I dislike the rule, how I think it can be improved, and hopefully spark others to give ideas on how to improve it. In particular, I think the part about not revealing who was at the event should be opt in. Partially, my goal is to eventually develop a modified version that might be used at future events.

The Chatham House Rule states that "When a meeting, or part thereof, is held under the Chatham House Rule, participants are free to use the i... (Read more)

I see a simple fix.

Rather than trying to go full "Chatham House", make your own rules that work for the group (that's what Chatham House did), call it what you like.

Define those rules and make sure everyone understands. (a 'traffic light' system for levels of security at different meetings if necessary?)

Rules should be sensible, simple and clear - and for everyone. (Opt-in opt-out - who can remember who's agreed to what?).


How do I respond if I met them at a Chatham House Rule event?

Vaguely. Move the conversation swif... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post

Antimemes
2112d1 min readShow Highlight

Antimemes are self-keeping secrets. You can only perceive an antimeme if you already know it's there. Antimemes don't need a conspiracy to stay hidden because you can't comprehend an antimeme just by being told it exists. You can shout them to the heavens and nobody will listen. I'll try to explain with a fictitious example.

Suppose we all had an invisible organ behind our ears and our brains kept it secret from our consciousness. If I told you "you have an invisible organ behind your ear" you wouldn't believe me. You'd only believe it exists if you deduced its existence from a trail of evidenc

... (Read more)
1CuriousMeta9hI'd love to read more on the topic. A longer list of what LW folk consider to be antimemes would be pretty interesting, too. I like to think I gained some insight from the mention of Lisp and entrepreneurship.

Any sane person could write up a list of antimemes, but no sane person would post it.

It would tend to have the effect of making most people give up on the idea of antimeme, concluding that it's something that only insane people think about.

1CuriousMeta10hI'd propose as examples "most stuff taught at university". Even outside of teaching institutions, complex ideas commonly spread memetically if the incentives for acquiring them are sufficiently visible from the outset. Think Evolutionary Theory, Object-Oriented Programming, or Quantum Physics.

Theorem: Fuzzy beliefs (as in https://www.alignmentforum.org/posts/Ajcq9xWi2fmgn8RBJ/the-credit-assignment-problem#X6fFvAHkxCPmQYB6v ) form a continuous DCPO. (At least I'm pretty sure this is true. I've only given proof sketches so far)

The relevant definitions:

A fuzzy belief over a set is a concave function such that (where is the space of probability distributions on ). Fuzzy beliefs are partially ordered by ... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post

(cross posted on my roam blog)

There's an old Paul Graham Essay, "Keep Your Identity Small". It's short so it's worth it to read the whole thing right now if you've never seen it. The yisbifiefyb ("yeah it's short but i'm functionally illiterate except for your blog") is roughly "When something becomes part of your identity, you become dumber. Don't make things part of your identity."

I read that post some time in high school and thought, "Of course! You're so right Paul Graham. Cool, now I'll never identify as anyth... (Read more)

1Nicholas Garcia2hI find this very true. In fact, portraying a STRONGER identity often is met more easily results in better responses. The trick is that you can be strategic about it. By selecting between "personas" or "roles" you can select what kind of responses you want to get. I find it helpful to think about the different situations I am in (work meetings, studying in cafes, meeting friends, etc.), and then think about "what is the most ideal response I could get" - and think about "what kind of person / action would provoke that kind of response?" Then, for the given situation make sure that everything is coherent - appearance, energy level, behaviors, speech cadence, etc. Coherence is very powerful. We already do this when we have a "work self" and a "home self". But for most of our activities it is not pre-planned. We just want to be "ourselves" - i.e. not have to strategically prepare for each situation. As for "social identity theory" and feeling attacked, I don't think KYIS quite applies. When you are part of a tribe or subculture or whatever, there are several factors at play. (1) Defending the tribe may gain you status in the tribe. (2) Allowing attacks on fellow tribe-members to go unprovoked may put you personally at risk as well - thus the tribe makes it a value to protect fellow tribe-members. KYIS may mean "don't join any tribes". Or more realistically - only feel kinship or trust toward those you personally know, not any abstract larger categories of people. Some would argue that this is how China used to work [https://en.wikipedia.org/wiki/From_the_Soil]. However, as societies scale up in size, we typically do join social groups with abstract myths that bind people together, provide standards, and allow coordination among strangers. Anyway, I guess it gets pretty complex as you unpack it. I suppose if you have skills that are in demand by many people, you do not need to be "married" to any one tribe, nation, or company. You can flit from one to the next if
1MakoYass9hI was really hoping you were going to provide an actionable version of "keep your tribal identity small" For me, the most useful parts of the KYIS outlook were, meeting people with a fresh slate without saying "yes I'm one of those people", not feeling like you personally are being threatened when people criticise your group, not feeling that impulse to delude yourself and everyone around you into thinking the outgroup are monsters. The issue is, I notice that we can only stay in this state of neutrality for so long. Eventually, we find our tribe, we develop an ideology (cluster of beliefs about how the world works and how to do good) that is simply too useful to step outside of, we become publicly associated with controversial projects. That will happen. If we don't learn how to move soundly in that fire we wont end up moving soundly at all.

I guess the actionable version is to develop transferable skills, abilities, wealth, or social capital that are highly valued by many different tribes.

Then you have the leverage to flit from one to the next, and not care about standing up for any particular tribe.

However, the game to acquire wealth, social capital, and valued skills is basically the game that we are all playing and has lots of competition. The only way to "opt out" is to join a local monopoly (i.e. a tribe). Also, in the real world, tribes often "loan us resources" to ... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post

7Hazard15hLol, this is the post I wanted to write but better. Thanks Kaj! To anyone who ended up here, go read Ruby's post.

There’s an essay that periodically feels deeply relevant to a situation:

Someday I want to write a self-help book titled “F*k The Karate Kid: Why Life is So Much Harder Than We Think”.

Look at any movie with a training montage: The main character is very bad at something, then there is a sequence in the middle of the film set to upbeat music that shows him practicing. When it's done, he's an expert.

It seems so obvious that it actually feels insulting to point it out. But it's not obvious. Every adult I know--or at least the ones who are depressed--continually suffers from something like sticker

... (Read more)

This is an interesting comment. Some thoughts after reflecting a bit:

Awhile ago you wrote a comment saying something like "deliberate practice deliberate practice until you get really good identifying good feedback loops, and working with them." I found that fairly inspiring at the time. 

I didn't ever really dedicate myself to doing that thoroughly enough to have a clear opinion on whether it's The Thing. I think I put maybe... 6 hours into deliberate practice, with intent to pay attention to how deliberate practice generalized. I got value out of tha

... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post
7mr-hire5hThis is (sort of) true but I think you're overstating it. For instance, TIm Ferris' whole thing is about breaking down skills into functional causal models, and he certainly does a good job of becoming proficient at them, but you'll notice he never becomes EXPERT at them. Similarly, Josh Waitzkin also wrote a whole book about learning and breaking down skills into causal models, but still wanted at least 5 YEARS to train BJJ 24/7 before being comfortable going to the world finals (he never actually ended up going to the world finals so we'll never be sure if this was enough time). Your example below is a solo skill, but I wager if it was a competitive skill you'd find that while you outpace your peer newbies, it still feels quite slow to catch up to veterans. I suspect something similar with disagreements. Someone skilled at disagreements can take a deep disagreement that would have taken 10 years and turn it into 5 years. But that's still 5 years.
2romeostevensit6hLKY apparently regularly destroyed debate opponents in 3 different languages in long marathon sessions.
1TAG18h"Well, I've never seen one". There is still better and worse epistemology. I have been arguing that cautious epistemology is better than confident epistemology. Even if there is a set of claims that are very silly, it doesn't follow that Aumann-style agreement is possible.

Without using eminent domain, a large chunk of the possible future value goes to surrounding land-owners who may have done little or nothing to create that value. It does not seem economically possible to build a city that is cheap to live in without locking the price of land down in some way, at some point. It is not obvious how to do this well, but eminent domain seems to be a necessary component of it. If even fairly rural land starts out pre-speculated, there is no hope, there is a value/livability value that cities cannot ever rise above.

Apparently, the Charter Cities Institute for all th... (Read more)

It does not seem economically possible to build a city that is cheap to live in without locking the price of land down in some way, at some point. It is not obvious how to do this well, but eminent domain seems to be a necessary component of it.

From their website:

Building the Future of Governance
for the Cities of Tomorrow

They're interested in better solutions. "Impossible" just means "it's never been done".

Elevator pitch: Bring enough light to simulate daylight into your home and office.

This idea has been shared in Less Wrong circles for a couple years. Yudkowsky wrote Inadequate Equilibria in 2017 where he and his wife invented the idea, and Raemon wrote a playbook in 2018 for how to do it yourself. Now I and at least two other friends are trying to build something similar, and I suspect there's a bigger-than-it-looks market opportunity here because it's one of those things that a lot of people would probably want, if they knew it existed and could experience it. And it's only recently become c

... (Read more)

Thanks, this comment is really useful!

It is generally accepted that you do not need to go to direct sunlight type lux levels indoors to get most of the benefits.... I am not an expert on the latest studies, but if you want to build an indoor experimental setup to get to the bottom of what you really like, my feeling is that installing more than 4000 lux, as a peak capacity in selected areas, would definitely be a waste of money and resources.

Do you have any pointers to where I might go to read the latest studies?

1benkuhn3hWhere are you getting this number? As far as I know, the most efficient LEDs today are around 50% efficient.

Yesterday:

There is this very, very old puzzle/observation in economics about the lawyer who spends an hour volunteering at the soup kitchen, instead of working an extra hour and donating the money to hire someone...

If the lawyer needs to work an hour at the soup kitchen to keep himself motivated and remind himself why he's doing what he's doing, that's fine.  But he should also be donating some of the hours he worked at the office, because that is the power of professional specialization and it is how grownups really get things done.  One might consider the check as buying the righ

... (Read more)

Does anyone really track the marginal utility of their possible investment this way? Utilons - sure. But ROI on status? ROI on "warm fuzzies"?

Also, this assumes we have good estimates of the ROI on all our options. Where do these estimates come from? In the real world, we often seem to spread our bets - constantly playing a game of multi-armed bandit with concept drift.

Ungendered Spanish
1814h1 min readShow Highlight

Spanish has gramatical gender in a way English doesn't:

una amiga ruidosa — a loud (female) friend
un amigo ruidoso — a loud (male) friend
unas amigas ruidosas — some loud (female) friends
unos amigos ruidosos — some loud (not-all-female) friends

I remember when I was studying Spanish, learning the rule that even if you had a hundred girls and one boy you would use the male plural. My class all thought this was very sexist and unfair, but our teacher told us we were misapplying American norms and intuitions.

It's been interesting, ~twenty years later, following the developm... (Read more)

5MakoYass6hMy perception as a nonbinary is that this order of events makes things difficult Edit: A more succinct way of saying this is; making the neutral pronoun mean "third gender" will make it harder for it to come to mean "indeterminate gender", although The Third Gender is often defined as indeterminacy, I'm not sure how true or obvious that is for a lot of nbs Having the nonbinary identity enter public consciousness seems to have caused the neutral pronoun to take on a weight and colour that makes it harder to apply it to non-nonbinary people. In English, since use in situations where gender is irrelevant is already grammatical, so I'd guess this has a negligible effect on usage (though it does seem to have caused a notable amount of brain inflammation in terfs and reactionaries that I must mention but probably shouldn't go into depth about), but in a different place, seems like this might be more of a thing If you make it about identity first, gender-neutral terms become charged, and the second phase of making them common and truly neutral and uncharged will be delayed. Some other force I'm not aware of could overwhelm these ones. I just find it a little hard to imagine. Oh well. Most cultural shifts, at some point, were hard to imagine. But, as an alternative: The internet is an environment where reference-without-knowing-gender is likely to frequently occur. Maybe it would be better to start by advocating the use of genderless pronouns on spanish internet as a default, and talk about why that's important for everyone (why is it important for everyone?), and then start talking about nonbinary people later.
5jkaufman6hThis isn't how I think the path of 'they' has gone in English? Using it where gender is irrelevant is super new ("my friend said they might be late") and felt wrong to me ten years ago. Having there be specific individuals who go by 'they' feels like it has done a lot to get people to practice and be comfortable with 'they', though it's possible I'm paying too much attention to my local communities?

It's interesting to hear that, I didn't realise that much change had occurred.

I would guess that the normalisation would have come from people spending a lot of time online/being in more situations where they don't want to and don't have to disclose a person's gender. Hm. I can see how the "they seem queer, don't want to assume their gender" might have promoted adoption by a lot.

4Viliam8hLucky they had the "-e" available. (I don't suppose a language would exhaust all vowels by having five genders, but I could imagine a language were e.g. all "female" words are randomly assigned endings "-a", "-e", "-i", and all "male" words are randomly assigned endings "-o", "-u", and there is no gender-neutral vowel left.)

Consider the following program:

f(n):
    if n == 0:
        return 1
    return n * f(n-1)

Let’s think about the process by which this function is evaluated. We want to sketch out a causal DAG showing all of the intermediate calculations and the connections between them (feel free to pause reading and try this yourself).

Here’s what the causal DAG looks like:

Each dotted box corresponds to one call to the function f. The recursive call in f becomes a symmetry in the causal diagram: the DAG consists of an infinite sequence of copies of the same subcircuit.

More generally, we can represent any Tu

... (Read more)

For what it's worth, my current thinking on brain algorithms is that the brain has a couple low-level primitives, like temporal sequences and spatial relations, and these primitives are able to represent (1) cause-effect, (2) hierarchies, (3) composition, (4) analogies, (5) recursion, and on and on, by combining these primitives in different ways and with different contextual "metadata". This is my opinion, it's controversial in the field of cognitive science and I could be wrong. But anyway, that makes me instinctively skeptical of world-modeling theories

... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post

I was inspired by the APOD pictures and discussion here and here. The conditions for the 'experiment' are:

  • For some mysterious reason, all the water on Earth (minus that in living things - the biosphere) suddenly & immediately converges into 17 identical sphere and let to drop down to the land below right away.
  • The places of those water balls are the centers of 17 biggest tectonic plates as listed here.
  • The bottoms of the huge balls just touch the highest points on the ground there. Also from my calculation, which may be wrong, their radius is 258 km.
  • The ISS is currently at its hig
... (Read more)

Woah, tks a bunch man. But exactly what happens starting from t=0? I suppose that at 1st the water must be falling down, right? How will the Earth's surface be altered by the tremendous force of water? How will the potential energy from height turn 40% of water into vapor? I mean, how will it happen over time? If it takes time, then maybe some people will have a chance to understand what's going on & run into the nearest underground mine, no?

Regarding the ISS, I suppose that even at the hypothetical altitude of 460km, it will still burn. But ... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post

What is Abstraction?Ω
181d4 min readΩ 7Show Highlight

Let's start with a few examples (borrowed from here) to illustrate what we're talking about:

  • We have a gas consisting of some huge number of particles. We throw away information about the particles themselves, instead keeping just a few summary statistics: average energy, number of particles, etc. We can then make highly precise predictions about things like e.g. pressure just based on the reduced information we've kept, without having to think about each individual particle. That reduced information is the "abstract layer" - the gas and its properties.
  • We have a bunch o
... (Read more)
1Donald Hobson13hI think that there are some abstractions that aren't predictively useful, but are still useful in deciding your actions. Suppose I and my friend both have the goal of maximising the number of DNA strings whose MD5 hash is prime. I call sequences with this property "ana" and those without this property "kata". Saying that "the DNA over there is ana" does tell me something about the world, there is an experiment that I can do to determine if this is true or false, namely sequencing it and taking the hash. The concept of "ana" isn't useful in a world where no agents care about it and no detectors have been built. If your utility function cares about the difference, it is a useful concept. If someone has connected an ana detector to the trigger of something important, then its a useful concept. If your a crime scene investigator, and all you know about the perpetrators DNA is that its ana, then finding out if Joe Blogs has ana DNA could be important. The concept of ana is useful. If you know the perpitrators entire genome, the concept stops being useful. A general abstraction is consistent with several, but not all universe states. There are many different universe states in which the gas has a pressure of 37Pa, but also many where it isn't. So all abstractions are subsets of possible universe states. Usually, we use subsets that are suitable for reasoning about in some way. Suppose you were literally omniscient, knowing every detail of the universe, but you had to give humans a 1Tb summary. Unable to include all the info you might want, you can only include a summery of the important points, you are now engaged in lossy compression. Sensor data is also an abstraction, for instance you might have temperature and pressure sensors. Cameras record roughly how many photons hit them without tracking every one. So real world agents are translating one lossy approximation of the world into another without ever being able to express the whole thing explicitly. How y
2johnswentworth11hI think this is technically true, but less important than it seems at first glance. Natural abstractions are a thing, which means there's instrumental convergence in abstractions - some compressed information is relevant to a far wider variety of objectives than other compressed information. Representing DNA sequences as strings of four different symbols is a natural abstraction, and it's useful for a very wide variety of goals; MD5 hashes of those strings are useful only for a relatively narrow set of goals. Somewhat more formally... any given territory has some Kolmogorov complexity, a maximally-compressed lossless map. That's a property of the territory alone, independent of any goal. But it's still relevant to goal-specific lossy compression - it will very often be useful for lossy models to re-use the compression methods relevant to lossless compression. For instance, maybe we have an ASCII text file which contains only alphanumeric and punctuation characters. We can losslessly compress that file using e.g. Huffman coding [https://en.wikipedia.org/wiki/Huffman_coding], which uses fewer bits for the characters which appear more often. Now we decide to move on to lossy encoding - but we can still use the compressed character representation found by Huffman, assuming the lossy method doesn't change the distribution of characters too much.

An abstraction like "object permanence" would be useful for a very wide variety of goals, maybe even for any real-world goal. An abstraction like "golgi apparatus" is useful for some goals but not others. "Lossless" is not an option in practice: our world is too rich, you can just keep digging deeper into any phenomenon until you run out of time and memory ... I'm sure that a 50,000 page book could theoretically be written about earwax, and it would still leave out details which for some goals would be critical. :-)

CO2 Stripper Postmortem Thoughts
1207d8 min readShow Highlight

[EDIT: A crucial consideration was pointed out in the comments. For all the designs I've looked at, it's cheaper to just get a heat exchanger and ventilation fans, and blow the air outside/pull it inside and eat the extra heating costs/throw on an extra layer of clothing, than it is to buy a CO2 stripper. There's still an application niche for poorly ventilated rooms without windows, but that describes a lot fewer occasions than my previous dreams of commercial use.]


So, I have finally completed building a CO2 stripper that removes CO2 from the air to (hopefully) improve cognitio... (Read more)

I was very confused about your proposed setup after reading the wikipedia article on heat exchangers, since I couldn't figure out what thermal masses you proposed exchanging heat between. But I found this article which resolved my confusion.

[Link]The Lesson To Unlearn
286h1 min readShow Highlight

The most damaging thing you learned in school wasn't something you learned in any specific class. It was learning to get good grades.

I found it a surprisingly fresh take, given so many shared starting assumptions. I really enjoy reading someone thinking aloud for themselves, even on a topic that's been talked about so much. And a surprisingly optimistic conclusion.

Load More