Wiki Contributions


To get aligned AI, train it on a corpus generated by aligned humans.

Except that we don't have that, and probably can't get it.

I'm not sure why all the people who think harder than I do about the field aren't testing their "how to get alignment" theories on humans first -- there seems to be a prevailing assumption that somehow throwing more intellect at an agent will suddenly render it fully rational, for reasons similar to why higher-IQ humans lead universally happier lives (/s).

At this point I've basically given up on prodding the busy intellectuals to explain that part, and resorted to taking it on faith that it makes sense to them because they're smarter than me.

If your goals include communicating your ideas to a wide audience, please consider dropping your average sentence length and using sub-headings to encapsulate the core idea of each section. I'm low on intellectual energy today and tried to do my usual thing of skimming an article with a neat title to see if it seems like something I want to engage more deeply with, and the current format basically renders it un-skimmable.

If this is more writing-as-artistic-expression and you don't mind if it only gets through to people experiencing top-decile mental functioning at the moment they come across it, keep up the good work.

I initially wanted to paraphrase your description of tapping out in physical pursuits as "a question is answered to my satisfaction", but that's not quite right. More precisely, it sounds like a signal of the inflection point between "better to continue" and "better to stop". The standards of "better to continue" might include pursuing a hope of turning the situation around or getting closer to the goal of an exercise, and standards of "better to stop" might include avoiding immediate or delayed pain or injury, or to simply avoid wasting time after the goal of a drill has been achieved.

"tapping out" carries too much cultural baggage about loss and submission to seem promising, but we would certainly benefit from a term capturing the moment when a line of discussion hits diminishing or even negative returns for its participants and is thus a waste of time to continue.

This makes me realize that I don't even have a particularly good term for the category of topic that gets its social/emotional hooks into the participants and forces/coerces/drags them into continuing discussing/debating it long after any real value has already been extracted from the exchange. (scissor statements come close, and might be a superset of this category, but aren't exactly what I'm referring to)

LW is maximally convenient for me, although I'm not against other options if you or others have strong preferences. The recently released Dialogues feature may do well for this sort of thing, too.

Comment threading is a good fit for the fractal nature of exploring ideas, and we could do that with comment threading on this post or on a shortform.

Bio risks are a nicely concrete case for a subtlety of the prepper mindset that I expect I may explain poorly/incompletely on my first few tries: Surviving and thriving when part of what we take for granted goes away looks very similar regardless of why the thing went away.

The rule of 3s is a good touchstone here, concretely to bio risk:

  • Need air. Bio risk may render some/all air unsafe to breathe directly. Having the means to detect whether air is safe to breathe, and remove some/all contaminants from air you're about to breathe when appropriate, is good for preparedness/survival.

  • Need shelter. Bio risk fallout/consequences may have economic impacts that affect one's ability to keep one's accustomed housing/shelter. Good financial preparedness and a backup plan for shelter will address these consequences of bio risk.

  • Need water. Bio risk may contribute to newly acquired water being unsafe to drink without bad health outcomes. Prepare through a combination of storing some safe water, and having the means to turn unsafe water into safe water (various combinations of disinfection, filtration, and distillation may be appropriate depending on the situation)

  • Need food. Bio risk may contribute to newly acquired food being unsafe to consume, or to no food available due to supply chain impact. Prepare by storing some food and optionally having a long-term plan to produce some/all of your own food in a long-term crisis.

Now, each of these practical interventions is also really good for addressing other risks:

  • Prepping to breathe clean air in a bad atmosphere increases your chances if a volcano makes a bunch of ash, an earthquake makes a bunch of dust, a nuclear event puts particles into the air that you don't want to be breathing, there's a particularly bad flu year and you still want to go to the grocery store without breathing what everyone is coughing, etc.

  • Prepping for continuity/eventualities/options of shelter also increases your chances of [quality * quantity] of life if natural disaster destroys your home, economic disaster destroys your ability to keep your home, maybe even if personal disaster destroys your home if you include redundancy to storage locations. Prepping for shelter redundancy is also great in thermal emergencies -- in extreme cold, living in a tent in a house is warmer/better than living in just the house or just the tent.

  • Prepping for clean water is good if you're on municipal water and get a boil water advisory, or natural disaster impacts water supply, or if the water company makes a mistake and shuts yours off, or if your well pump breaks, etc.

  • Prepping for safe food is also good if you lose access to regular supply chains for any reason -- local/regional disaster where the food is grown, crop failure, economic issues meaning you can't afford new food, and more

This is basically a low-resolution screencap from a complex film about how the correct preparedness actions to take are very often independent of the risk that a specific disaster caused the action to be relevant.

Caveats, of course, are that some disasters have unique preps that help with them disproportionately. Potassium iodide pills in case of nuclear emergency are the classic example -- every 14 doses is basically a "get out of extreme thyroid cancer risk free" card for 1 adult in the specific scenario of radioactive iodine being present in the environment after a disaster. Similarly, there may be some unique preps for certain classes of bio emergency -- maybe petri dishes, agar medium, and reagents to do rudimentary analysis and detect whether a surface has been contaminated by one thing rather than another? I don't know enough about bio risk to give a good example there.

This toy example also highlights the skill component:

  • Prepping to breathe clean air is only as good as the correctness of your mental model about invisible contamination in air. You also need a flexible model -- during wildfire season when the whole atmosphere is smoky, indoors breathing the same air over and over is net safer than outside. But during flu season, indoors with lots of people is net more hazardous than outside.

  • Stuff you own to provide yourself with improvised shelter is only as good as your ability to use it. If you set up a tent in a location that becomes a big puddle when it rains, and/or fail to stake it properly on a windy night, you're about to be a lot worse off than you were before.

  • Water purification is only as good as your ability to use it and the correctness of your mental model about how contamination works. If you "purify" your water in a way that mixes it with contaminated water, the output is still contaminated. Conversely, if you try to distill everything when a few minutes of boiling would have sufficed, you're going to waste a whole lot of fuel that you probably didn't have to spare.

  • Stored food is only as good as your ability to prepare it. Classic vignette of the guy in the bomb shelter with all the canned goods and no can opener, or the old lady who only has an electric can opener and thus can't get to her food during a power outage. If you go further and store whole wheat, you'd better own a grain mill if you want to turn it into flour.

So my first criticism is that it is unclear what is being prepared for.

Having been prepper-adjacent for longer than I've been knowingly rationalist-adjacent, I notice that I am surprised you've had this experience.

I used to spend way too much time in /r/preppers, so I'm inclined to take their wiki as characteristic of the community. Second line of the first post lined by the intro wiki page (, "Simply put emergencies happen that's why.", sums it up.

Outside of rationalist circles, changing someone's mind about whether they ought to prepare is not generally a positive experience. Enough bad times resulting from such attempts can lead one to believe that attempting to change peoples' minds on the fundamentals is pointless, so effort is better spent writing guides for people who have already identified a scenario that they find compelling.

I agree with your premise that it's a good thing to help people figure out what they want to prep for, but I also see how the problem hasn't been addressed yet because it runs into a lot of annoying social stuff.

I think as you start on a guide, that process itself may help clarify just how personal every individual's risk profile is, and the magnitude of deviation between how different individuals can approach the "same" problem.

or I have done a botch job of my first LW post

LW's expressed priorities have shifted immensely since the whole AI thing really took off, and many/most of those popular posts are from before it. Also, you're jumping in with a large and cognitively demanding post, whereas many of the most popular prepper posts on here are much smaller. Don't take it personally, nor give up on the hope of turning a manual into a popular series on here as it digests down into smaller and more approachable components.

Got a list of tasks you're hoping to tackle on this, or shall I read the original post in more depth and infer it? If I meet task-sized pieces of the question, I'm likely to throw blocks of text at you for awhile, because writing is a lot of fun in relatively low-stakes or high-safety-net contexts like this (as in, I'm not scared that a mistake in something I throw at you would harm others, because I'm imagining a lot of layers of editorial and fact-check review before it gets out into the world).

I read your first couple sections and then realized my views on preparedness are sufficiently different from yours that I'm probably going to end up unhelpfully nitpicking if I try to assess the rest without getting closer to your starting views first, so I've only skimmed the rest for now.

Which existing guides are you critiquing? There are a lot of inadequate guides out there; this is a known issue with preparedness in general. Guides ultimately help one imagine possible scenarios, and then one has to answer for oneself how those scenarios would go.

What are your beliefs/assumptions about whether people are following the available guides? This segments the population into rough quadrants:

  • care about guides, follow advice from guides as best they can
  • don't care/know about guides, incidentally happen to already be taking the steps that the guides propose for other reasons
  • know/care about guides, don't follow the advice from them
  • don't know/care about guides, don't care about preparedness, don't take preparedness actions

I'm curious whether you're targeting this more toward the "don't know / don't care" population, or the people who are already following existing guides, or what.

One clarification that might help me see where you're coming from is what you think gets someone into the target audience of your guide. When you've discussed the idea with people in that audience, what are their top complaints about existing resources?

The activities list section might have a chance at making the case for skills being more important than items. Practicing the universal edibility test on any pantry item past its best-by date should probably make the cut. Another worthwhile activity is to set a random alarm and when it goes off, figure out how you'd make fire with only what's in your immediate environment. Test hypotheses about fire only in a safe space, of course (and understand fire well enough to tell if a space is safe). Activities for using PPE correctly include fit testing a respirator with aerosolized scents or flavorings, and finger painting with rubber gloves on then removing the gloves without getting any paint on your hands.

like a recipe with specific and easy actions someone can take.

IMO, this will ultimately deter people from preparedness in the long term, because following a generic formula won't add convenience in situations that users actually encounter. A more promising format would feel like a flowchart: "If the stores were closed, what would you eat? How long would that last you?" "what do you keep in your house already because you use it? How much more of that could you store such that you'll use each item before its best-by date?"

You're probably already imagining more of a flowchart than a recipe already -- for instance, someone who can't eat gluten should not be storing pasta -- but it'll work better for a broader audience if those assumptions end up as clear and explicit as possible.

tl;dr I'm easily nerd-sniped by preparedness logistics, and happy to construct logistical advice if you find yourself wanting it. I'm not your person for the statistical justification side of things, and not much for graphics when it's time to make stuff pretty, but as a professional pessimist I can find and often fix a lot of vulnerabilities to what-about-ism.

I agree that fundamentally original art has traits that make it better than fundamentally plagiarized art.

However, humans can plagiarize too. We do it a bunch, although I'd argue that on the whole, art plagiarized by an AI will look "better" than art plagiarized by a human. While the best human plagiarists may create better work than the best AIs for now, the average human plagiarist (perhaps a child or teen tracing drawing their favorite copyrighted character) creates output far below the quality that the average AI can generate.

When you make the question about what species of entity created a piece of art instead of whether the art is original in the way that makes it better, it would follow that a human plagiarist creates a higher quality product than a robotic one, in ways that directly contradicts my experience of AI vs human plagiarism.

What I find wrong with saying "human" to mean "person" is what others have found wrong with with saying "man" or "citizen" to mean "person" in the past. If you can imagine AIs eventually being "people" in a way that would render them deserving of empathy, it's hard to justify normalizing species-based linguistic shortcuts that allow the accident of one's birth to artificially cap one's maximum attainable value in society.

Then again, I believe that if "person" is a status that an entity has to prove that it deserves, humans should prove their way into it just like we expect other creatures and entities to do. This concept is neither popular nor practical to implement.

In this vein, the only behavior displayed in the original post that reads as less "intelligent" to me is assuming the [existence * importance] of trainable abstract intelligence.

I notice that people who've gotten a lot of the cultural "you're so smart" feedback tend on the whole to be skeptical of abstract intelligence as an independent trait, perhaps because of the repeated experience of being told one has a trait that doesn't subjectively feel like it has a specific presence or location.

This gets me wondering if the feeling that one doesn't "have intelligence" in the way that one "has height" or "has happiness" or even "has verbal fluency" is universal, and the difference in how individuals interpret the absence-of-experience could be fully explainable by social context and feedback.

Answer by nimNov 08, 202331

For problem-solving in particular, I've really enjoyed how this youtuber articulates specific strategies.

Why do you want greater intellect -- what do you want to do with it?

Consider how to get directly to those ends with the tools you have available. Consider the option of disregarding the abstract concept of "intellect" entirely, and simply going out and doing whatever it is you're waiting to do until you "get it", modifying your approach after each mistake.

Often when those around me describe something I said or did as intelligent, it actually feels from inside as if I simply identified my goal and went straight toward it rather than getting hung up on the expectations about what was or wasn't "supposed to" work. The insights and solutions that seem intelligent to onlookers come less from having some ephemeral "iq" trait and more from my willingness to notice the irrelevance of imagined constraints and disregard them.

Also, if you reflect on your attempts to learn things, it refines the meta-skill of learning stuff. I've been surprised by my ability to learn things I'd previously assumed I'd hit the intellectual wall on when I revisit them after having had more experiences of learning things in different ways. Avoid the trap of assuming that failing to grasp something quickly in one situation means you'll never grasp it quickly in another.

I haven't read through the links, so you may have answered this elsewhere, but I'm curious how you manage to maintain motivation in the face of uncertainty about the effectiveness of whatever giving you choose. It seems like the feedback loop of telling whether a donation made a difference in something like global poverty would be on the order of years, whereas the feedback loop of working directly with people in need might be on the order of minutes.

Also, how do you reason about the amount of additional giving necessary to offset any negative societal consequences of the field that you work in? For instance, an oil rig worker would be able to give a lot more than a schoolteacher, but working on an oil rig directly facilitates a lot of environmentally questionable outcomes in a way that corralling peoples' kiddos for them would not.

Load More