I agree that it's valuable to note that information hazards can sometimes hurt the person who gets the information themselves. And I agree that Bostrom's sense of information hazards is definitely broader than just that, so if people are using infohazards to mean just information that harms the person who knows it specifically, then clearing up their confusion seems good.

But I don't know if memetic hazards is a great term for that, because it seems most natural to use the label "memetic hazards" for a superset of information hazard... (read more)

Showing 3 of 11 replies (Click to show all)
6MichaelA2mo(Personal, quick-fire views - not Convergence's) Ok, your comments have definitely updated me towards thinking the non-intuitive (in my view) usage of "memetic hazards" is more established than I'd thought, and thus towards being less confident about trying to appropriate the term for what I still do think is a more intuitive usage. I also definitely agree that conflicting terminology is worth making efforts for avoid, where possible. One thing I should add is that we're ideally aiming to not just point at a superset of infohazards, but also to emphasise the "memetic" aspects (mutations, proliferation of more catchy ideas, etc.). I think I agree that there's not that much value in a term just staking out the idea of "true or false info that causes harm", but a term that also highlights those "memetic" aspects does seem (to me) worthwhile. And unfortunately, "communication/concept hazard" doesn't seem to capture that (although I did consider "concept hazard" for a moment, when I started reading your comments and thus started again on trying to find alternatives). And reading Dagon's comment has also updated me further towards thinking that a term that highlights the memetic aspects is useful, and that the current usage of "memetic hazards" is not ideal! So ultimately I mostly feel unsure how best to proceed. I do feel pretty confident that it'd be better to use a term other than "memetic hazard" for an info hazard where it's the knower who's hurt by knowing. I think several of the suggestions that have been made seem workable, including "direct information hazard" (thanks for highlighting that option). "Direct information hazard" could also intuitively mean other things, but it doesn't seem to have one obvious intuitive meaning that conflicts with what we want to use it for, so it beats "memetic hazards" on that front. (So this paragraph is me suggesting that, while perhaps we shouldn't repurpose the term "memetic hazards", we should still avoid spreading or fur
5willbradshaw2moThanks for this. I think that even with the edits I was probably too confrontational above, so sorry about that. I'm not sure why this issue is emotional for me, that seems weird. To start off, I agree that, ceteris paribus, the current usage of "memetic hazard" is strange. It has the advantage over e.g. "direct IH" of sounding cool and scary, which was probably desirable for SCP-like uses but is perhaps not ideal for people actually trying to do serious work on info-hazardy concepts. I notice a conflict in my thoughts here, where I want to be able to refer to knower-harming hazards with a term that is (a) distinctive, evocative and catchy (such that it seems compelling and is actually used in real situations) and (b) sober, precise and informative (such that it can be used productively in technical writing on the subject). "Memetic hazard" satisfies (a) but not (b); "direct information hazard" satisfies (b) but not (a). This is not ideal. I think for academic-ish work the term "direct info hazard" or something similarly bland is a fine descriptor for "knower-harming information". I'm not sure what sort of term we would want to use for more popular work. "Knowledge hazard" seems okay to me? But I agree more suggestions here would be valuable. -------------------------------------------------------------------------------- Insofar as "memetic hazard" is being used simply to mean "knower-harming information hazard", this seems reasonable. The term is still obscure enough that if enough people jumped on a new term it could probably gain more traction, and "memetic hazard" can be left as an obscure and kinda-confusing synonym of [whatever the new term is] that people bring up from time to time but isn't widely used outside SCP. [One counter-consideration. Having skimmed some existing usage of "memetic hazard" on the internet, it seems some people are using it to mean a directly (?) harmful idea that also encourages its bearers to spread it. The blandest form of t

I've just posted an introduction to/summary of/clarification of the concept of information hazards on the EA Forum. I was halfway through writing that when I came across this post and made these comments. You and eukaryote helped bring to my attention that it'd be valuable to note the "direct information hazards"/"knowledge hazards"/whatever subset, and that it's just a subset, so that's in there (and thanks for that!). The post doesn't touch on the "memetic" side of things - Convergence will get to th... (read more)

A point of clarification on infohazard terminology

by eukaryote 2 min read2nd Feb 202021 comments


TL;DR: “Infohazard” means any kind of information that could be harmful in some fashion. Let’s use “cognitohazard” to describe information that could specifically harm the person who knows it.

Some people in my circle like to talk about the idea of information hazards or infohazards, which are dangerous information. This isn’t a fictional concept – Nick Bostrom characterizes a number of different types of infohazards in his 2011 paper that introduces the term (PDF available here). Lots of kinds of information can be dangerous or harmful in some fashion – detailed instructions for making a nuclear bomb. A signal or hint that a person is a member of a marginalized group. An extremist ideology. A spoiler for your favorite TV show. (Listen, an infohazard is a kind of hazard, not a measure of intensity. A papercut is still a kind of injury!)

I’ve been in places where “infohazard” is used in the Bostromian sense casually – to talk about, say, dual-use research of concern in the biological sciences, and describe the specific dangers that might come from publishing procedures of results.

I’ve also been in more esoteric conversations where people use the word “infohazard” to talk about a specific kind of Bostromian information hazard: information that may harm the person who knows it. This is a stranger concept, but there are still lots of apparent examples – a catchy earworm. “You just lost the game.” More seriously, an easy method of committing suicide for a suicidal person. A prototypical fictional example is the “basilisk” fractal from David Langford’s 1988 short story BLIT, which kills you if you see it.

This is a subset of the original definition because it is harmful information, but it’s expected to harm the person who knows it in particular. For instance, detailed schematics for a nuclear weapon aren’t really expected to bring harm to a potential weaponeer – the danger is that the weaponeer will use them to harm others. But fully internalizing the information that Amazon will deliver you a 5-pound bag of Swedish Fish whenever you want is specifically a danger to you. (…Me.)

This disparate use of terms is confusing. I think Bostrom and his intellectual kith get the broader definition of “infohazard”, since they coined the word and are actually using it professionally.*

I propose we call the second thing – information that harms the knower – a cognitohazard.

5-pound bag of swedish fish

Pictured: Instantiated example of a cognitohazard. Something something red herrings.

This term is shamelessly borrowed from the SCP Foundation, which uses it the same way in fiction. I figure the usage can’t make the concept sound any more weird and sci-fi than it already does.

(Cognitohazards don’t have to be hazardous to everybody. Someone who hates Swedish Fish is not going to spend all their money buying bags of Swedish Fish off of Amazon and diving into them like Scrooge McDuck. For someone who loves Swedish Fish – well, no comment. I’d call this “a potential cognitohazard” if you were to yell it into a crowd with unknown opinions on Swedish Fish.)

Anyways, hope that clears things up.

*For a published track record of this usage, see: an academic paper from Future of Humanity Institute and Center for Health Security staff, another piece by Bostrom, an opinion piece by esteemed synthetic biologist Kevin Esvelt, a piece on synthetic biology by FHI researcher Cassidy Nelson, a piece by Phil Torres.

(UPDATE: The version I initially published proposed the term “memetic hazard” rather than “cognitohazard.” Commentor MichaelA kindly pointed out that “memetic hazard” already meant a different concept that better suited that name. Since I had only just put out the post, I decided to quickly backpeddle and switch out the word for another one with similar provinence. I hate having to do this, but it sure beats not doing it. Sorry for any confusion, and thank you, MichaelA!)