Great, interesting, thought-provoking post, Eliezer. As you suggest towards the end, a multiplex of strategies regarding relating science to the public may be best.

I'm for creating straussian brands of widespread and/or fast-growing belief systems to compete with their sincere or exploitive versions. For example, rather than a cult of science made from scratch, I think a version of evangelical christianity where the top executives are focused on making the behavior of the masses more in our rational self-interest would be the better approach (although it would be interesting to see both tried).

Insightful, as always, but this seems like it may have the esoteric value of some knowledge the wrong way around. There are certain questions, like "What is the meaning of life?" that science cannot answer the way people want to hear (as, "that questions is incoherent and pointless" is rarely viewed as satisfactory, regardless of its accuracy). It seems people choose religion because they are seeking answers to some such question (or, because their parents chose it), and they end up swallowing the earth being 6000 years old almost as an afterthought.

I love most of your posts, but I think you might be off on this one.

Why successful cultures reward people who tried to discover and understand knowledge that was already known? I don't think they would; I think they'd reward people who go after secrets, mysteries, etc. People who make use of knowledge (preexisting or otherwise) for their own gain are of course rewarded anyway, but for the most part I don't think that describes science very well, does it?

What would be the purpose of rewarding people (who cannot make use of the knowledge) to 'discover' things already in the public domain? Wouldn't we rather have them striving to solve puzzles? I know its enjoyable for many of us to 'discover' science on our own, but for your average person I think thats a waste of time. In other words, I think scarcity in knowledge is beneficial for society, though maybe not for science nuts.

Can't you usually audit courses in most universities for free?

I think I, along with Grant, may have to disagree with portions of this entry. It seems your basic point is that if science was set up in the way the scientology and other cults are, it would obtain an increase in interest and pursuit. The problem I see with that is that cult knowledge is easy; science is hard. Learning a couple of simple beliefs at each level is simple. However, much like many people who would love to be a CEO of a major corporation, but abstain from attempting to achieve this goal due to the increased stress and demands of the position, the "cult of science" would face a similar issue. Learning advanced mathematics is not for the timid.

Well, ideally one would want us to be smart enough to not automatically overvalue hidden information or undervalue availible information. :)

Also, I'm reminded of a bit from the Discworld book Thief of Time. Kaos says to Sweeper something to the effect of "...You know all the hidden wisdoms. No, more than that. I suspect you are even wiser. You know the explicit wisdoms, the knowledge hidden in plain sight and ignored by most as they seek hidden knowledge."

Eliezer: "The Singularity Institute has a human-resources problem and a PR problem. I've been taking haphazard potshots at this for 8 years and it hasn't worked. So I'm taking the time to write out full explanations, and the basics, and will eventually produce a book, and teenagers will read the book, go on to read other books, and then be hired by SIAI 7 years later. That, I'm afraid, is how it has always worked around here."

Interesting. And probably a good strategy. In my humble opinion, the most worthwhile things that have been written on the subject of Friendly AI have been written by Eliezer Yudkowsky. But who's to say that Eliezer has all the answers? It takes a lot of wisdom to realize that your best strategy might be to convince a large number of others to work on a problem that might be too difficult for any one person - including yourself - to solve.

I can personally testify that Eliezer's writing was instrumental in alerting me to the urgency and importance of working on friendly AI, and that is indeed what I plan to do.

Most people don't know how light bulbs work because it doesn't matter how light bulbs work. They can't use that knowledge in their daily lives, so it really doesn't matter how difficult it is to acquire that data - as long as it's difficult enough, it's not worthwhile to expend the effort.

Being a physicist doesn't give you any nifty powers, and the phenomena they have the knowledge to predict don't affect the everyday lives of human beings on a perceptible level.

Most people weren't clamoring to learn the Pythagorean secrets, either.

Many more people are studying science than can actually hope to find jobs in the field.

The real problem is not a scarcity of people, but a scarcity of smart people. The average guy in the street will not improve his own life or anyone else's by the study of science. Posts for lab technicians are easy enough to fill, after all.

Conversely, the people who really can make a difference by and large do not need any encouragement.

On a practical note, I would be very interested in a discussion of the best ways an individual can make a monetary / political / social contribution to the development of an AGI. Assuming this has already been argued out, does anyone have a link?

Psychohistorian: Insightful, as always, but this seems like it may have the esoteric value of some knowledge the wrong way around. There are certain questions, like "What is the meaning of life?" that science cannot answer the way people want to hear (as, "that questions is incoherent and pointless" is rarely viewed as satisfactory, regardless of its accuracy).

I don't think that's the answer science gives, at least, not the complete answer. This would be an excellent example of "wrong questions" that Eliezer Yudkowsky discussed before (note: he linked this despite it not containing the solution to qualia suggested in the anchor text) and an excellent opportunity to right the wrong question.

"What is the meaning of life?"; rephrase as your confusion--> "Why do I want to know the meaning of life?"; taboo "meaning of" --> "Why do I want to know what signficance life has beyond what I can observe?" and so on.

Dismissing the first question should not be the end of it.

"So I'm taking the time to write out full explanations, and the basics, and will eventually produce a book, and teenagers will read the book, go on to read other books, and then be hired by SIAI 7 years later."

I would certainly enjoy reading your book, and I'm confident that thousands of other people would enjoy it as well. However, I'm not confident that a book on rationality alone will do much for recruiting- Hofstadter's GEB was well-written and very widely read, and it didn't help recruit people for any specific organization.

"I'm just explaining why it is that I won't tell anyone the Secret of how the ineffable difference between blueness and redness arises from mere atoms for less than $100,000 -"

I'll give you $100 for a reasonably detailed explanation of this, as long as you publish it under the GFDL or a similar license.

"I'm just explaining why it is that I won't tell anyone the Secret of how the ineffable difference between blueness and redness arises from mere atoms for less than $100,000 -"

I'll give you $100 for a reasonably detailed explanation of this, as long as you publish it under the GFDL or a similar license.

I can think of a reason why Eliezer Yudkowsky won't take you up on this offer...

I mean, above and beyond not being able to hold up his end ;-)

Mastering science is hard work...and it requires a willingness to ask hard questions and explore them wherever they may lead. Most secret cults survive by demanding allegiance to the faith, obedience to doctrine, and rejecting any critical examination of those beliefs or their origins.

Science, despite its lack of pop culture acclaim, is responsible for most of the progress in this world---progress against disease, against poverty, progress in describing and understanding the world and the universe in which we live. The openness of science to questions and to critical examination, and its adherence to standards of verification, make it the best method we have for separating truth from myth.

It does not however rid the world of stupid people....and that, my friend, is the real problem

In Nick Bostrom's paper on the survival of humanity, several potential catastrophe scenarios are technological ones. That makes me think that it might actually be a bad idea to popularize science.

The irony here is that information about how to create a catastrophe - how to make a nuke, how to construct viruses in a laboratory, how to make a nanobot - is just about the only scientific information that people are hiding. (Fortunatetly, though, they don't make a big deal about the fact they're hiding it.)

Animal sacrifices have been replaced by DRM.

I haven't followed this to any of numerous possible conclusions, but I found the analogy irresistible. Think about it.

@spindizzy:

No, this hasn't been "argued out", and even if it had been in the past, the "single best answer" would differ from person to person and from year to year. I would suggest starting a thread on SL4 or on SIAI's Singularity Discussion list.

John: quite right. This actually reminds me of one of the common threads in Michael Crichton's works. From Jurassic Park:

'...Malcolm said. "A karate master does not kill people with his bare hands. He does not lose his temper and kill his wife. The person who kills is the person who has no discipline & no restraint, and who has purchased his power in the form of a Saturday night special. And that is the kind of power that science fosters, and permits. And that is why you think that to build a place like this is simple." "It was simple!", Hammond insisted. 'Then why did it go wrong?"'

Or:

'"I will tell you what I am talking about," he said. "Most kinds of power require a substantial sacrifice by whoever wants the power. There is an apprenticeship, a discipline lasting many years. Whatever kind of power you want. President of the company. Black belt in karate. Spiritual guru. Whatever it is you seek, you have to put in the time, the practice, the effort. You must give up a lot to get it. It has to be very important to you. And once you have attained it, it’s your power. It can't be given away: it resides in you. It is literally the result of your discipline.

Now what is interesting about this process is that, by the time someone has acquired the ability to kill with his bare hands, he has also matured to the point where he won't use it unwisely. So that kind of power has a built-in control. The discipline of getting the you so that you won't abuse it. But scientific power is like inherited wealth: attained without discipline. You read what others have done, and you take the next step. You can do it very young. You can make progress very fast."'

The thing is, even in our world most religious people don't in fact join esoteric orders.

In the US, a lot of religious people are mainstream Christians, and many of those don't even read the Bible, let alone study theology. They pick up a "password"-level understanding of Christianity which allows them to signal membership, and are content with that. The same is basically true of other religions.

In your alternate world, I suspect what you'd actually end up with is a lot of cargo cults.

Anathem takes a stab at this idea, by the way. It's delightful reading, but it doesn't take place in a universe very much like our own.

"Why haven't you given a public demonstration of your Eighth Level powers, like the Physicists?"

I continue to find this quote an invaluable heuristic. Thank you for the great post :)

Ahh... So this is why Harry and Draco meet at night wearing hooded masks instead of just studying in the library behind a quietus charm. :)

This was a fun post. While I enjoy the public access and general lack of rock climbing ablilities associated with learning science, it is a fun thing to contemplate. Maybe there is a way to implement this sort of thing in elementary school classrooms. Maybe kids would think science is more fun if you offered to teach them the amazing secrets if and only if they were able to give you a non-password answer for what hypotheses and theories are.

Learning Japanese was a very fun experience for me. Why? Because it felt I was infiltrating some kind of secret cult. Learning to actually read all those incomprehensible moonrunes was like learning how to decode their secret messages. It was... exciting.

You seem to be working on the assumption that imbuing science with an aura of mysteriousness, at the expense of availability of knowledge, would somehow attract more enthusiasts than it would repel. Or, in other words, that there are more people whose main reason for not being into science is an (inaccurate) idea of it as a dry, boring, completely un-mystical pursuit, than there are scientifically-inclined people that would become frustrated with the cultishness and the secrecy and eventually quit, in the eventuality of a scenario like the one you proposed.

This seems quite unlikely to me, and I'll explain why.

For the most part it's a feeling that the scenario you came up with is pandering to the wrong audience. Consider the present state of things. Science is more or less out in the open, if you have the resources (mental, financial, time) to learn; a large majority of people is apathetic about science even after (presumably) having had encounters with it at various points during their education; some people that have probably had about the same encounters with science have gotten enthusiastic about it, and obviously did not need any science cults to get them to pursue it. Your thought experiment seems to be designed to attract more of the first.

First there's the concern about whether that "untapped" audience really has the potential to bring contributions that match the effort spent with attracting them. These people have been exposed to science already -- and it had no effect on their interests! In this case, do you think they'd really join a mathematics "cult" for the equations, and not for the social standing or need to adhere to a group? If not, then do they hold promise as potential thinkers?

As for the second category of people, those who wouldn't need any of that bullshit social incentives anyway, like I said, the unavailability of knowledge might be a disincentive. Science is hard enough as it is. It takes years and years to gather a decent enough body of knowledge in order to actually get to make advances in your own field; introducing silly social rituals along the way won't speed up the learning process. Moreover, there's the danger of not getting your message across in the first place, of it not reaching all of the people that could have been interested. So, all in all, bad idea.

Of course, I know you meant this more as speculation than anything; the premise just seemed faulty to me.

I don't know to what extent we can hack our own perceptions of scarcity by intentionally directing our thoughts, but it seems like it's something worth trying to do:

"Scientific information is widely available. As a result, people will pay less attention to it than they would if it was hidden. As a result, it's better hidden than if it were kept partially secret. This means that scientific information is very scarce, and almost nobody knows that it is scarce."

Is there a way to phrase the above statement so that it carries the same psychological weight as, "Only a few people realize this now, but there's about to be a beef shortage"?

Edit: (... and is that the whole point of this post along with the story after it?)

Ooh, good point with those textbooks! ...and a painful one to think about.

I think that the assumptions that people join cults for gaining knowledge is flawed. Most people don't care about knowledge and truth.

Take scientology as example. Luke Muehlhauser about taking scientology 101:

This class - a religious class I took as an atheist in order to achieve an unrelated goal - turned out to be one of the most important classes I have ever taken in my life.

Most science classes on the other hand don't teach lessons that the student can afterwards use in his daily life. Science 101 is boring to a lot of students.

Most students of science classes don't feel like they get knowledge in the class that they has an immdieate effect. Religious rituals do something to people.

I can't help but think of the Ritual of the Calling of the Engineer. They date back nearly a century, and their ceremonies are conducted in private. The initiation ritual is conducted by the Corporation of the Seven Wardens, with only new and past initiates present, but no guests. Upon initiation though, each new member is given a symbolic Iron Ring, which is rumored to be forged from the mangled iron scavenged from the Quebec Bridge disaster on 1907. The ring is distinctly angular rather than smooth and comfortable, and is worn on the pinky of the dominant hand so that it will slide across an engineer's drafting table or paper as he or she writes. This is a constant reminder of an engineers lofty duties and responsibilities, and of those who died during the Quebec Bridge collapse.

For people in the US, there's the Order of the Engineer. Before joining, members must swear an oath to abide by "The Obligation of an Engineer":

I am an engineer, in my profession I take deep pride.

To it I owe solemn obligations.

Since the stone age, human progress has been spurred by the engineering genius.

Engineers have made usable nature's vast resources of material and energy for humanity's benefit.

Engineers have vitalized and turned to practical use the principles of science and the means of technology.

Were it not for this heritage of accumulated experience, my efforts would be feeble.

As an engineer, I pledge to practice integrity and fair dealing, tolerance, and respect, and to uphold devotion to the standards and the dignity of my profession, conscious always that my skill carries with it the obligation to serve humanity by making the best use of Earth's precious wealth.

As an engineer, I shall participate in none but honest enterprises.

When needed, my skill and knowledge shall be given without reservation for the public good.

In the performance of duty and in fidelity to my profession, I shall give the utmost.

(this is the second copy of this comment, the first was regrettably lost in a browser crash. Use systems that back up your comments automatically)

This advice seems to fly in the face of Richard Hamming's advice to keep an open door. However perhaps the difference is subtle: Hamming suggested to have an open door but not necessarily to share your secrets, so perhaps there is room for a big science mystery cult to retain its own mysteries at every level of initiation. Perhaps there is a middle ground[1] to be found between this and current 'open science' wherein secrets and ritual are more emphasized, but where the public has the ability to always query deep into the bureaucracy of the science temple/university.

More likely, however the best approach is all of the above, some kinds of thinking are enhanced by a certain size of a team, and there may be some problems that require an open-science sized 'ingroup', and some problems that are more tractable with an ingroup the size of a mystery cult.

• 1 (keep in mind the middle ground is not necessarily the best path -- it's just a path)

We should not also neglect the material side of the scientists status. For example, a junior in a family may be told by his father: "look our neighbor Bob, he spent 30 years of his life on science and look what a junkie car he has and what a small house he lives in. Better have a simple but reliable job, like buying t-shirts in China and selling them here or alike, that will give you a reasonable living." I have seen quite a lot of such justifications in real life and this has to be taken into account because it reflects the factual state of things.

Regarding making the products of scientific knowledge hidden and secret, well.... Eliezer has not made this site's content protected by initiation rituals and it is publicly accessible :P If I had that $100,000 I would put on a grey robe, go to that Electricity Monastery and grant it to Eliezer to become my sensei and teach me about blueness and redness implied from atoms :)

  And this effect is especially strong with information—we're much more likely to try to obtain information that we believe is secret, and to value it more when we do obtain it.

There is an additional benefit to the process - filtering. Today there is so much information, that finding the info. you're looking for can be hard to find. And when the quality of sources varies so much, and can be difficult to judge, that driving a lack of interest does make sense. (As does forcing people to do a thing badly.)

For this world, I might recommend employing the Ikea effect - don't study X, build X/do what sounds fun in that space. Are there limits to what you can build? In this, Empiricism may be the way to go - the impossible hasn't been done yet, and you don't know until you've tried.

Perhaps in a world where calculus hadn't been invented it would be harder to reinvent - I still don't see why derivatives are important as a thing unto themselves, rather than as a special case where h=0. But if you make something, you have not only a better grasp of it, and context, but of where it does and doesn't work - yet. What someone else gives you, you may forget. What you have made once, you can make again - perhaps better this time.

Perhaps making a computer and an OS from scratch and programming languages and reinventing everything would take too long. But just as the journey of a thousand miles begins with a single step, the pursuit of growth that never ends may go far indeed.

I could always walk into my local university library and find all kinds of crazy cool knowledge. Friends at the time were all excited about the Anarchist Cookbook, I kept trying to tell them the really interesting stuff was at the library (a specialized library, but still). Now we have the internet. You only need the books and whatnot if you want a piece of paper that says you know the stuff (I prefer stronger proofs).

I'm pretty pissed about the journals and data being paywalled, and dig the take on that (hey, it's not "public", so it's not exactly verifiable), here's something along those lines:

https://www.currentaffairs.org/2020/08/the-truth-is-paywalled-but-the-lies-are-free

It is all about value and utility though, I think. There are so many facets to why we are where we are when we are. As a fan of paradox, here's some ruminations on one that might kind of fit in here, at least for some mulling over:

https://en.wikipedia.org/wiki/Paradox_of_value

I've got faith we'll get there, eventually, even though we could be "there" already. FWIW, Science is not Truth, but it is pretty useful, unlike raw belief.

How scientific is the idea that there is nothing but science to learn? Or perhaps one must reinvent the wheel to drive a car.

I know I'm really late with this, but what do you consider as "studying science"? Making a career of it? Does being an engineer count (I guess it does)? Or is getting (an amount of knowledge equivalent to) a B.Sc. enough too? Maybe even less than that, learning cool nuggets of science as a hobby? I think this should be better defined. If it's just a career that counts, I'm afraid that the main inhibitor is not interest, but fear for career prospects. Most often when I head people's reasons not to pursue a career in science, it's because they don't think they'll make a good living out of it, or because it's hard and they don't think they'll make it. If it's the hobbyist population you're worried about, I think it's pretty decent, after factoring in access to prerequisite knowledge, free time, and upbringings. Though there is a LOT of room for improvement on that front. Those who actually don't find science interesting seem to think that way mostly because of bad teacher experiences or the social stigma of "nerds", as far as I've seen.

This is essentially a riff on gnosticism. In a sense, there is a streak of that within science as a social institution, at least among a certain adherents of the cult of scientism. The motive is one of power: secret knowledge is attractive because it promises power over others. It is the same perverse attraction that people have to magic and the occult (libido dominandi). It also promises to make you feel special for knowing what others do not. I don't think that's what you want science as a social institution to be about. Science should appear boring to everyone of ill will and only those pure of heart should see its as worth their time. 

Btw, when you say that there is "nothing but science to learn", apart from the strangeness of the phrasing (really, the ultimate aim isn't to know science as such except when it is the object of study itself, but to know reality and science is construed as a means of knowing reality), I sincerely hope that you don't mean science in the narrow sense because then your article is by its own criteria worthless and imparts nothing. The claim itself is not scientific.

One possible problem for this is that people that would have otherwise been interested do not get into science while those who want to know hidden secrets may be learning science just to say that they have them, which isn't really the goal. If the two above instances do not occur, this sounds like a splendid idea, as mystery in and of itself sparks curiosity.

Extremely fun, entertaining and thought provoking read. Here is an example of the "thought provoking" aspects for me.

The proposed approach will not work for me personally and I think for most scientists in our current society.

When something has "cult characteristics" (robes, oxen etc.) I immediately dismiss it as "not worth it". The reason, I think, is that we learned that "cults" usually lack substance or require one to accept everything as axioms.

On the other hand, in the hypothetical society described the "scientific cults" will obviously be taken, very, very seriously! Imagine eighth level physicists demonstrating their power (literally; in J/s; e.g. Castle Bravo).

I think that such a society will have to be very different from ours in many other ways and it is very interesting to try to imagine in what ways.