We agree that an individual can manage their goal better - keeping an eye on the big picture, not ending up with lost purposes - than an organization. But what are you proposing (are you proposing anything?) Should we simplify, get rid of large and complex organizations? Seems that even when you count all the waste of lost purposes, we're more productive with them, at least in many cases ("better off" is another question, but let's leave that aside since it doesn't really bear directly on this issue). Complexity and scale have advantages, as ...
The general point is valid, but in particular cases one should be cautious in concluding than an org has lost it purpose - it might have a different purpose than you realized. School in particular my not have the purpose to teach, but to sort students by abilities.
I don't see how that, even if true, would make what schools are doing look any more sensible. Surely that purpose could be accomplished without wasting entire years of children's lives on sitting in a room doing nothing particularly useful.
In cases of delayed effects, linking pensions to outcomes years later might provide incentives.
For instance, teachers pensions could depend on the eventual earnings of their pupils, or elections could have tickboxes for evaluating the last politician.
(which doesn't help the problem of credit being due to many people)
Great post again!! This seems to be a good portal to the series as well! But my gosh I feel infoleptic!! I feel as though I would need 100 more brains to be able to do and read and learn everything I would like to with 1 brain! And if I would have a 100 brains I would probably feel I would need a 1000! etc. I think it is really sad this blog didn´t have some kind of rating system! There seems to be so many good posts. And if someone totally new to this would come here it would be hard to know where to start. So a list of the most important posts would be ...
As always, well put. However, "Maybe you realized it was all madness, but I bet you did it anyway. You didn't have a choice, right?" Many students do have a terribly mercenary attitude. But a good number do voluntary study topics with no evident financial incentive.
"...I didn't need to know what the words meant, it would work anyway." That was a stupid thing for them to say. However, a common approach in these circumstances, similar but crucially different is "you don't need to know what the words mean now, just read them over an...
Not to get too sidetracked, because your overall point about education is well-taken, but this:
A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation. Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level. I seem to recall (though I can't find the source) that just taking mandatory te...
If they have no interest in knowing the information for its own sake, that sounds like a problem with them, not with the university.
Not everyone might be familiar with the expression "K-5". I had to google for it:
K-5 (pronounced "kay through five") is an American term for the education period from kindergarten to fifth grade.
If you think about this for a while, you'll begin to realize how much of our civilization is based on lost purposes.
Why do men wear ties? Why do we build houses out of the one substance that rots and burns? Why do we type with QWERTY keyboards? Why isn't the department of defense also responsible for homeland security (what does defense mean, if not homeland security)?
There is an interesting bias in the way that we approach these issues. We see ourselves as the prisoners of the system not the jailors. Or maybe I should say that we don't like to admit that we are lords as well as serfs.
One political response to the problem of lost purposes is to focus on government coercion which leads to libertarianism. I don't think that we should rush so quickly to a conclusion. First we should examine the origins of the problem more closely.
The key concept is contestability: is there an alternative BA course you can take that does not require mastery of 12 century knitting patterns. This not lead automatically to free-marketry. One can image contestible socialism.
Think about driving licences. Why does there have to be a single agency that issues them. The government could fund two, splitting the money in proportion to the number of licences issued. If queues are too long at one office, drivers will go to the over. It gets rather interesting if the competing offices get to set different driving tests. Do you go for the easy one or the hard one. Perhaps drivers who have passed the hard one get cheaper insurance rates. Perhaps that also tells you something impor...
As someone who actually practices medieval fighting arts I have to take objection to this:
Miyamoto Musashi said:
"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means."
This is of course a generalization. The reality is more complex. Never forget that the real primary goal of yours is to stay alive and unharmed(at least that's my goal), so sometimes a parry makes sense, even if you can't cut the enemy in the same movement. You could cut or kill the enemy a lot of times if you simply ignore his blade and go straight for the attack/counter-attack, but that would also mean that you would be killed or injured.
Btw you don't want to always cut the enemy. Sometimes your objective will be to disarm him and take him captive without seriously injuring him.
But in GENERAL his advice is correct: defend and attack(cut) in the same movement, that's a good guideline although it's not always possible to do that depending on the actual situation.
Btw generating the entire art from withhin yourself would be kinda hard like trying to generate physics from yourself, you better have a good teacher. Of course you probably mean that you should be able to understand the idea behind each move so if the teacher doesn't explain it you better think about it and try to figure it out for yourself.
MM should be understood in context... the fighting style he talks about is caught up in a whole lot of issues about "honour".
Fights are to the death because they are over the matter of honour. So he doesn't even bother to talk about fights that are intended to dis-arm only. It makes sense (for him) to talk only of the fights that involve "killing the enemy" as the ultimate goal.
What you describe above is actually exactly what he was talking about. If you are thinking two thoughts (eg "I must not die" AND "I must kill the enemy") - the two goals conflict... and in a fight - you kinda only have time for one goal. As somebody who also has a lot of experience in medieval swordcraft, I can attest to that.
A sword comes at you - do you parry or lunge? ... you hesitate, you die.
So, you must pick one goal over the other.
All things being roughly equal, if you pick "I must stay alive" and your enemy picks "I must kill my enemy" - then your enemy will probably win, because he gives no thought to protecting himself (except insofar as it allows him to strike back next time and kill you) and is willing to take more risks in achieving his goal than you are.
I think that was MM's point.
I don't understand your use of the word "versus" in the following sentence:
"The corresponding essential of pragmatic rationality - decision theory, versus probability theory - is to always see..."
versus means: As the alternative to or in contrast with.
I thought that probability theory is an integral part of decision theory, so the "versus" doesn't make sense.
A brilliant post with many links to the Yudkowsky Canon. It has just become a bookmark.
One quip: the study which revealed that a majority of research findings were false seemed to rely on a simulation, and on one meta-study performed earlier by the group. Have I understood this correctly?
Perhaps the p for biological experiments should be lower, but my first inclination is to defend the field I work in and its custom of p<0.05 .
Every time I open up an animal for surgery, the animals nerves are lying in slightly different places. There is a different amount of muscle-tissue obscuring the nerve I have to record from. I have to re-manufacture my electrode after every few surgeries. The hook on my electrode is differently shaped each time. When I put the nerve on the electrode, I pull it up with a different amount of force, in a different position, and do a different degree of damage to the nerve's processes.
Every animal is slightly different, and every surgery goes slightly differently.
Therefore, I think it might be understandable, why I need a p that is higher than a physicists, because if the physicist uses the same stock of materials, he has much less variability in his setup to begin with.
It seems to me there are plenty of 'sane' corporations out there, even very large ones. Eliezer-
Corporations have the singular purpose of making money, and its generally very simply to determine if and how well a given company is doing this. The problems you describe with our education system or Soviet shoe factories apply to a certain extent within a corporation itself, but if a company strays too much from its purpose of making money (starts going 'insane'), this becomes quickly apparent in the form of declining profits. And of course if it can't find...
Stuart Buck, Why is the Barone quote the only vestige of the 93 stuff? What was the jist of that lunch?
As a high school dropout and aspiring philosopher of self-education, I salute you.
My son is homeschooled, and the homeschooling consists of... nothing. He studies spontaneously to solve problems that are authentic to him, just as I do. Mostly these involve video games and online fantasy role-playing games. It's like A.S. Neill's Summerhill school except he's all alone. There are disadvantages to this sort of education, but inauthenticity is not among them.
I am writing a book along these lines. It's about how we can creates ourselves as individual thinkers....
Stuart Buck, Apologies for the trollish behavior. I read your blog for a short while about 6 years back when I found that stuff intriguing, and seemed to recall it being the reason Barone contacted you. It makes perfect sense to erase that stuff (considering who was waving it around), but I found it interesting to find the Barone quote up there as a vestige. I'd love to know what he said about it all.
mtc, you could make a few volumes of Dilbert your required reading, to inform your faith in the 'sane corporation'. In fact, in such high powered company, I'm surprised to see non-qualified discussion of the intentionality of any organisation. Organisations don't have intentionality, individuals do (perhaps... if we don't let the Evolutionary Psychology crowd take it away from us..). As all Eliezer's anecdotes illustrate, these get lost in the wash of large numbers and multiple levels, networks, and gridlocks, so we are left with the dross of the lat...
Robin Hanson: School in particular my not have the purpose to teach, but to sort students by abilities.
Excusez moi, but what exactly is that supposed to mean? As far as I know, most principals want ALL of their students to go to Ivy League and make millions. I can't make any sense of your comment.
It means that contrary to his protestations, Robin really is a libertarian, and he is trying to push the meme that libertarian positions are really vanilla mainstream positions. Someone who is doing the exact opposite of overcoming bias, iow.
Tibba, when you are in a bar, do you see an attractive person and say to yourself, "I think I'll initiate sexualized body language, so that I can mate with that person, thereby increasing the frequency of my genes in future generations"?
There is another post addressing your incorrect objection, but I can't remember what it's called, maybe someone else can dig it up.
At Mt. Obaku temple in the Ko district, Yudkowksy-Sensei was approached by an Individualist during morning meditation.
"All of us are ultimately selfish; we care only about our own states of mind. The mother who claims to care about her son's welfare, really wants to believe that her son is doing well - this belief is what makes the mother happy. She helps him for the sake of her own happiness, not his." said the Individualist.
Yudkowsky-Sensei remained sitting on his zafu, and said nothing.
The Individualist continued: "She did it because she valued that choice above others - because of the feeling of importance she attached to that decision."
Yudkowksy-Sensei stood up and walked out of the temple to his car, where he proceeded to open and close the driver-side door several times before saying:
"There is no chocolate at the supermarket."
"Tibba, when you are in a bar, do you see an attractive person and say to yourself, "I think I'll initiate sexualized body language, so that I can mate with that person, thereby increasing the frequency of my genes in future generations"?"
Of course not. I want to get laid. That's my explicit desire. Promoting my genes is just a side effect of this. And what is the explicit desire of a principal? Perhaps to teach children. Perhaps to go to work, get paid and ensure that nobody sues him. (He'd still like it if they all go to Harvard.) But...
You don't have to have complex human organizations to get lost purposes. Runaway sexual selection is basically a subgoal of evolution (finding a good mate) taking over far too many resources from the primary goal (survival and reproduction). Of course, since this is probably the process that generated the human mind in the first place, we shouldn't judge the phenomenon too harshly. Our minds and culture are founded on distraction, which explains a lot, like why I am commenting here instead of working.
As for human organizations, I've always felt that, give...
It's not okay for organizations, corporations, or governments to be misdirected and focused on surface matters; but it's not uncommon, either. Musashi was unique for his insight -- with little training, he defeated masters from an early age, all because of clear perception. (He was later defeated by a man who, in a dream, saw that the jo staff (arm pit height staff) would allow him to defeat any man in Japan.) Perception like that is exceptional. A clear sense of purpose is a gift.
How does one cultivate this? The constantly feel the sharp distinction between success and failure -- as Musashi must have -- is one way. Are lives are gentler than that; I'm not sure what we moderns can do. Education does not seem to help.
mtc, you could make a few volumes of Dilbert your required reading The Logical Fallacy of Generalization from Fictional Evidence Better evidence for dysfunctional corporate governance is here and here.
It means that contrary to his protestations, Robin really is a libertarian, and he is trying to push the meme that libertarian positions are really vanilla mainstream positions. Someone who is doing the exact opposite of overcoming bias, iow. Robin Hanson has contrarian rather than vanilla mainstream positions, and is quite explicit about that. When he mentio...
Jason: AFAIK it's only Shindo Muso Ryu practitioners who claim that Muso Gonnosuke defeated Miyamoto Musashi. Is there any other documentation supporting this?
Excusez moi, but what exactly is that supposed to mean? As far as I know, most principals want ALL of their students to go to Ivy League and make millions. I can't make any sense of your comment.
Dilbert isn't quite an example of "The Logical Fallacy of Generalization from Fictional Evidence": Scott Adams, in his non-fiction books, repeatedly mentions that no matter how crazy he makes his Dilbert strips, people keep telling him "That's just like my job." In other words, Dilbert is a documentary. ;)
Also, there's this book. If you want to know how corporations are actually run, you need to read it.
people keep telling him "That's just like my job." In other words, Dilbert is a documentary. ;)
Or a horoscope...?
Back on topic (sort of)... Some of the discussion here seems to highlight to problem of resolving dual and conflicting purposes. One key purpose of schools is to educate people; in a very real sense, though, they also necessarily serve to sort those of differing abilities (whether this is an overt purpose, pace Tiiba, is irrelevant: it is a function which schools are clearly set up to facilitate, and so is a de facto purpose). These tw...
Tiiba,
I think Robin may have meant by "purpose" role in society rather than explicit intention of the principal. (In other words, the signaling is why employers continue to value education, and why people keep going to school.)
Similarly, even if you don't, for instance, go to a bar with the explicit purpose of propagating your genes, you are only naturally inclined to get laid because it's an evolutionarily fit behavior. This doesn't mean you can't explicitly hold another purpose in your mind.
I have seen one or two occasions when he has denied it. But I give more weight to actions, and I have not seen anything he has posted which runs contrary to that assumption. If he truly is a contrarian, you'd expect to see more posts arguing for points that are more commonly associated with socialism or communism. I haven't seen them.
To clarify, yes, by "purpose" I meant social function, not conscious intention, and yes I meant private schools as well as public.
A computer company won't hire you, no matter how much you know about 12th century knitting patterns, though. So even employers, AFAIK, value actual knowledge. And I don't think the purpose of a school should be determined from the point of view of companies. It should be seen according to those who actually determined whether it will be there - the senator who wrote it into the water bill, exc.
some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment ...
@Tiiba:
The choices that create a school are not the only choices responsible for the schooling that goes on. Students choose to attend, or others choose for them (parents, gov't, etc.). They do so in response to incentives created by others (businesses, higher educational institutions, etc.).
And of course employers value some kinds of knowledge. But knowledge and schooling are not the same thing. At least, people behave w/ regard to school in ways I can't imagine their behaving if it were just about knowledge (e.g. caring about grades, fulfilling major...
Part of the thinking behind NCLB and similar policies is the focus on measurability/transactionalization. One of underlying ideas is: if something can't be easily modeled/measured, it's not to be considered. The corollary to that is that things that are easily measured (test scores, in this case) are piled on, sometimes thoughtlessly.
http://www.huffingtonpost.com/charles-h-green/software-programming-and-_b_69287.html
is a brief, but enlightening article on the subject.
mtc has it right in that the difference between private organisations and state (and state-funded) ones is that private organisations have an in-built feedback mechanism, they exist to make a profit, the profit they make is calculable by the owners, who will either keep their organisation profitable ie true to its purpose, or sell out or go bust. State organisations by contrast may not have a clear purpose, often find that purpose diverted or confused by politicians, have no feedback mechanism, and almost never go out of existence for any reason including gross failure.
or sell out or go bust
...or, if their function is sufficiently essential, get bailed out by the state.
"..classroom time is now spent on preparing for tests..."
Sounds a bit like learning.
NCLB is just a way for the Feds to see if they are getting their moneys worth. They spend a lot per student and should know if the student is learning anything. It costs the school one day a year for the test. Everything on the test is what the student is supposed to be learning anyway.
Believe me, preparing for a standardized test is VERY DIFFERENT from actually learning and understanding material.
Depends who you are, doesn't it? I think a lot of us privileged folk shudder at the idea of "teaching to the test," because we went to schools were everyone was so far above the standard that this would have been dull and stunting. But NCLB doesn't exist to help high-proficiency students, and I imagine it has next to no effect on them. The classrooms that need to "teach to the test" are the ones that have trouble passing it, and they generally consist of students whose main problem is that they can't read or can't do basic arithmetic...
I think a lot of us privileged folk shudder at the idea of "teaching to the test," because we went to schools were everyone was so far above the standard that this would have been dull and stunting.
Um... that seems rather implausible. I went to perfectly normal and average schools, and they were already dull and stunting without "teaching to the test".
You just written a lot of words, it could be better to write a substract or bolden your main sentences
Caledonian, if I understand Holden correctly, he's just arguing that many people here may have had an untypical educational experience.
Both my parents are veteran educators (teachers, then administrators) in a tough urban school district in the Midwest, and they both laud No Child Left Behind not because it helps children directly, but because it's the only "objective" information available that can be politically wielded to attack known underperforming schools and employees. Administrators are often too well aware how terrible their schools, teachers, and students are, and NCLB testing doesn't tell them anything that they didn't already know. Instead, the tests are acceptable &...
Why would any individual lose track of their purposes in a swordfight? If someone else had taught them to fight, if they had not generated the entire art from within themselves, they might not understand the reason for parrying at one moment, or springing at another moment; they might not realize when the rules had exceptions, fail to see the times when the usual method won't cut through.
On August 26, 1346, a smaller contingent of English soldiers met a numerically superior French force outside of the Norman village of Crecy. The outnumbered English focused primarily on defense, using mainly dismounted spear- and longbowmen, whereas the French employed a highly mobile, offensive strategy utilizing mainly armored horsemen; the result was a decisive victory for the English, and a complete rout for the French. 10 years later, English and French forces would meet again at Poitiers; the French still had the numerical advantage, but their strategies were reversed: thistime, the French fielded dismount archers, while English charged in on horseback. The English still won, and even managed to capture the French king, forcing him to sign over a sizable chunk of his kingdom as ransom. W...
Adhocracies seem specifically designed to avoid this problem at an institutional level.
The word is a portmanteau of the Latin ad hoc, meaning "for the purpose", and the suffix -cracy, from the ancient Greek kratein (κρατεῖν), meaning "to govern",
With characteristics such as those described in the Wikipedia article I linked, is it possible to institutionalize avoidance of institutional incompetence, such as what we described here?
When I was very young I was also very curious, but I sated my curiosity by telling myself that I would know it all when I was grown up. It wasn't my problem to be curious about these things, other people were handling them. Maybe Eliezer's classmates were thinking that this didn't really make sense, but they trusted the adults enough to do it anyways. Like how a lit lightbulb turns less mysterious and wonderful when you know that someone else already knows the scientific reason for it. All you need to do is follow the instructions and you'll get light.
"I wish I lived in an era where I could just tell my readers they have to thoroughly research something, without giving insult."
Is that not what this entire site is accomplishing?
"More than anything, you must be thinking of carrying your movement through to cutting him. You must thoroughly research this." (Emphasis added.)
I wish I could upvote this post multiple times. Especially the "You must thoroughly research this" part, although also all the examples of institutions failing to accomplish much of anything, even when you might naively have really thought they would (e.g., individual health procedures work; how could large-scale institutions for doing those procedures not work?). I read this post back in 20...
Interesting. It seems your early Jewish teachers were firmly stuck in the mindset that was one of the things Christianity originally arose to combat. (Not that Christians should feel too haughty about that since the vast majority of Christian sects fell right back into the same trap early on. I find it amusing that "The Lord's Prayer" is repeated by-rote in most sects when in-context it was given as an example of how you shouldn't waste your time with by-rote prayers...)
This is as opposed to the Jewish community in and around Salt La...
"Likewise with people not automatically noticing when supposedly selfish people give altruistic arguments in favor of selfishness, or when supposedly altruistic people give selfish arguments in favor of altruism" - maybe it's because they want to encompass everyone? So the altruistic guy thinks "other altruistic guys like myself are not to be encourage to be altruistic but I need to appeal somehow to those who are currently selfish", and vice versa?
It was in either kindergarten or first grade that I was first asked to pray, given a transliteration of a Hebrew prayer. I asked what the words meant. I was told that so long as I prayed in Hebrew, I didn't need to know what the words meant, it would work anyway.
That was the beginning of my break with Judaism.
As you read this, some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper.
Maybe you realized it was all madness, but I bet you did it anyway. You didn't have a choice, right?
A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation. Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level. I seem to recall (though I can't find the source) that just taking mandatory tests was 40% of classroom time in one school.
The old Soviet bureaucracy was famous for being more interested in appearances than reality. One shoe factory overfulfilled its quota by producing lots of tiny shoes. Another shoe factory reported cut but unassembled leather as a "shoe". The superior bureaucrats weren't interested in looking too hard, because they also wanted to report quota overfulfillments. All this was a great help to the comrades freezing their feet off.
It is now being suggested in several sources that an actual majority of published findings in medicine, though "statistically significant with p<0.05", are untrue. But so long as p<0.05 remains the threshold for publication, why should anyone hold themselves to higher standards, when that requires bigger research grants for larger experimental groups, and decreases the likelihood of getting a publication? Everyone knows that the whole point of science is to publish lots of papers, just as the whole point of a university is to print certain pieces of parchment, and the whole point of a school is to pass the mandatory tests that guarantee the annual budget. You don't get to set the rules of the game, and if you try to play by different rules, you'll just lose.
(Though for some reason, physics journals require a threshold of p<0.0001. It's as if they conceive of some other purpose to their existence than publishing physics papers.)
There's chocolate at the supermarket, and you can get to the supermarket by driving, and driving requires that you be in the car, which means opening your car door, which needs keys. If you find there's no chocolate at the supermarket, you won't stand around opening and slamming your car door because the car door still needs opening. I rarely notice people losing track of plans they devised themselves.
It's another matter when incentives must flow through large organizations - or worse, many different organizations and interest groups, some of them governmental. Then you see behaviors that would mark literal insanity, if they were born from a single mind. Someone gets paid every time they open a car door, because that's what's measurable; and this person doesn't care whether the driver ever gets paid for arriving at the supermarket, let alone whether the buyer purchases the chocolate, or whether the eater is happy or starving.
From a Bayesian perspective, subgoals are epiphenomena of conditional probability functions. There is no expected utility without utility. How silly would it be to think that instrumental value could take on a mathematical life of its own, leaving terminal value in the dust? It's not sane by decision-theoretical criteria of sanity.
But consider the No Child Left Behind Act. The politicians want to look like they're doing something about educational difficulties; the politicians have to look busy to voters this year, not fifteen years later when the kids are looking for jobs. The politicians are not the consumers of education. The bureaucrats have to show progress, which means that they're only interested in progress that can be measured this year. They aren't the ones who'll end up ignorant of science. The publishers who commission textbooks, and the committees that purchase textbooks, don't sit in the classrooms bored out of their skulls.
The actual consumers of knowledge are the children - who can't pay, can't vote, can't sit on the committees. Their parents care for them, but don't sit in the classes themselves; they can only hold politicians responsible according to surface images of "tough on education". Politicians are too busy being re-elected to study all the data themselves; they have to rely on surface images of bureaucrats being busy and commissioning studies - it may not work to help any children, but it works to let politicians appear caring. Bureaucrats don't expect to use textbooks themselves, so they don't care if the textbooks are hideous to read, so long as the process by which they are purchased looks good on the surface. The textbook publishers have no motive to produce bad textbooks, but they know that the textbook purchasing committee will be comparing textbooks based on how many different subjects they cover, and that the fourth-grade purchasing committee isn't coordinated with the third-grade purchasing committee, so they cram as many subjects into one textbook as possible. Teachers won't get through a fourth of the textbook before the end of the year, and then the next year's teacher will start over. Teachers might complain, but they aren't the decision-makers, and ultimately, it's not their future on the line, which puts sharp bounds on how much effort they'll spend on unpaid altruism...
It's amazing, when you look at it that way - consider all the lost information and lost incentives - that anything at all remains of the original purpose, gaining knowledge. Though many educational systems seem to be currently in the process of collapsing into a state not much better than nothing.
Want to see the problem really solved? Make the politicians go to school.
A single human mind can track a probabilistic expectation of utility as it flows through the conditional chances of a dozen intermediate events - including nonlocal dependencies, places where the expected utility of opening the car door depends on whether there's chocolate in the supermarket. But organizations can only reward today what is measurable today, what can be written into legal contract today, and this means measuring intermediate events rather than their distant consequences. These intermediate measures, in turn, are leaky generalizations - often very leaky. Bureaucrats are untrustworthy genies, for they do not share the values of the wisher.
Miyamoto Musashi said:
(I wish I lived in an era where I could just tell my readers they have to thoroughly research something, without giving insult.)
Why would any individual lose track of their purposes in a swordfight? If someone else had taught them to fight, if they had not generated the entire art from within themselves, they might not understand the reason for parrying at one moment, or springing at another moment; they might not realize when the rules had exceptions, fail to see the times when the usual method won't cut through.
The essential thing in the art of epistemic rationality is to understand how every rule is cutting through to the truth in the same movement. The corresponding essential of pragmatic rationality - decision theory, versus probability theory - is to always see how every expected utility cuts through to utility. You must thoroughly research this.
C. J. Cherryh said:
I have seen many people go astray when they wish to the genie of an imagined AI, dreaming up wish after wish that seems good to them, sometimes with many patches and sometimes without even that pretense of caution. And they don't jump to the meta-level. They don't instinctively look-to-purpose, the instinct that started me down the track to atheism at the age of five. They do not ask, as I reflexively ask, "Why do I think this wish is a good idea? Will the genie judge likewise?" They don't see the source of their judgment, hovering behind the judgment as its generator. They lose track of the ball; they know the ball bounced, but they don't instinctively look back to see where it bounced from - the criterion that generated their judgments.
Likewise with people not automatically noticing when supposedly selfish people give altruistic arguments in favor of selfishness, or when supposedly altruistic people give selfish arguments in favor of altruism.
People can handle goal-tracking for driving to the supermarket just fine, when it's all inside their own heads, and no genies or bureaucracies or philosophies are involved. The trouble is that real civilization is immensely more complicated than this. Dozens of organizations, and dozens of years, intervene between the child suffering in the classroom, and the new-minted college graduate not being very good at their job. (But will the interviewer or manager notice, if the college graduate is good at looking busy?) With every new link that intervenes between the action and its consequence, intention has one more chance to go astray. With every intervening link, information is lost, incentive is lost. And this bothers most people a lot less than it bothers me, or why were all my classmates willing to say prayers without knowing what they meant? They didn't feel the same instinct to look-to-the-generator.
Can people learn to keep their eye on the ball? To keep their intention from going astray? To never spring or strike or touch, without knowing the higher goal they will complete in the same movement? People do often want to do their jobs, all else being equal. Can there be such a thing as a sane corporation? A sane civilization, even? That's only a distant dream, but it's what I've been getting at with all these blog posts on the flow of intentions (aka expected utility, aka instrumental value) without losing purpose (aka utility, aka terminal value). Can people learn to feel the flow of parent goals and child goals? To know consciously, as well as implicitly, the distinction between expected utility and utility?
Do you care about threats to your civilization? The worst metathreat to complex civilization is its own complexity, for that complication leads to the loss of many purposes.
I look back, and I see that more than anything, my life has been driven by an exceptionally strong abhorrence to lost purposes. I hope it can be transformed to a learnable skill.