Just Try It: Quantity Trumps Quality

by atucker1 min read4th Apr 201183 comments


Skill BuildingPractical

Followup to: Don't Fear Failure

In the same theme as the last article, I think that failure is actually pretty important in learning. Rationality needs data, and trying is a good source of it.

When you're trying to do something new, you probably won't be able to do it right the first time. Even if you obsess over it. Jeff Atwood is a programmer who says Quantity Always Trumps Quality

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the "quantity" group: fifty pound of pots rated an "A", forty pounds a "B", and so on. Those being graded on "quality", however, needed to produce only one pot - albeit a perfect one - to get an "A".

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the "quantity" group was busily churning out piles of work - and learning from their mistakes - the "quality" group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

Where have I heard this before?

  1. Stop theorizing.
  2. Write lots of software.
  3. Learn from your mistakes.

Quantity always trumps quality. 

When it comes to software, the same rule applies. If you aren't building, you aren't learning. Rather than agonizing over whether you're building the right thing, just build it. And if that one doesn't work, keep building until you get one that does.

The people who tried more did better, even though they failed more too. Of course you shouldn't try to fail, but you shouldn't let the fear of it stop you from tyring.

I wouldn't go as far as to say that quantity always trumps quality, but where the cost of failure is low lots of failures that you pay attention to is a pretty good way of learning. You should  hold off on proposing solutions, but you also need to get around to actually trying the proposed solution.

I'm normed such that I'll spend more time talking about if something will work than trying it out to see if it works. The problem is that if you don't know about something already, your thoughts about what will work aren't going to be particularly accurate. Trying something will very conclusively demonstrate if something works or not.

I originally had this as part of Don't Fear Failure, but that post got too long.


83 comments, sorted by Highlighting new comments since Today at 4:01 PM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Long ago, I forget where, I saw a blog post that applied this to writing. It pointed out that if we model the quality of your writing as having a mean X and variance Y, then the only way to hit those unlikely exceptionally good texts is to write a lot. Yes, while doing so you might also come up with the same number of exceptionally bad texts, but nobody forces you to show those to anyone. Plus writing a lot will give you practice, gradually pushing up the mean.

From personal experience, I'd also err on the side of publishing even texts you're not personally all that impressed by. I've noticed that I'm relatively bad at estimating what's going to be popular. Some of my biggest hits have been blog posts I'd never have thought would be popular.

Writing massive amounts of text also helps with self-estimation as a person who can write arbitrary amounts of text. My evolution as a writer started with me going "meh, I can't finish anything, I have all these ideas that I sometimes start but then I lose interest after a few pages". Then I started writing collaboratively with a friend, which was so much fun that I could say, "I like writing, at least of this kind, so much that I was on Utah time in Scotland to stay up and write more, for as much as seventeen hours straight." Then I wrote a finished novel... it was fanfiction, but I already considered worldbuilding my strength and character creation also on said list. Then I did it again. At some point I started being "a writer", who can decide to do things like "write a book" and have books exist as a result of this decision.

8rhollerith_dot_com10yMaybe so, but I'm not going to keep watching someone's blog or eir user page here unless eir average quality is quite high.
7Zvi10yThere's a difference between average quality produced and average quality published. Ideally you sit on the stuff that isn't any good.
1rhollerith_dot_com10yWe are talking about writing. Do you really think that most writers who need to improve know which of their writings isn't any good?
4Zvi10yI have been an aspiring writer of sorts, and wrote articles at least once a week for several years without getting much if any feedback on the quality of my writing as opposed to its content. It is fairly easy for me to look back and see a steady improvement in writing quality. I also usually (not always) have no trouble knowing which of my writing isn't any good, and don't remember it having been otherwise. I could be deluding myself but I certainly think some of my writing is better and some of it is worse.
0rhollerith_dot_com10yI see. Thanks.
1Zetetic10yWell, I'm not an aspiring writer but as an amateur musician and visual artist I can say that I can generally tell when my works are not as good as I would like them to be, and I can generally guess ahead of time which pieces will have a better reception among my more critical friends. In addition; I know a couple of aspiring writers and if anything I would say that they are often very self critical and judging by what they have openly shown me against what they have reluctantly shown me I would say that they did indeed have a good sense of what was good and what wasn't. Based on my experiences, I would say that it is fairly common for artists to have a fairly accurate awareness of their own shortcomings; whether or not they can successfully ascertain a workable procedure for overcoming them is a different issue.
0[anonymous]10yWell, I'm not an aspiring writer, but as an amateur musician and visual artist I can say that I can generally tell when my works are not as good as I would like them to be and I can generally guess ahead of time which pieces will have a better reception among my more critical friends. In addition; I know a couple of aspiring writers and if anything I would say that they are often very self critical and judging by what they have openly shown me against what they have reluctantly shown me I would say that they did indeed have a good sense of what was good and what wasn't. Based on my experiences, I would say that it is fairly common for artists to have a fairly accurate awareness of their own short comings; whether or not they can successfully ascertain a workable procedure for overcoming them is a different issue.
3Kaj_Sotala10yIf one's average quality is low enough that people don't find it worth becoming regular readers, then one is probably better off practicing a lot anyway.
0sark10yYes but they assess your blog mostly on its most recent posts. So you should just be out with it and improve anyway. This way you'll always have the best audience your skills can currently get you.
5rhollerith_dot_com10yHmm. Almost all the blogs I continue to follow update infrequently. I always assumed that that was because the bloggers had some way of telling which of their posts or posts-in-planning are really good and had a policy of only posting those, rather than posting every writing exercise they undertake.
6sark10yHmm, also, writing can be an excuse for generating ideas! The obvious thing to do would be to wait for good ideas to come into your head, then write them up for the world to see. But in my experience, writing and having my writing read boosts my ego, which somehow encourages my subconscious to throw up ideas which can be written up to derive yet more ego boosts. It's a virtuous cycle. Which makes not writing because you can't think up any good ideas a vicious one.
3Kaj_Sotala10yThis is very true. Not only does writing help me come up with ideas of unrelated posts, writing a post often gives me ideas for further related posts as I think about the issue more. I've seen at least one professional writer refer to creativity as a muscle - it gets stronger the more you use it. This seems right in my experience.
1Zetetic10yI strongly agree with this sentiment. I keep a folder for this very purpose; whenever an interesting thought comes to mind I type it up along with as many of the related strands of thought as I can as quickly as possible and then save it to the folder and move on. I've found that this is a fairly useful procedure for organizing my thoughts and documenting my progress in my various areas of interest.
0Mark_Neznansky10yWouldn't doing that (instead of writing up the whole argument in a full text) make you feel as if you've already achieved the materialization of the idea, hence reducing your motivation to write it in the future (which might lead to never actually writing the text)?
1Zetetic10yI'm only talking about rough sketches, very short, maybe three or four paragraphs. The material in and of itself is not something I would even want use in the future. Think of it as an artist's sketch pad; I practice (1) the purely technical aspect of my writing (2) my ability to quickly convey ideas (3) I consolidate information that was previously floating about in my skull into a nice package by anchoring it to a single event. It's much the same with guitar or visual art (a least for myself): I may work creatively on one technique by using it to write some nice riffs on guitar or I may try to consolidate the technique of pointillism into my repertoire by using it to draw a face or a land scape. The outcomes of each of these mini-studies is not an end in itself, but rather a stepping stone to mastery.
3taryneast10yHave you tried NaNoWriMo [http://nanowrimo.org] ? It's really good for putting this into practice :)
3Kaj_Sotala10yWon it twice, though those "stories" were pretty solid garbage and I suspect it taught me a bunch of bad habits. It was good for showing that I can do things if I put in enough effort in them, though.
2taryneast10yYeah - but the idea isn't to write something brilliant - it's to get into the practice of writing every day. If you won - then you did that perfectly :) Also - if you really are interested in taking it further, read Steven King's "On writing" [http://www.amazon.com/gp/product/0743455967/ref=as_li_ss_tl?ie=UTF8&tag=rubycolouredg-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0743455967] in which he points out that every first draft is terrible. You make a real book after the first draft is over. In fact - if you haven't already, you night be interested in picking up the NaNoWriMo handbook "No plot, No problem!" [https://store.lettersandlight.org/merchandise/no-plot-no-problem-book-autographed] - all made a lot more sense after I read that.
3Kaj_Sotala10yThanks. Maybe I should look into those. Though it should be noted that while "every first draft is terrible" may be correct for Stephen King, it's not necessarily correct for everyone. There are writers who only do minor revisions to their first draft, while others do several drafts before it gets great.
3taryneast10yThat's true - though I think it'd be safe to say that "every first draft is terrible" is something you could say about the vast majority of writers... for sure there are mozarts in the world of writing - but I'd be very surprised if there were many.
1Kaj_Sotala10yMost of my short stories [http://www.xuenay.net/stories.html] tend to be first drafts, not counting minor edits like changing individual words or making occasional refinements to sentence structure. But then my short stories really are pretty short, so I don't know to what degree this will generalize to novels yet. As for my non-fiction books, the process has usually been such that the concept of a first draft isn't really valid. I don't write them straight through and then refine, instead I'll write parts of one chapter and then another in non-linear order, then revise some of what I've already written, toss out parts of the book to be replaced with something better, and so forth. By the time I finally have a "first draft" of the whole book, large parts of the content have already been edited several times.
0taryneast10yFrom my (limited) experience, novels are very different to short stories. You can hold the whole concept of a short story in your head at one time.. but novels are big and slippery. Things change and develop as you write - so you often have to go back and rewrite or even delete huge swathes of things to fit the new pattern and flow. There are some people who can do without this - but IMO they're either brilliant writing geniuses (to whom we should not compare ourselves) or they're incredibly experienced writers who started out by going through the major-overhaul process... but who have now honed their talent so much they no longer need it... leaving us still in need, generally, of rewriting. This is not to say that maybe you're an exception - or maybe you're much better at planning (and sticking to the plan) than me :)
0tristanhaze9yCome to think of it, "every book is terrible" may also be correct for Steven King.
1Eliezer Yudkowsky10yBut that's just wrong. If you're doing it right your mean creeps steadily upward and that's how you hit high points.
3wedrifid10yNot all that wrong it would seem.
-1Eliezer Yudkowsky10yThe "plus" is right, the main idea is wrong.

Personally I would have put the main idea as the 'plus'. Perhaps overstated but clearly not wrong.

If the quality of works is distributed around a mean then more works you produce the more likely it is for a high quality work to emerge. The most remarkable works will come from the very best authors when they are having a really good day (or month or year). Producing more from the same distribution will obviously give more chances for you to produce something that is outstanding.

On a related note a 'one hit wonder' can be said to be regressing to his mean when his other works flop.

Not 'just wrong'. It's just obvious and less important overall than the training effect.

[-][anonymous]10y 20

i agree with the spirit of this post but I think you're leaving out an important part. It seems from the other comments that the experiment described never really happened. I think if it were tried it wouldn't really work out as described. I know if i had been in that class and been in the quantity group, I would have made some really crappy, really thick pots and been done as quickly as possible in order to goof off for the maximum amount of time. If I had been in the quality group I wouldn't have theorized about it, I just would have iterated on a lump of clay. Making a pot and then not firing it or anything, just mashing it back into a lump and starting over until I got to a really good one. I think I would have learned a lot more about pot making in the quality group.

What I have read and also experienced is that producing quantity is necessary but not sufficient for producing quality. If you want to get really good at something, rather than just getting somewhat good and then plateauing, you have to not only do it a lot, but you have to care deeply about how good you are doing, identify your weaknesses and work specifically to improve those. The problem with your story is that... (read more)

The problem with your story is that the quantity kids have no incentive to produce quality, so they probably just won't.

No incentive? Don't you think they signed up for pottery class to, you know, learn how to do good pottery? That nobody wanted to be proud of their work?

(Btw, I heard this pottery story from a different source, and IIRC it was an adult pottery class, not a kids' one.)

8drethelin10yI would say the majority of classes are signed up for because they're easy or part of required credits for a program.
1ameriver10yI'm not sure that's true once you limit it to adult classes (far more likely to be taking the occasional class for fun), and particularly in the case of an art class.
0Mark_Neznansky10yA "class for fun" implies that grade shouldn't matter to the participants, so, allegedly, the two different grading schemes wouldn't affect the participants' behavior. But things (such as motivation) change as a person who did pottery for fun at home, goes to do pottery for fun in a class, don't they?
2[anonymous]10yTen years or ten thousand hours of "deliberate practice" - PDF link [http://graphics8.nytimes.com/images/blogs/freakonomics/pdf/DeliberatePractice\(PsychologicalReview\] .pdf) - is what's typically talked about these days to become one of the greats. Yeah, I know - ten years??? But "deliberate practice" of any length is better than messing around and will get you good, just not world champ level, long before ten years. That sounds like what you're talking about.
0NancyLebovitz10yA different angle from Kenny Werner, though this is about music-- his Effortless Mastery [http://www.amazon.com/s/ref=nb_sb_ss_i_0_18?url=search-alias%3Dstripbooks&field-keywords=effortless+mastery&x=0&y=0&sprefix=effortless+mastery] is about learning to be calm and non-judgmental, then gradually adding more challenge (achieve state, approach instrument, touch instrument, pick up instrument, make sound without aiming for good sound, then slowly add work on getting skillful without disrupting meditative state). I haven't seen this approach applied to purely cognitive work like programming, though this doesn't seem impossible.
2TheOtherDave10yThis is also more or less the approach I adopted to physical therapy, after my stroke, to pretty good results.

This story is about rapid iteration rather than quantity. The "quantity" is the detritus of evolution created while learning to produce a perfect pot. If a machine was producing pots it would generate great quantity but the quality would not vary from one iteration to the next.

There are many stories and heuristics in engineering lore suggesting rapid iteration converges on quality faster than careful design. See also: OODA loops, the equivalent military heuristic.

2atucker10yThat's true. Originally, this post was part of "Don't Fear Failure". I intended for it to talk about low-cost failures, how practicing helps, and then do a bit of talking about how rationalists should be able to pay mindful attention to their mistakes in order to learn from them before they get right back up and try again. So basically advocating rapid iteration after desensitizing people to failing. However, I wasn't quite able to tie it all together and it just felt like it dragged on. So instead I split it up into a post which says that failing isn't that bad, and another about how practice pays off. I could follow up with another post which more clearly spells out rapid iteration, but that might be a bit much. I'd rather move on to talking about perfectionism and unduly favoring the status quo. Could be wrong though.

I'm brand new to Less Wrong, and very pleased that I found a topic right away that I have given a great deal of thought to already, since it's affected me throughout my life. I grew up with a mother who was constantly critical, and stingy or withholding of praise, with the result that my sister and I, who are both in our late 40s, still converse about the negative affect that my mother had on us when it comes to making mistakes, and attempting to do new things.

I used to feel that I was being scolded because I didn't know something I "ought to have known" in advance. I'm not referring to breaking some established rule in the family. I'm talking about being blindsided by sudden harsh words in a loud volume about something I had never heard about or considered before, something that I had had the nerve to "get wrong." This happened often enough that I began thinking that I had better not try to do things unless I knew EVERYTHING there was to know BEFORE I took any action. This, not surprisingly, had the affect of paralyzing me into inaction, fearing the reprisals for "mistakes", and of course, the judgment about what was right and what was wrong was base... (read more)

Fear of failure is a big problem in my life right now. Its why I don't have a job, since I'm silly and am afraid of being rejected. This reframed something I think I already knew, but I'm sure it will help anyway. Time to really get on to things now.

7Kaj_Sotala10yI found that looking for jobs got a lot easier when I stopped thinking of it as a process evaluating my worth for the employer. Instead I started thinking of it as a process where I look for the employer who deserves me by virtue of realizing just how valuable I am.
1NancyLebovitz10yI've also seen a recommendation that authors think of themselves as displaying their work to editors rather than submitting it.
1Chala10yIt is stil frustrating to be ignored for a position which you would be more than adequate for, and which you are confident that you would be harder working at and more dilligent in than the hired help. I guess thats one of the things that bothers me, having to jump through arbitrary hoops in a pointless process that fails to relate to reality. Also I probably just don't need/want a job that much ;)
1Kaj_Sotala10yIn that situation, I try to just shrug and think it's their loss. :-)
2teageegeepea10yWhat's your work/educational background? I delayed looking for work in the past because I didn't actually need the money. I do have more expenses now, but not enough that I feel ambition for anything better (even within my company). I'm kind of okay with that, but I'd rather not slip up enough to lose this pretty decent gig and have to find another one. In case that reminds anyone of the motivation in Office Space, I am indeed a programmer like everyone else on the internet. I think fear of uncomfortable interactions applies more in regular social situations for me. I've started practicing acting extroverted by just talking to strangers on the street or wherever, confident that I'll never see them again and there are no consequences of bad impressions I might make. Sometimes it results in talking too fast or unclearly though (that also happens at work).
1Chala10yI have a degree in biomedical science, aka a totally useless degree. I'm going back to uni and am just looking for part time work in the mean time to make my time - since I'm finally at the stage of actually really needing the money. Also I'm going back to uni to do computer science and become a programmer. I guess everyone on the internets realise is in that profession ;). Regarding social interactions, I am actually (and unusually for less wrong) a very social and extroverted person. I spend much of my time at the moment socialising - ergo my need for money. Socialising is expensive. However I am also one who has be conditioned to not try, my upbringing was such that any failure was focused on and any triumph taken for granted. Which is no excuse, but still I tend to avoid situations where I am tested at least when I am not guaranteed to triumph (e.g. I am totally 100% ok in academic assessments).
1mutterc10yJob-hunting fits very well with the model in "Don't Fear Failure": the downside risk is zero. The worst case is accepting a bad job. Assuming you're a USian, jobs are at-will, so just leave then, and you're no worse off. As a job-hunter, I've learned to model the probability of getting any one job as infinitesimal, so I don't get too hung up on any one application. Let them do the rejecting.
1Gray10yIs this true? I was always told that employers look down on a spotty employment history; they are less likely to hire someone whose job history is littered with jobs that have been held for less than a year.
6mutterc10yYes. (Their worst-case scenario: You're a "professional plaintiff" who hires on, sues for something or other, gets a (confidential) settlement, and moves on). They also look down on being in the same job a long time (assumption: lack of motivation to advance, etc.). And they look down on gaps in employment (assumption: you were in prison). To summarize the summary of the summary, HR reps hate people.
1NancyLebovitz10yI've heard people claim that leaving a job in less than a year looks very bad on the resume. True?
0suecochran10yI have heard that many times over the course of my adult working life. I tend to agree with it mostly, although I doubt that it applies equally to all types of work, and it may have been more true in the past than it is in today's economy and with today's technology. I would think that it could vary wildly between say a position such as "Office Manager" and that of "Newspaper Reporter". The reason(s) for leaving would matter a great deal as well. Leaving a job for a much better job (better pay, more prestige, etc.) is quite different than leaving a job due to personality clash or poor work performance. There also could be a big difference depending upon the values of the employer in charge of doing the hiring. The person(s) with decision-making responsibility might place more emphasis on other traits and accomplishments, and not care terribly much that the employee left a job or jobs after a short time of being employed.

Paul Graham said something very similar about figuring out a program:

"I was taught in college that one ought to figure out a program completely on paper before even going near a computer. I found that I did not program this way. I found that I liked to program sitting in front of a computer, not a piece of paper. Worse still, instead of patiently writing out a complete program and assuring myself it was correct, I tended to just spew out code that was hopelessly broken, and gradually beat it into shape. Debugging, I was taught, was a kind of final pas... (read more)

4PhilGoetz10yThe people who taught you to architect programs before coding were also aware of this trade-off. It's a lot easier to write the small programs assigned in college that way, than the large programs you will write in the real world. This is not the top-down vs. bottom-up debate; both top-down and bottom-up design architect first. It is related to the concept of waterfall/iterative/incremental design; incremental designers can paint themselves into a corner. I've written a lot of big programs, and I've never regretted the time spent architecting them. I have sometimes wished I had spent more time designing them before starting. Fatal design errors that crop up down the road are more likely to be language-related: Your program gets so complicated that you have to rewrite it in a compiled language to run it in real-time or to avoid running out of RAM; or you discover that C++ templates don't work as advertised, or that Java can't allocate 2G of RAM, after you've already writtten 3000 lines.

I wish to expand on your conclusions and look for their limits. It might be more relevant to the "Go Try Things" post, but it being a kind of series of posts, I suppose it makes sense most to comment here.

So, data collection is good. But aside of getting one better at some area in which one tries to reach expertise or improvement, data collection is also good for discovering almost totally new facets of reality, territory that is outside the map's margins.

Data collection bring to light not only known unknowns, but unknown unknowns too. There's a ... (read more)

So far as I can tell, the real issue in telling someone that the only important thing is quality, is that it leads to a phenomenon known in some circles as "paralysis by analysis." For instance, a writer could spend a day debating whether or not a particular place needed a comma or not, and miss that the whole page is rubbish. In sports, it is often what is meant when someone is accused of "thinking too much." In football, a receiver might spend his time thinking about how to get around a defender once he has the ball, and forget to cat... (read more)

Did you write the title backwards?

0atucker10yNot intentionally, but it certainly looks better this way.

You should hold off on proposing solutions, but you also need to get around to actually trying the proposed solution.

I can almost swear that I was reading a post in the sequences where a statement was made to the effect that "If you already know where you think you'll end up on a decision/analysis/estimation, you should just head in that direction."

Does this ring any bells? I'd like to re-find that and have googled for it many, many ways and never found it again. It seems similar to the idea here, and thus I thought I'd ask for help here in finding it -- I really would like to read it again!

3Sniffnoy10yI suspect that you are referring to http://lesswrong.com/lw/jx/we_change_our_minds_less_often_than_we_think/ [http://lesswrong.com/lw/jx/we_change_our_minds_less_often_than_we_think/] .
1jwhendy10yWow -- I totally missed this comment! I just re-round this same thread, then tried to track back to this comment to let everyone know what the answer actually was... and found your answer pointing me there all along. Crazy. Thanks for the link.
2RichardKennaway10yOnce you know your destination, you are already there [http://wiki.lesswrong.com/wiki/Conservation_of_expected_evidence]. In the Way of Bayes, the prior probability equals the expected posterior probability: If you know your destination, you are already there [http://lesswrong.com/lw/ju/rationalization/]. You should not think that you know which direction [your belief] will go [after getting evidence], because by the laws of probability theory, if you know your destination, you are already there [http://lesswrong.com/lw/jz/the_meditation_on_curiosity/]. Those all have to do with belief and evidence. I'm sure Eliezer has said this about morality as well (if you want something to be the right action, you already think it is the right action), but I haven't tracked that down. There's this [http://lesswrong.com/lw/rr/the_moral_void/], but no pithy quote.
1jwhendy10yThanks for the links, but those aren't it. I'm familiar with that recurring tidbit, but the post I am thinking of, I believe, contained more of an encouraged heuristic; something like, "If you're finnagling over a decision and think you'll end up on one side, at a certain point, you should just get it over with and actually go to that side." As I was thinking about this, I thought it might have been in The Proper Use of Doubt [http://lesswrong.com/lw/ib/the_proper_use_of_doubt/]. While it's actually close, it's not what I remember. The quote I'm looking for would have embodied something kind of like this: Maybe I'm just making things up in my head!
0shokwave10yNo, I remember it too. It was something about a survey of students; of the group that could say they hadn't yet decided but would probably end up doing course A, almost every student chose course A. So the message was something like if you can guess where you're probably going to head, actually you've pretty much made the decision.
0jwhendy10yWell, that does sound a lot like what Richard came up with above: But... I take that comment to be somewhat of a negative statement. As in, you shouldn't know your destination, otherwise all of this rationality stuff is pointless. The phrase I recall (and perhaps what you're suggesting as well) was almost more of an encouragement -- like, "If you're endlessly deliberating, it will be more useful for you to head in the direction you're leaning vs. continuing to deliberate as if you don't know at all."
0Wilka10yMaybe it was "Once you can guess what your answer will be, you have probably already decided." from Hold Off On Proposing Solutions [http://lesswrong.com/lw/ka/hold_off_on_proposing_solutions/]

The scoring rule for group B is critically underspecified -- are they being graded on the best pot they produce? The worst? The mean? The median?

You should hold off on proposing solutions, but you also need to get around to actually trying the proposed solution.

I was under the impression that the whole point of holding off on proposing solutions was to make sure you came up with more than one idea.

As a general principle the right answer depends on the consequence of low quality and the balance of creativity versus practice. In medicine where the consequence of low quality (of a surgical operation for instance) may be death of the patient, there still is value to quantity and perfecting technique by frequent practice. Therefore simulations play a role. Ideally the surgeon makes use of much tacit knowledge in a successful operation, many instinctual elements that cannot be put into words. In many tasks like surgery, you want only intermittent creativity. Most efforts go into getting better and better at a task already mastered conceptually, not into improving the concept.

Wow, what a cool experiment.

4Desrtopa10yI'm surprised he was allowed to do it; I don't think many institutions will let professors apply inconsistent grading criteria like that.

Seems so surprising I would guess that it was apocryphal, not knowing the context.

I did a further search online; the quote appears verbatim in many places, but I can't find any additional information which would single out a school or teacher, so I would err on the side of assuming it's fictional.

8atucker10yNow I feel dirty. Apparently it comes from this [http://www.amazon.com/exec/obidos/ASIN/0961454733/lifeclever-20?tag=lifeclever-20] book. Unfortunately artists don't seem to do anything like peer review, so if it really did happen I wouldn't expect it to be shown any more verifiably than this. On the other hand, I don't particularly trust artists not to make up a story and call it an experiment.
5[anonymous]10yOne really shouldn't give probably apocryphal evidence like this in support of a theory.
7Eliezer Yudkowsky10yActually I think the next step is to run the experiment. It's a perfectly good experiment - even though I'd expect it not to happen inside a real school, because the students would complain they weren't being graded fairly. Concept is still important enough that I promoted it immediately.
9Giles10yI would expect it not to happen inside a real school because the school would be uninterested in improving their teaching via the experimental method. Of course I hope I'm wrong! (Anyone know of particularly enlightened schools?) If I'm right it would seem a much harder problem to solve than the being-graded-fairly one.
6atucker10ySomeone could just book a ceramics studio and advertise free lessons/use in exchange for participating in an experiment. Then split the people there into two groups, and tell one group that they get money (or other arbitrary desirable) based on how many pots they make, and the other on how good their pots are.
3thomblake10yMy wife intends to start teaching sculpture in a year or so... I'll make sure to bring this up to her. The tricky part will be objectively grading quality (an outside judge and blindness should be sufficient).
0twanvl10yMaybe it could be done across different schools, different classes or different years. For example, in year 1 teach subject focusing on quality and subject 2 focusing on quantity. Then in year 2 reverse the roles. But then you also need to be careful with the order of the subjects. Just splitting the students into two groups would be better though, aside from the complaints. This is a problem with A/B testing in general: people want to be treated fairly. Are there good ways to reduce (the risk of) such complaints?
1759846215310yTo me, the simplest solution that comes to mind is to grade on a curve at the end of the course, based on the quality of the work (or some other subjective measure - I'm not familiar with university art courses, but I assume there's some kind of widely-accepted grading methodology). That is to say - tell the students before the course they will be graded on quality or quantity depending on the group, but grade each student on a curve relative to their own section for their permanent grade once the course has finished. This would obviously still require lying to the students (at the very least by omission, arguably), however.
[-][anonymous]10y -1

I'm a bit incredulous about the experiment. The students' quantity of production can always be maximized by decreasing the quality. Why wouldn't they? (Even if the students maintain quality, how could the teacher justify assuming this in advance.)

Evaluating the study formally as an experiment, attention to quality is confounded with spending time on the work. (Reiteration is a different story, but in this case we seem to be talking about moving on to a different project, rather than perfecting a single design.) Did the students evaluated on quality produce... (read more)

5taryneast10yI agree - mere quantity is not enough... but the act of creating quantity gives a person the chance to gain the experience needed to learn how to improve the quality. Very few people can get to Quality without going through the quantity... Quantity is a necessary, but not sufficient condition of quality.