In 2007, the Department of Children, Youth, and Families (DCYF) held a seminar for the nonprofits vying for a piece of $78 million in funding. Grant seekers were told that in the next funding cycle, they would be required — for the first time — to provide quantifiable proof their programs were accomplishing something.
The room exploded with outrage. This wasn't fair. "What if we can bring in a family we've helped?" one nonprofit asked. Another offered: "We can tell you stories about the good work we do!" Not every organization is capable of demonstrating results, a nonprofit CEO complained. He suggested the city's funding process should actually penalize nonprofits able to measure results, so as to put everyone on an even footing. Heads nodded: This was a popular idea.
Actually, these objections might not be quite as insane as they might sound at first.
The issue is that rigorously measuring results is hard, and frequently when people try to quantify results, they screw it up and force people to spend their time gaming a dysfunctional metric instead of doing real work. Just look at everyone who complains about academia forcing researchers to publish everything they can in as small bites as possible in order to maximize citations, instead of being able to do things in a way that'd be more useful for everyone. Or look at the software companies that used to measure programmer productivity in terms of lines of code written, and - as far as I know - still haven't managed to come up with any very good objective metric for comparing their workers.
The fact is that there are plenty of cases where we know something, but don't have any way of showing it in an objective and easy-to-quantify way. A boss might know for sure who's a valuable researcher or programmer on the basis of her interactions with them, but be unable to prove it rigorously. And these are still relatively simple domains - take something very open-ended like "the impact of nonprofits", and things get even worse.
Given that people are generally bad at designing good ways of quantifying such things, and that bad measures will produce worse results than no measures at all, then it can actually make perfect sense for somebody interested in helping people to object to the creation of such measures. Better (the thought goes) to give everyone money and end up funding both useless and high-impact organizations, than to concentrate all the money to a few organizations which are good at gaming the metrics and most probably all useless.
But there is always some metric to be gamed. There is always some causality chain which results in a specific distribution of money to organizations. Just because we close our eyes, it does not make the causality go away.
Instead of organizations wasting money on and time a dysfunctional official metric, there will be organizations wasting money and time on alternative ways (bribery, fraud, advertising, lobbying, propaganda...) of convincing government that it's they who should get a bite from the budget.
But there is always some metric to be gamed. There is always some causality chain which results in a specific distribution of money to organizations. Just because we close our eyes, it does not make the causality go away.
This doesn't necessarily mean that the system is gameable. If we suppose that what we're measuring is actually exactly what we want to get out of the program, then the only way the program can get ahead is by providing more of what we want. The system can only be "gamed" to the extent that there's a divorce between what we want from it, and what sort of output we measure.
I'm leery of organizations providing their own statistics on how effective they are, which may just be another form of lobbying and propaganda. I'd lean towards carving out from the budget a group that independently assesses effectiveness of each of the organizations. It's admittedly imperfect, but it would be more impartial than what seems to be in place now, and agencies wouldn't necessarily be at a disadvantage for lacking their own internal measurement tools. That still leaves the problem of choosing the right metrics. Something simple like budget percentages and ratios would be a good place to start. There are a lot of hard-to-compare types of services out there; after school programs aren't like Meals on Wheels programs. It's hard to come up outcome based metrics to say which service is better than another when there's so many different categories. Adopting something along the lines of the financial ratings at Charity Navigator could at least get everyone on the same page for controlling costs at their organizations.
It's hard to come up outcome based metrics to say which service is better than another when there's so many different categories.
At least we could compare organizations within the same category. Splitting the budget across categories would remain a political decision, but within a category, inefficient organizations should be ignored.
Even in absence of measurements, it would be good if all organizations would have to make and publish their reports, which would have to approved by a person who could point at different places in the report and say "be more specific".
For example if the first approximation is "Our organization supports world peace", the 20th approximation would be "We have spent $1,000,000 on salaries of our employees; $500,000 on our building; $100,000 on food; and $100 on printing 200 flyers with big colored letters 'World Peace is a Great Thing'. Then we used volunteers to distribute those flyers on the university campus. Also, we paid $10,000 for design." And then the Chief Specificity Officer would say: "OK, this is specific enough, you may publish it online."
Instead of organizations wasting money on and time a dysfunctional official metric, there will be organizations wasting money and time on alternative ways (bribery, fraud, advertising, lobbying, propaganda...) of convincing government that it's they who should get a bite from the budget.
Sure, but the existence of bribery and corruption can actually be viewed as an argument for not establishing any rigorous metrics: the worse things are, the more likely it is that any supposedly objective metrics get turned into instruments for giving specific organizations plenty of money and shutting everyone else out. That happens all the time in the public sector, with e.g. lucrative contracts being supposedly offered for anyone who can fill the requirements and bids the lowest, but with the requirements being intentionally crafted so that only a few favored organizations can match them. Without such a setup, other organizations might actually get their offers into consideration.
That happens all the time in the public sector, with e.g. lucrative contracts being supposedly offered for anyone who can fill the requirements and bids the lowest, but with the requirements being intentionally crafted so that only a few favored organizations can match them. Without such a setup, other organizations might actually get their offers into consideration.
If someone is corrupted enough to design the requirements so that they match a favored organization... what will happen if the same person is allowed to make the same decision, without the condition of choosing the lowest price? I guess the same favored organization will get the contract, only the price will be much higher.
(Case study: This is how highways are built in Slovakia from European Union's PPP money. When one specific political party is in government, only 20 kms per year are built for the same money that was enough to build 100 kms per other years. The reason is that all contracts go to companies belonging to the boss of the given party. All competing companies are refused, officially because their prices are "suspiciously low".)
The issue is that rigorously measuring results is hard, and frequently when people try to quantify results, they screw it up and force people to spend their time gaming a dysfunctional metric instead of doing real work.
This is a problem in business as well. Marketo is able to charge companies thousands per month for tracking online advertising outcomes in companies with long, relationship-based B2B sales cycles (who might be aiming to make a few huge sales per year).
John Wanamaker: "Half the money I spend on advertising is wasted; the trouble is I don't know which half."
Oh dear, it's like San Francisco is being run by liberals as stereotyped by right-libertarians.
Is your point (1) "See, the right-libertarians are right: put liberals in power and this is what they do" or (2) "Don't take this article too seriously; it was probably written by right-libertarians and reflects their stereotypes rather than reality" or (3) something else?
Something else. I'm not too familiar with the newspaper, so I wouldn't know whether it is right-libertarian biased (JoshuaZ says it's not, and the fact that it is based in SF is also evidence that it's not). The article had enough specifics that it's pretty hard to attribute all of its claims to bias anyway. And 1 isn't quite right either; the New England states are easy counterexamples. I was simply noting that a lot of the problems in San Francisco are fairly similar to the ones right-libertarians are often concerned about. I guess my comment was a bit vague; sorry.
San Francisco is so rich that it's not pressured to get rid of insane policies that harm people. Saudi Arabia is so rich that it's not pressured to get rid of insane policies that harm people. And what does this have to do with liberal democracy?
almost at its best
The Worst-Run Big City in the U.S.
I'm interested by the article, and I enjoy that you linked it here. And I would prefer that you kept it even farther away from the ideological end of the spectrum.
What I meant by "almost at its best" is that the raw human capital of San Francisco is on a global scale incredibly good. There are few cities that have higher quality inhabitants.
I'm not quite sure why that is an ideological statement. As to the title I just reused the original, I don't literally agree with it since I can think of at least one big US city that is worse run.
You were pressing it into the service of a sweeping generalization, and all generalizations are ideological*. And, because of the selection implied in the very title of the post, generalization here is hasty.
*joke
Not every organization is capable of demonstrating results, a nonprofit CEO complained.
James Q. Wilson noted to important variables about service organizations: whether their input could be easily tracked (or not) and whether their output could be easily tracked.
An example of both being easy is the Social Security administration (payments to the elderly). The amount of each check is easily verified as correct, as is the number of checks cashed by recipients. An example of both being difficult is restaurant health inspection. The number of violations found or businesses checked is difficult to relate to the desired goals. Likewise, the level of sickness in the population is difficult to correlate with the cleanliness of the food industry (at least, at the levels of food inspection that are politically feasible).
One path to Lost Purpose is to use a bad measure what quantifying input or output is hard - and not realize that you are using a bad measure.
Thanks for the link.
I don't see evidence that SF is the worst-run city, though due to its wealth it may waste the most money.
Similar problem, and scarier-- I think one way the liberal consensus has gone wrong is to assume that competence will take care of itself, so that all the moral pressure goes to preventing various sorts of prejudice.
I agree that efficient charity (if it can be kept honest) could make a huge positive difference.
I think one way the liberal consensus has gone wrong is to assume that competence will take care of itself, so that all the moral pressure goes to preventing various sorts of prejudice.
You seem to have a prejudice against incompetent people. :P
San Francisco historian Charles Fracchia recalls Mayor George Christopher's ploy after his plan to lure the New York Giants to San Francisco hit a snag in the late 1950s. It all hinged on building Candlestick Park, and doing that hinged on buying land in Hunters Point from real estate magnate Charlie Harney for $65,000 an acre. The trouble was, the city had sold that same land to Harney only five years previously for a fraction of the price.
"There was opposition to this from high-minded people in San Francisco," Fracchia says. "So Christopher got his opponents as well as his proponents together, and had 10 cases of scotch delivered up to this meeting at the Pacific Union Club. The scotch was drunk, and everyone came to the conclusion — yes, keep Candlestick Park."
When it comes to mismanaging a city, San Francisco has pulled a 180 — in half a century, we've gone from "city fathers" (if you liked them) or "oligarchs" (if you didn't) operating with limited input from the people to a hyperdemocracy. Overpaying for a Candlesticklike bad land deal today wouldn't be settled during a drunken soirée, but via years of high-decibel public meetings, developers being made to bleed funds to nonprofits of city supervisors' choosing, and any number of bond measures or other trips to the ballot box — all of which, when put together, could conceivably cost as much as the bad land deal itself. Maybe more.
For all its scotch-soaked flaws, the city of yore did not suffer from these problems. While archaic and stridently antidemocratic by today's standards, the system of government cobbled together by a citizens' commission in 1931 largely did what our forebears wanted it to do — mind the store and eliminate rampant corruption.
From 1932 until 1996, much of city government was handled by a powerful chief administrative officer (CAO), appointed to a 10-year term and tasked with overseeing the city's largest departments. The job was to take politics out of city management. (Today's San Francisco is so intensely saturated with politics down to the minutiae that the supervisors' recent appointment of a transit expert to a transit board — and not a union plumber — was seen as a deeply political move and an affront to organized labor.) The CAO was charged with making the city's largest decisions in an apolitical manner; the major portion of the job was keeping the books on the most vital departments and making sure they were running smoothly. In a manner of speaking, the CAO was a living, breathing accountability measure. The city certainly made its share of lousy calls, but the sloth, waste, and dysfunction emblematic of today's city government would have been shocking.
Over time, however, the CAO's purview was replaced by that hyperdemocracy. The reasonable notion that the people of San Francisco should have input into how things are run has turned into the democratic equivalent of death by a thousand cuts; as everybody gets a voice, democracy votes accountability down. When everyone's in charge, no one is. "In the old days, they ran roughshod over opposing views," Fracchia says. "Today, all ya got is opposing views. Pick your poison."
Wait maybe he has been reading Unqualified Reservations?
There are ways San Francisco can maintain its rampant democracy while establishing a system that abhors waste and incompetence:
Return much of the day-to-day control of city operations to an unelected, long-term city manager — who would also be responsible for negotiating union contracts.
Institute detailed citywide planning to avoid waste and duplication of services, while ensuring essential city functions are provided for.
Emphasize best practices in each individual city department, and let go of workers who aren't needed because of productivity gains.
Eliminate all budget set-asides and mandatory staffing levels, and let the city develop budgets that meet the needs of today and tomorrow, not yesterday.
Fire people who are incompetent — and that includes those at the top-heavy manage-ment level.
Instead of telling us how much money has been spent on a problem, focus on whether the problems are getting solved.
Yet it would take a seismic event to spur the city to shake off caked-on layers of status quo — a literal earthquake, or a figurative one. (Perhaps a meteor vaporizing City Hall in 2012.)
The far more likely scenario is that nothing will happen. The city will continue its orgy of waste and incompetence. San Francisco can afford plenty of both: We're rich — and getting richer all the time. According to the controller's office, San Franciscans' per-capita income jumped from an already-generous $58,244 in 2004 to $74,515 last year.
Of course, for many San Franciscans, those numbers represent another failure. They point to an exodus. The city's middle class is melting away faster than polar ice. With them, economists and demographers say, goes any realistic hope that voters will demand serious change in search of long-term reform.
On second thought maybe living in San Francisco and thinking about it too much makes you sound like Moldbug.
You are on record as saying that you don't like the level of political discussion on this site. Adding more political discussion from your favored side is not an intervention likely to decrease the frequency of political discussion on this site.
You think this post (and the comments in response) raise the level of political discourse on this site?
I think this post is about on par (in terms of quality) with other recent posts. Konkvistador has explicitly said that he thought those posts were lowering the quality of discussion here. My question for him is why he thinks this post will do better, as opposed to presenting a viewpoint he agrees with - a reasonable but very different goal.
Note that I quoted the part that was not that far away of some of my policy preferences and more explicitly questioned the instrumental usefulness of democracy in a comment not at the centre of discussion (opening post).
I stated it because I wanted to share that the article caused me to update away from "Moldbug has a lot of insight into the failure of democratic government" towards "Moldbug has a lot of insight into the failure of democratic government in SF and may be wrong about other places".
You think this post (and the comments in response) raise the level of political discourse on this site?
Presumably Konkvistador is posting with that intent.
And I'm trying to alert him that he may not have achieved that intent. Further, the intent and the action may be hopelessly mismatched.
My question for him is why he thinks this post will do better, as opposed to presenting a viewpoint he agrees with - a reasonable but very different goal.
In theory we should be more reasonable when discussing means rather than goals since clippy and a baby eater can both easily agree on which strategy is paper-clip-maximizing given the same information and resources. I honestly didn't expect this article to be seen as being partisan or political because the author explicitly endorses the goals San Francisco seems to pursue and just criticizes the means by giving examples of how they don't seem to be working well.
Despite its good intentions, San Francisco is not leading the country in gay marriage. Despite its good intentions, it is not stopping wars. Despite its spending more money per capita on homelessness than any comparable city, its homeless problem is worse than any comparable city's. Despite its spending more money per capita, period, than almost any city in the nation, San Francisco has poorly managed, budget-busting capital projects, overlapping social programs no one is certain are working, and a transportation system where the only thing running ahead of schedule is the size of its deficit.
I find it amusing that if I was to write out a well argued or researched post questioning whether any particular item among these should be a goal for San Francisco to work towards it would probably be considered less political. This seems to indicate that doing politics, at least here, is not only not about policies as Robin Hanson says, but it may not be about values either.
In theory we should be more reasonable when discussing means rather than goals since clippy and a baby eater can both easily agree on which strategy is paper-clip-maximizing given the same information and resources.
You might think so, but when the means include getting the compliance of other people, I've seen some very nasty flame wars. My theory is that most people's models for getting people to comply are established early in life, emotionally fraught, and not reconsidered. Almost any method can look as though it works some of the time.
This.
If anything we used to talk about politics in the sense I did, giving real world examples of possible institution failure or success, much more in the old Overcoming Bias days. And while before karma the worst comments where much worse and before Politics is the Mindkiller we where less careful about how we approached it, I would say we where still less mindkilled and more capable of discussing it.
In the example thread TimS gave the goal was already assumed and not made explicit, this would be less of a problem if it was an academic exercise rather than directly tied to changing the LW community. And I should emphasise that since I don't believe in most kind of political activity a citizen is supposed to engage in the governing style of SF for me really is an academic question with little real world implications beyond tangential lessons on how to organize a non-profit or for-profit company.
Recall that in my recent comment I also stated I was distressed no one else wrote the arguments I did. I believe we are seeing evaporative cooling of users belonging to ideologies besides the dominant one. This is I think bad for our explicit goal of "refining the art of human rationality". I should emphasise see little evidence of quality thinkers and rationalists who used to disagree having changed their mind in the direction of it as much as given up on the site.
If you check my older comment history especially on the Rational Romance thread I have changed my position on how best to deal with this several times I think based on reasonable grounds. I don't think it is an easy question partially because I mistrust myself on it. So your input on the issue would be most welcomed! Maybe even a new open thread discussion so we clear away the object level baggage from here.
Also unless we are capable of discussing San Francisco's governance and competence at achieving a set of goals its inhabitants or employees would like it to, I can't see how we could be capable of talking about economics at all.
And we really should because that field is where so many of LW's assumptions and data about human minds as well as strategy come from!
Recall that in my recent comment I also stated I was distressed no one else wrote the arguments I did. I believe we are seeing evaporative cooling of users belonging to ideologies besides the dominant one.
Given that other users are worried about apparently the same thing in respect to certain classically non-dominant ideologies, this seems more likely to be some variant of the hostile media bias effect than anything else.
Why assume both are wrong rather than just one of them being wrong? Also we do have weak statistical evidence that LW has been drifting in the direction I think it has and strong evidence the positions I talk about are a vulnerable minority here to begin with (see Yvain's polls).
Why assume both are wrong rather than just one of them being wrong?
Because the prior given the standard cognitive biases here is that these sorts of claims in general are so inaccurate that the impressions from people are barely informative if at all.
we do have weak statistical evidence that LW has been drifting in the direction I think it has
Can you expand on this?
I laughed. But Moldbug is authoritarian, not libertarian. Any government, for-profit or otherwise, with absolute power is not liberty friendly.
I would agree with a slightly modified version of this. On LessWrong libertarian politics is the neutral baseline of economics and liberal politics is the neutral baseline of morality. All other options are "mindkilling".
In addition to this I will note our similarity to academia. Much of the tribal attire of "libertarians" is endorsed among economists and much of the moral tribal attire of "liberals" is endorsed by bioethics and philosophy. Not that these are necessarily good signals...
It's annoying and stupid, as it precludes any attempt to use the supposedly superior powers of rationality found hereabouts on quite a range of actual practical problems. Worse yet, whenever someone does try, the proscription proves well-founded. Sigh.
Worse yet, whenever someone does try, the proscription proves well-founded.
Does this happen for issues besides feminism?
I agree.
Also I just plain enjoy well written and well thought out defences of Libertarian economic and Liberal moral positions I disagree with that only happen when they actually get challenged. I recommend Yvain's blog for the latter btw.
As someone who makes weekly forays into San Francisco, I strongly suspect that many of Molbug's more extreme beliefs regarding the current state of US politics are a result of him generalizing from living in SF. I also suspect I'd be far less sympathetic to Moldbug if I grew up in, say, rural Texas rather than the Bay Area.
I also like the show (it is like public choice economics made into comedy), but why not tell us why you recommend it in the media thread?
The Worst-Run Big City in the U.S.
A six page article that reads as a very interesting autopsy of what institutional dysfunction in the intersection of government and non-profits looks like. I recommend reading the whole thing.
Minus the alleged harassment, city government is filled with Yomi Agunbiades — and they're hardly ever disciplined, let alone fired. When asked, former Board of Supervisors President Aaron Peskin couldn't remember the last time a higher-up in city government was removed for incompetence. "There must have been somebody," he said at last, vainly searching for a name.
Accordingly, millions of taxpayer dollars are wasted on good ideas that fail for stupid reasons, and stupid ideas that fail for good reasons, and hardly anyone is taken to task.
The intrusion of politics into government pushes the city to enter long-term labor contracts it obviously can't afford, and no one is held accountable. A belief that good intentions matter more than results leads to inordinate amounts of government responsibility being shunted to nonprofits whose only documented achievement is to lobby the city for money. Meanwhile, piles of reports on how to remedy these problems go unread. There's no outrage, and nobody is disciplined, so things don't get fixed.
You don't say?
In 2007, the Department of Children, Youth, and Families (DCYF) held a seminar for the nonprofits vying for a piece of $78 million in funding. Grant seekers were told that in the next funding cycle, they would be required — for the first time — to provide quantifiable proof their programs were accomplishing something.
The room exploded with outrage. This wasn't fair. "What if we can bring in a family we've helped?" one nonprofit asked. Another offered: "We can tell you stories about the good work we do!" Not every organization is capable of demonstrating results, a nonprofit CEO complained. He suggested the city's funding process should actually penalize nonprofits able to measure results, so as to put everyone on an even footing. Heads nodded: This was a popular idea.
Reading this I had to bite my hand in frustration.
There are two lessons here. First, many San Francisco nonprofits believe they're entitled to money without having to prove that their programs work. Second, until 2007, the city agreed. Actually, most of the city still agrees. DCYF is the only city department that even attempts to track results. It's the model other departments are told to aspire to.
But Maria Su, DCYF's director, admitted that accountability is something her department still struggles with. It can track "output" — what a nonprofit does, how often, and with how many people — but it can't track "outcomes." It can't demonstrate that these outputs — the very things it pays nonprofits to do — are actually helping anyone.
"Believe me, there is still hostility to the idea that outcomes should be tracked," Su says. "I think we absolutely need to be able to provide that level of information. But it's still a work in progress." In the meantime, the city is spending about $500 million a year on programs that might or might not work.
What the efficient charity movement has done so far looks much more impressive in light of this. Reading the rest of the article I think you can on your own identify the problems caused by lost purposes, applause lights and a dozen or so other faults we've explored here for years.
Discussions here are in many respects a comforting illusion, this is what humanity is like out there in the real world, almost at its best, well educated, wealthy and interested in the public good.
Yes it really is that bad.