Open Thread, May 5 - 11, 2014

Previous Open Thread

You know the drill - If it's worth saying, but not worth its own post (even in Discussion), then it goes here.


Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one.

3. Open Threads should start on Monday, and end on Sunday.

4. Open Threads should be posted in Discussion, and not Main.

284 comments, sorted by
magical algorithm
Highlighting new comments since Today at 8:46 AM
Select new highlight date

Is Less Wrong dying?

Some observations...

  • The top level posts are generally well below the quality of early material, including the sequences, in my estimation.
  • 'Main' posts are rarely even vaguely interesting to me anymore.
  • 'Top Contributors' karma values seem very low compared to what I remember them being ~9-12 months ago.
  • 'Discussion' posts are littered with Meetup reminders.

About all I look at on LW anymore is the Open Discussion Thread, Rationality Quotes and the link to Slate Star Codex. I noticed CFAR and MIRI's websites gave me the impression they were getting more traction and perhaps making some money.

Has LW run it's course?

I think it's a little early to predict the end, but there's less I'm interested in here, and I'm having trouble thinking of things to write about, though I can still find worthwhile links for open threads.

Is LW being hit by some sort of social problem, or have we simply run out of things to say?

I'd add "Metacontrarianism is on the rise" to your list. Many of the top posts now are contrary to at least the spirit of the sequences, if not the letter, or so it feels to me.

Maybe it's because the important things have started, and moved to real life, outside of the LW website. There are people writing and publishing papers on Friendly AI, there are people researching and teaching rationality exercises; there are meetups in many countries. -- Although, if this is true, I would expect more reports here about what happens in the real life. (Remember the fundamental rule of bureaucracy: If it ain't documented, it didn't happen.)

Anyway, this is only a guess; it would be interesting to really know what's happening...

Has LW run it's course?

It seems to be a common sentiment, actually. I mentioned this a few times on #lesswrong and the regulars there appear to agree. Whether this is a some sort of confirmation bias, I am not sure. Fortunately, there is a way to measure it:

Count interesting articles from each period and compare the numbers.

I blame Facebook. Many of the discussions that are had there were of the type that used to invigorate these here boards.

Hm. I think you have a much higher level of sophistication in your FB friend group. I get a lot of Tea Party quotes and pictures of peoples' dinner.

It's mostly that Eliezer has taken to disseminating his current work via open Facebook discussions. I can see how that choice makes sense, from his position, but it's still sad for the identity-paranoid and the nostalgic remnants still roaming these forgotten halls. Did I have a purpose once? It's been so long.

Also, it's much harder (impossible?) to find older discussions on FB.

And perhaps harder to grow, at least through the usual means - the Facebook discussions wouldn't show up on Google searches (or at least not highly ranked, I think), and it's a less convenient format to link someone to for an explanation of a concept.

It turns out that while there may be no good way to use Facebook to find old discussions on Facebook, I used google and found an old Facebook post.

I would say LW is evolving.

The Sequences are and always were the finger that points at the objective, not the objective unto itself. The project of LW is "refining the art of human rationality." But we don't have the defininition of human rationality written on stone tablets, needing only diligence in application to obtain good results. The project of LW is thus a dynamic process of discovery, experimentation, incorporating new data, sometimes backtracking when we update on evidence that isn't as solid was we had thought.

You correctly observe that the style of participation has changed over time. This is probably mostly the result of certain specific high volume contributors moving on to other things. It could also be the result of an aggregated shift in understanding as to what kinds of results can actually be produced by discussing rationality in a vacuum, which may perhaps be why these contributors have moved on. Or maybe they just said all they felt they needed to say, I don't know. I have a 101.1 F fever right now.

I remember people saying things like "Less Wrong is dying" for a long time, from 2010 at least. Which doesn't invalidate the claim that LW's much more quiet than it used to be, of course, but it does challenge the claim that this would be a recent development.

I personally believe it's basically dead—at least for me. The sequences are great... But I wouldn't recommend LW to anyone at this point in terms of it's recent content, and that is a big change for me.

It's been a good run.

The LW census get's every year more participants. If LW would be dying I would expect the opposite.

I'm not sure total participants is a good metric to use in making that determination. It depends on people's level of participation and engagment, I think.

When it comes to engagement we do have a bunch of in person meetups that we didn't have a few years ago.

There do seem to be more meetups globally, but I'd say the SF Bay Area meetup scene -- where MIRI is based and many prominent contributors live or have lived --- is well off its peaks. This is perhaps an unreasonable time to be saying so, since the South Bay and East Bay meetups have just gone through major shakeups and haven't yet stabilized; but even ignoring that we're well down from two or three years ago in terms of engagement with high-karma users, in terms of number of local meetup groups, and probably in terms of people as well.

As per issue #389, I've just pushed a change to meetups. All future meetup posts will be created in /r/meetups to un-clutter /r/discussion a little bit.

Hmm, I just noticed that the 'Nearest Meetup' feature is mostly removed (you can still see the field when you refresh before everything has loaded), so you cant see any notification anywhere for local meetups happening soon unless you are specifically checking /meetups or r/meetups.

I understand why Luke and co wanted this change asap (people have been complaining about the clutter), but I suspect that this change will have a big overall impact on LW Meetups turnouts. I'm fairly certain that a lot of non-regulars decide to go to a specific meetup because they are randomly reminded of it in the sidebar or in discussion, and not because they actively check.

Anyway, is there any chance you know why the 'Nearest meetup' area was removed (no mention of the removal in the issues)? I am not sure what the benefit is of having Upcoming Meetups over Nearest Meetups, but the latter at least provides a reminder for people of posted local meetups. Alternatively, is there anything else planned to serve as a reminder?

PS: I would've published this as a comment on the issue itself, but that didn't look very appropriate.

I currently see 'nearest meetups'.

I've noticed that when I'm at work (but still logged in), it shows me 'upcoming meetups' instead. My first guess, that I've made no attempt to confirm or disconfirm, is that it tries to determine your location from your IP address. If it succeeds it shows you 'nearest meetups', and if it fails it shows you 'upcoming meetups'.

I feel like there should definitely be a link to 'meetups' next to 'main' and 'discussion'. It's so easy to miss things in the sidebar.

I, too, expect this to reduce meetup turnout.

I've noticed that when I'm at work (but still logged in), it shows me 'upcoming meetups' instead.

Looks like it is the same for me - I posted the above comment from work, however, I see 'Nearest Meetups' now that I am home. Your theory sounds reasonable.

philh is correct, and nothing I pushed should've changed the sidebar behaviour.

For those that are worried about meetup attendance being affected:

How many people discover meetups through /r/discussion as opposed to the sidebar and /meetups? Perhaps I should poll this:

Before this change, how did you discover LW meetups? If none apply, please write in.


Below is an edited version of an email I prepared for someone about what CS researchers can do to improve our AGI outcomes in expectation. It was substantive enough I figured I might as well paste it somewhere online, too.

I'm currently building a list of what will eventually be short proposals for several hundred PhD theses / long papers that I think would help clarify our situation with respect to getting good outcomes from AGI, if I could persuade good researchers to research and write them. A couple dozen of these are in computer science broadly: the others are in economics, history, etc. I'll write out a few of the proposals as 3-5 page project summaries, and the rest I'll just leave as two-sentence descriptions until somebody promising contacts me and tells me they want to do it and want more detail. I think of these as "superintelligence strategy" research projects, similar to the kind of work FHI typically does on AGI. Most of these projects wouldn't only be interesting to people interested in superintelligence, e.g. a study building on these results on technological forecasting would be interesting to lots of people, not just those who want to use the results to gain a bit of insight into superintelligence.

Then there's also the question of "How do we design a high assurance AGI which would pass a rigorous certification process ala the one used for autopilot software and other safety-critical software systems?"

There, too, MIRI has lots of ideas for plausibly useful work that could be done today, but of course it's hard to predict this far in advance which particular lines of research will pay off. But then, this is almost always the case for long-time-horizon theoretical research, and e.g. applying HoTT to program verification sure seems more likely to help our chances of positive AGI outcomes than, say, research on genetic algorithms for machine vision.

I'll be fairly inclusive in listing these open problems. Many of the problems below aren't necessarily typical CS work, but they could plausibly be published in some normal CS venues, e.g. surveys of CS people are sometimes published in CS journals or conferences, even if they aren't really "CS research" in the usual sense.

First up are 'superintelligence strategy' aka 'clarify our situation w.r.t. getting good AGI outcomes eventually' projects:

  • More and larger expert surveys on AGI timelines, takeoff speed, and likely social impacts, besides the one reported in the first chapter of Superintelligence (which isn't yet published).

  • Delphi study of those questions including AI/ML people, AGI people, and AI safety+security people.

  • How big is the field of AI currently? How many quality-adjusted researcher years, funding, and available computing resources per year? How many during each past previous decade in AI? More here.

  • What is the current state of AI safety engineering? What can and can't we do? Summary and comparison of approaches in formal verification in AI, hybrid systems control, etc. Right now there are a bunch of different communities doing AI safety and they barely talk to each other, so it's hard for any one person to figure out what's going on in general. Also would be nice to know which techniques are being used where, especially in proprietary and military systems for which there aren't any papers.

  • Surveys of AI subfield experts on “What percentage of the way to human-level performance in your subfield have we come in the last n years”? More here.
  • Improved analysis of concept of general intelligence beyond “efficient cross-domain optimization.” Maybe just more specific: canonical environments, etc. Also see work on formal measures of general intelligence by Legg, by Hernandez-Orallo, etc.
  • Continue Katja’s project on past algorithmic improvement. Filter not for ease of data collection but for real-world importance of the algorithm. Interesting to computer scientists in general, but also potentially relevant to arguments about AI takeoff dynamics.
  • What software projects does the government tend to monitor? Do they ever “take over” (nationalize) software projects? What kinds of software projects do they invade and destroy?
  • Are there examples of narrow AI “takeoff”? Eurisko maybe the closest thing I can think of, but the details aren't clear because Lenat's descriptions were ambiguous and we don't have the source code.

  • Cryptographic boxes for untrusted AI programs.

  • Some AI approaches are more and less transparent to human understanding/inspection. How well does each AI approach's transparency to human inspection scale? More here.
  • Can computational complexity theory place any bounds on AI takeoff? Daniel Dewey is looking into this; it currently doesn't look promising but maybe somebody else would find something a bit informative.
  • To get an AGI to respect the values of multiple humans & groups, we may need significant progress in computational social choice, e.g. fair division theory and voting theory. More here.

Next, high assurance AGI projects that might be publishable in some CS conferences/journals. One way to categorize this stuff is into "bottom-up research" and "top-down research."

Bottom-up research aimed at high assurance AGI simply builds on current AI safety/security approaches, pushing them along to be more powerful, more broadly applicable, more computationally tractable, easier to use, etc. This work isn't necessarily focused on AGI specifically but is plausibly pushing in a more safe-AGI-helpful direction than most AI research is. Examples:

To be continued...


Top-down research aimed at high assurance AGI tries to envision what we'll need a high assurance AGI to do, and starts playing with toy models to see if they can help us build up insights into the general problem, even if we don't know what an actual AGI implementation will look like. Past examples of top-down research of this sort in computer science more generally include:

  • Lampson's original paper on the confinement problem (covert channels), which used abstract models to describe a problem that wasn't detected in the wild for ~2 decades after the wrote the paper. Nevertheless this gave computer security researchers a head start on the problem, and the covert channel communication field is now pretty big and active. Details here.
  • Shor's quantum algorithm for integer factorization (1994) showed, several decades before we're likely to get a large-scale quantum computer, that (e.g.) the NSA could be capturing and storing strongly encrypted communications and could later break them with a QC. So if you want to guarantee your current communications will remain private in the future, you'll want to work on post-quantum cryptography and use it.
  • Hutter's AIXI is the first fully-specified model of "universal" intelligence. It's incomputable, but there are computable variants, and indeed tractable variants that can play arcade games successfully. The nice thing about AIXI is that you can use it to concretely illustrate certain AGI safety problems we don't yet know how to solve even with infinite computing power, which means we must be very confused indeed. Not all AGI safety problems will be solved by first finding an incomputable solution, but that is one common way to make progress. I say more about this in a forthcoming paper with Bill Hibbard to be published in CACM.

But now, here are some top-down research problems MIRI thinks might pay off later for AGI safety outcomes, some of which are within or on the borders of computer science:

  • Naturalized induction: "Build an algorithm for producing accurate generalizations and predictions from data sets, that treats itself, its data inputs, and its hypothesis outputs as reducible to its physical posits. More broadly, design a workable reasoning method that allows the reasoner to treat itself as fully embedded in the world it's reasoning about." (Agents build with the agent-environment framework are effectively Cartesian dualists, which has safety implications.)
  • Better AI cooperation: How can we get powerful agents to cooperate with each other where feasible? One line of research on this is called "program equilibrium": in a setup where agents can read each other's source code, they can recognize each other for cooperation more often than would be the case in a standard Prisoner's Dilemma. However, these approaches were brittle, and agents couldn't recognize each other for cooperation if e.g. a variable name was different between them. We got around that problem via provability logic.
  • Tiling agents: Like Bolander and others, we study self-reflection in computational agents, though for us its because we're thinking ahead to the point when we've got AGIs who want to improve their own abilities and we want to make sure they retain their original purposes as they rewrite their own code. We've built some toy models for this, and they run into nicely crisp Gödelian difficulties and then we throw a bunch of math at those difficulties and in some cases they kind of go away, and we hope this'll lead to insight into the general challenge of self-reflective agents that don't change their goals on self-modification round #412. See also the procrastination paradox and Fallenstein's monster.
  • Ontological crises in AI value systems.

These are just a few examples: there are lots more. We aren't happy yet with our descriptions of any of these problems, and we're working with various people to explain ourselves better, and make it easier for people to understand what we're talking about and why we're working on these problems and not others. But nevertheless some people seem to grok what we're doing, e.g. I pointed Nik Weaver to the tiling agents paper stuff and despite not having past familiarity with MIRI he just ran with it.

Here's a comment that I posted in a discussion on Eliezer's FB wall a few days back but didn't receive much of a response there, maybe it'll prompt more discussion here:


So this reminds me, I've been thinking for a while that VNM utility might be a hopelessly flawed framework for thinking about human value, but I've had difficulties putting this intuition in words. I'm also pretty unfamiliar with the existing literature around VNM utility, so maybe there is already a standard answer to the problem that I've been thinking about. If so, I'd appreciate a pointer to it. But the theory described in the linked paper seems (based on a quick skim) like it's roughly in the same direction as my thoughts, so maybe there's something to them.

Here my stab at trying to describe what I've been thinking: VNM utility implicitly assumes an agent with "self-contained" preferences, and which is trying to maximize the satisfaction of those preferences. By self-contained, I mean that they are not a function of the environment, though they can and do take inputs from the environment. So an agent could certainly have a preference that made him e.g. want to acquire more money if he had less than $5000, and which made him indifferent to money if he had more than that. But this preference would be conceptualized as something internal to the agent, and essentially unchanging.

That doesn't seem to be how human preferences actually work. For example, suppose that John Doe is currently indifferent between whether to study in college A or college B, so he flips a coin to choose. Unbeknownst to him, if he goes to college A he'll end up doing things together with guy A until they fall in love and get monogamously married; if he goes to college B he'll end up doing things with gal B until they fall in love and get monogamously married. It doesn't seem sensible to ask which choice better satisfies his romantic preferences as they are at the time of the coin flip. Rather, the preference for either person develops as a result of their shared life-histories, and both are equally good in terms of intrinsic preference towards someone (though of course one of them could be better or worse at helping John achieve some other set of preferences).

More generally, rather than having stable goal-oriented preferences, it feels like we acquire different goals as a result of being in different environments: these goals may persist for an extended time, or be entirely transient and vanish as soon as we've left the environment.

As an another example, my preference for "what do I want to do with my life" feels like it has changed at least three times today alone: I started the morning with a fiction-writing inspiration that had carried over from the previous day, so I wished that I could spend my life being a fiction writer; then I read some e-mails on a mailing list devoted to educational games and was reminded of how neat such a career might be; and now this post made me think of how interesting and valuable all the FAI philosophy stuff is, and right now I feel like I'd want to just do that. I don't think that I have any stable preference with regard to this question: rather, I could be happy in any career path as long as there were enough influences in my environment that continued to push me towards that career.

It's as Brian Tomasik wrote at :

There are a few basic life activities (eating, sleeping, etc.) that cannot be ignored and have to be maintained to some degree in order to function. Beyond these, however, it's remarkable how much variation is possible in what people care about and spend their time thinking about. Merely reflecting upon my own life, I can see how vastly the kinds of things I find interesting and important have changed. Some topics that used to matter so much to me are now essentially irrelevant except as whimsical amusements, while others that I had never even considered are now my top priorities.

The scary thing is just how easily and imperceptibly these sorts of shifts can happen. I've been amazed to observe how much small, seemingly trivial cues build up to have an enormous impact on the direction of one's concerns. The types of conversations I overhear, blog entries and papers and emails I read, people I interact with, and visual cues I see in my environment tend basically to determine what I think about during the day and, over the long run, what I spend my time and efforts doing. One can maintain a stated claim that "X is what I find overridingly important," but as a practical matter, it's nearly impossible to avoid the subtle influences of minor day-to-day cues that can distract from such ideals.

If this is the case, then it feels like trying to maximize preference satisfaction is an incoherent idea in the first place. If I'm put in environment A, I will have one set of goals; if I'm put in environment B, I will have another set of goals. There might not be any way of constructing a coherent utility function so that we could compare the utility that we obtain from being put in environment A versus environment B, since our goals and preferences can be completely path- and environment-dependent. Extrapolated meta-preferences don't seem to solve this either, because there seems to be no reason to assume that they'd any less stable or self-contained.

I don't know what we could use in place of VNM utility, though. At it feels like the alternate formalism should include the agent's environment/life history in determining its preferences.

I also have lots of objections to using VNM utility to model human preferences. (A comment on your example: if you conceive of an agent as accruing value and making decisions over time, to meaningfully apply the VNM framework you need to think of their preferences as being over world-histories, not over world-states, and of their actions as being plans for the rest of time rather than point actions.) I might write a post about this if there's enough interest.

I've always thought of it as preferences over world-histories and I don't see any problem with that. I'd be interested in the post if it covers a problem with that formulation

Robin Hanson writes about rank linear utility. This formalism asserts that we value options by their rank in a list of options available at any one time, making it impossible to construct a coherent classical utility function.

Yeah, that was my first link in the comment. :-) Still good that you summarized it, though, since not everyone's going to click on the link.

Oops, I frankly did not see the link. The one time I thought I could contribute ...

Well, like I said, it was probably a good thing to post and briefly summarize anyway. If you missed the link, others probably did too.

I don't think of things like "what I want to do with my life" as terminal preferences - just instrumental preferences that depend on the niche you find yourself in. Terminal stuff is more likely to be simple/human universal stuff (think Maslow's hierarchy of needs)

I think you'll probably find Kevin Simler's essays on personality interesting, and he does a good job explaining and exploring this idea.

What I think is happening is that we're allowed to think of humans as having VNM utility functions ( see also my discussion with Stuart Armstrong ), but the utility function is not constant over time (since we're not introspective recursively modifying AIs that can keep their utility functions stable).

I recently saw an advertisement which was such a concentrated piece of antirationality I had to share it here. Imagine a poster showing a man's head and shoulders gazing inspiredly past the viewer into the distance, rendered in posterised red, white, and black with a sort of socialist realism flavour. The words: "No Odds Too Long. No Dream Too Great. The Believer."

If that was all, it would just be a piece of inspirational nonsense. But what was it advertising?

Ladbrokes. A UK chain of betting shops.

That is a hilariously apposite name for a chain of betting shops.

Isn't it? The first time I read about the British betting industry and Betfair & Ladbrokes, I had to look the latter up on WP to verify it was randomly named after a building and wasn't a mockery of their customers.

There is a lot of interest in prediction markets in the Less Wrong community. However, the prediction markets that we have are currently only available in meatspace, they have very low volume, and the rules are not ideal (You cannot leave positions by selling your shares, and only the column with the final outcome contributes to your score)

I was wondering if there would be interest in a prediction market linked to the Less Wrong account? The idea is that we use essentially the same structure as Intrade / Ipredict. We use play money - this can either be Karma or a new "currency" where everyone is assigned the same starting value. If we use a currency other than Karma, your balance would be publicly linked to your account, as an indicator of your predictive skills.

Perhaps participants would have to reach a specified level of Karma before they are allowed to participate, to avoid users setting up puppet accounts to transfer points to their actual accounts

I think such a prediction market would act as a tax on bullshit, it would help aggregate information, it would help us identify the best predictors in the community, and it would be a lot of fun.

Why would LWers use such a prediction market more than PredictionBook?

Good point . I actually didn't know about PredictionBook. Now that it has been pointed out to me, I see that there is already a decent option, so my suggestion would be less valuable. However, I still think it would be useful to have a prediction market that operates with Intrade rules. Whether that is worth writing the code is another matter..

I don't think karma matters as much as people think it does, but if that's the only reason, LW could be programmed to look on for a matching username and increase karma based on the scores or something, much more easily than an entire prediction market written.

That has the problem that people can inflate their scores by repeatedly predicting that the sun will rise tomorrow.

I think it's a very good idea. I also like the "tax on bs" metaphor. I like the idea of bullshitters getting punished! :)

I think it should be remembered, though, that wrt many predictions, luck is as least as important as skill/knowledge. Of course if you have many question the luck/noise element is reduced and the signal/skill element is strengthened, but it nevertheless is something to consider.

I would personally allow free account creation but give peo