I haven't devoted much time to this because I don't think anybody who has ever interacted with us in person has ever thought this was likely, and I'm not sure if anyone even on the internet has ever made the accusation - though of course some have raised the vague possibility, as you have. In other words, I doubt this worry is anyone's true rejection, whereas I suspect the lack of peer-reviewed papers from SIAI is many people's true rejection.

An "outside view" might be to put the SI in the reference class of "groups who are trying to create a utopia" and observe that previous such efforts that have managed to gain momentum have tended to make the world worse.

I think the reality is more complicated than that, but that might be part of what motivates these kind of questions.

I think the biggest specific trust-related issue I have is with CEV - getting the utility function generation process right is really important, and in an optimal world I'd expect to see CEV subjected to a ... (read more)

6orthonormal9ySkepticism about SIAI's competence screens off skepticism about SIAI's intentions, so of course that's not the true rejection for the vast majority of people. But it genuinely troubles me if nobody's thought of the latter question at all, beyond "Trust us, we have no incentive to implement anything but CEV". If I told you that a large government or corporation was working hard on AGI plus Friendliness content (and that they were avoiding the obvious traps), even if they claimed altruistic goals, wouldn't you worry a bit about their real plan? What features would make you more or less worried?
0wedrifid9yAm I to take this as a general answer to the overall question [http://lesswrong.com/r/discussion/lw/8c3/qa_with_new_executive_director_of_singularity/56re] of trustworthiness or is this intended just as an answer to the specific example?

Q&A with new Executive Director of Singularity Institute

by lukeprog 1 min read7th Nov 2011182 comments


Today I was appointed the new Executive Director of Singularity Institute.

Because I care about transparency, one of my first projects as an intern was to begin work on the organization's first Strategic Plan. I researched how to write a strategic plan, tracked down the strategic plans of similar organizations, and met with each staff member, progressively iterating the document until it was something everyone could get behind.

I quickly learned why there isn't more of this kind of thing: transparency is a lot of work! 100+ hours of work later, plus dozens of hours from others, and the strategic plan was finally finished and ratified by the board. It doesn't accomplish much by itself, but it's one important stepping stone in building an organization that is more productive, more trusted, and more likely to help solve the world's biggest problems.

I spent two months as a researcher, and was then appointed Executive Director.

In further pursuit of transparency, I'd like to answer (on video) submitted questions from the Less Wrong community just as Eliezer did two years ago.


The Rules

1) One question per comment (to allow voting to carry more information about people's preferences).

2) Try to be as clear and concise as possible. If your question can't be condensed into one paragraph, you should probably ask in a separate post. Make sure you have an actual question somewhere in there (you can bold it to make it easier to scan).

3) I will generally answer the top-voted questions, but will skip some of them. I will tend to select questions about Singularity Institute as an organization, not about the technical details of some bit of research. You can read some of the details of the Friendly AI research program in my interview with Michael Anissimov.

4) If you reference certain things that are online in your question, provide a link.

5) This thread will be open to questions and votes for 7 days, at which time I will decide which questions to begin recording video responses for.


I might respond to certain questions within the comments thread and not on video; for example, when there is a one-word answer.