Survey advice

[-]noggin-scratcher3y238

There is also scope for helping people think through a thing in a way that they would endorse, e.g. by asking a sequence of questions.

As aptly demonstrated:

[-]Valdes3y50

I don't think this is a good illustration of point 6. The video shows a string of manipulative leading questions, falling short of the " in a way that they would endorse" criteria.

When people understand that a string of questions is designed to strong arm them into a given position they rarely endorse it. It seems to me that point 6 is more about benevolent and honest uses of leading questions.

Admittedly, I am making the assumption that " in a way that they would endorse" means "such that if people understood the intent that went into writing the string of questions in that way they would approve of the process".

[-]Alexander Gietelink Oldenziel3y9-3

I feel 4. can be explained by humans not having probability distributions on future events but something more like infradistrbutions/imprecise distributions. This a symptom of larger problem of Bayesian dogmatism that has taken hold of some parts of LW/rationalists.

Let me xplain how this works:

To recall: an imprecise distribution is the convex (closed) hull of a collection of probability distributions ${p_{i}}_{i \in I}$ . In other words it combines 'Knightian' uncertainty with probabilistic uncertainty.

If you ask people for 10%,50%,90% chance of AI happening you are implicitly asking for the worst case: i.e. there in at least one probability distributions $p_{i}$ such that $p_{i} ($ AGI) =10%,50%,90%

On the other hand when you ask for a certain event to happen for certain in 10,20,50 years you are asking for the dual 'best case' scenario, i.e. for ALL probability distributions $p_{i} \in I$ what probability $p_{i}$ (AGI in 10y), $p_{i}$ (AGI in 20y), $p_{i}$ (AGI in 50y) is and taking the minimum.

[-]JBlack3y6-1

This does seem to be a useful insight, though I don't think it's anywhere near so precise as that.

Personally, the Knightian uncertainty completely dominates my timeline estimates. If someone asks for which year the cumulative probability reaches some threshold, then firstly that sounds like a confusion of terms, and secondly I have or can generate (as described) a whole bunch of probability distributions without anything usable as weightings attached for each. Any answer I give is going to be pointless and subject to the whims of whatever arbitrary weightings I assign in the moment, which is likely to be influenced by the precise wording of the question and probably what I ate for breakfast.

It's not going to be the worst case - that's something like "I am already a simulation within a superintelligent AGI and any fact of the matter about when it happened is completely meaningless due to not occurring in my subjective universe at all". It's not going to be the best case either - that's something like "AGI is not something that humans can create, for reasons we don't yet know". Note that both of these are based on uncertainties: hypotheses that cannot be assigned any useful probability since there is no precedent nor any current evidence for or against them.

It's going to be something in the interior, but where exactly in the interior will be arbitrary, and asking the question a different way will likely shift where.

[-]Raemon3y21

Curated. I'm interested in this both from the perspective of personal epistemics and group epistemics. Surveys are a tool for figuring out things about the world, and they tend to also be a way to get-on-the-same-page about how the world looks.

Thanks Katja for sharing a bunch of lived experience on how execute surveys well. :)

[-]papetoast2y10

Guidedtrack.com

is now https://www.guidedtrack.com/

[-]Emily Thomas3y10

Some of these are strikingly similar to advice for how to interview users when designing user friendly software.
I guess it makes sense that there's some cross over.

I like it!

[-]kerspoon3y*10

This is not my research area but this list looks really relevant. Thanks for posting it!

For those that do not know. Survey methods and survey analysis is a field of academic research in itself. There are people who specialise in this topic - and hence we can learn from them or pay them to consult on the design of our surveys.

E.g. SMAG (survey methods and analysis group) at the university of Manchester. And NCRM (national centre for research methods) are two I know if in the UK.

There is a "journal of survey statistics and methodology" and "of social research methodology".

And undergraduate textbooks such as

Lohr's sampling design and analysis
Oppenheim's questionnaire design
Czaja & Blair's designing surveys
Fowler's survey research methods

[-]Zach Stein-Perlman3y10

Good post.

Interest in surveys doesn’t seem very related to whether a survey is a good source of information on the topic surveyed on. One of the strongest findings of the 2016 survey IMO was that surveys like that are unlikely to be a reliable guide to the future.

Can you say more?

How are these two sentences related?
The first sentence seems plausible, but why do you say it?
The second sentence seems plausible, but why do you say it? (Is it just because many responses were internally inconsistent and/or unreasonable?)

[-]KatjaGrace3y40

Second sentence:

People say very different things depending on framing, so responses to any particularly-framed question are presumably not accurate, though I'd still take them as some evidence.
People say very different things from one another, so any particular person is highly unlikely to be accurate. An aggregate might still be good, but e.g. if people say such different things that three-quarters of them have to be totally wrong, then I don't think it's that much more likely that the last quarter is about right than that the answer is something almost nobody said.

First sentence:

In spite of the above, and the prior low probability of this being a reliable guide to AGI timelines, our paper was the 16th most discussed paper in the world. On the other hand, something like Ajeya's timelines report (or even AI Impacts' cruder timelines botec earlier) seem more informative, and to get way less attention. (I didn't mean 'within the class of surveys, interest doesn't track informativeness much' though that might be true, I meant 'people seem to have substantial interest in surveys beyond what is explained by them being informative about e.g. AI timelines'
)

[-]Dagon3y-20

Point 10 should be 1 and probably a variant should be 2. And they're not even all that good at finding out what people think. They can sometimes find out how people feel or what their current reaction is.

To point 14, it depends on who's doing the rating. I'll point out that survey design and interpretation is a pretty big business - there's a reason Qualtrics charges so much (and that SAP paid $8B for the company), and a related reason that competitors universally suck - the actual presentation is the easy (and non-profitable) part. The design and analytics are difficult and command a lot of revenue.

LESSWRONG
LW

LESSWRONG
LW

93

93

93