Less Wrong Polls in Comments

You can now write Less Wrong comments that contain polls! John Simon picked up and finished some code I had written back in 2010 but never finished, and our admins Wesley Moore and Matt Fallshaw have deployed it. You can use it right now, so let's give it some testing here in this thread.

The polls work through the existing Markdown comment formatting, similar to the syntax used for links. Full documentation is in the wiki; the short version is that you can write comments like this:

What is your favorite color? [poll]{Red}{Green}{Blue}{Other}

How long has it been your favorite color, in years? [poll:number]

Red is a nice color [poll:Agree....Disagree]

Will your favorite color change? [poll:probability]

To see the results of the poll, you have to vote (you can leave questions blank if you want). The results include a link to the raw poll data, including the usernames of people who submitted votes with the "Vote anonymously" box unchecked. After you submit the comment, if you go back and edit your comment all those poll tags will have turned into [pollid:123]. You can edit the rest of the comment without resetting the poll, but you can't change the options.

It works right now, but it's also new and could be buggy. Let's give it some testing; what have you always wanted to know about Less Wrongers?

302 comments, sorted by
magical algorithm
Highlighting new comments since Today at 4:28 AM
Select new highlight date
Moderation Guidelines: expand_more

Which poll answer will receive the largest number of responses? [pollid:38]

After one month and 120 responses, I'm considering this poll closed. The results are:

1) The third one: 21%
2) The fifth one: 15%
3) The second one: 14%
4) The first one: 32%
5) The fourth one: 18%

A chi-squared test says that these results are non-uniform, with a p-value of 0.02.

The correct answer, #5 "the fourth one", was chosen by 18% of respondents. The most common answer was #4, "the first one".

This poll idea was taken from a gamefaqs poll which was linked on LW last year. The results of that poll (which had a much larger sample size) were:

1) The third one: 17%
2) The first one: 24%
3) The last one: 21%
4) The second one: 26%
5) The fourth one: 11%

My hypothesis about that poll was:

The first option is most salient, by virtue of being first. Level 0 players will tend to choose option 1. Level 1 players will realize that this is what level 0 players will do, so they will tend to choose option 2 ("the first one"). Level 2 players will realize that this is what level 1 players will do, so they will tend to choose option 4 ("the second one"). Level 3 players will realize that this is what level 2 players will do, so they will tend to choose option 5 ("the fourth one"). Apparently there are lots of level 1 & 2 players, but very few level 3 players.

That hypothesis predicts that the most common responses on the LW poll would be the level 1 response, #4 "the first one", and the level 2 response, #5 "the fourth one". The data provide partial confirmation of this hypothesis; in terms of levels the most common responses to the LW poll were:

32% Level 1 (#4 "the first one")
21% Level 0 (#1 "the third one")
18% Level 2 (#5 "the fourth one")
15% Level 3 (#2 "the fifth one")
14% Level 4 (#3 "the second one")

And for the gamefaqs poll:
26% Level 2 (#4 "the second one")
24% Level 1 (#2 "the first one")
21% Level 4 (#3 "the last one")
17% Level 0 (#1 "the third one")
11% Level 3 (#5 "the fourth one")

Some people don't pick very good schelling points.

Great features ! Thanks and congrats to those who made it happen.

One suggestion : you can't see the result until you voted, I guess it's not to bias/anchor the answer, but then it would be nice to add an option "I don't plan to vote, let me see the results", so someone who doesn't want to vote for any reason can still access the outcome. Or else, there is a risk of people not wanting to vote but wanting to see the outcome will vote "at random" and skew the result.

In the meantime, the effect can be simulated by proper selection of options. Example:

Which is your favourite superhero? [pollid:37]

Excluding the ponies (which I didn't vote on, because I am one of sixteen people remaining on the Internet who doesn't pony yet), this is the earliest radio poll where the sum of the numbers in the column matches the "Total" number at the bottom.

because I am one of sixteen people remaining on the Internet who doesn't pony yet

I am glad that you used the word "yet": accepting the possibility of getting better is an essential part of overcoming a problem.

I am one of sixteen people remaining on the Internet who doesn't pony yet

It's a relief to know I'm not the only one...

Results: 4+16+2+16+1+27(last option) = 144? WTF?

The issue of the bug in question has been discussed already, in this very thread -- but even if it hadn't, I don't think the discovery of a bug should stun you this much.

Somebody sounds grouchy :/ In fact, it would be completely unsurprising if I had read the other comments. Oops.

I love what this poll reveals about LW readers. Many sympathise with Batman, because of his tech/intellectual angle. The same with Iron Man, but he's a bit less cool. Then two have heard of superman, and most LWers are male. And most of us don't care.

What would be really funny is if, when you select "I don't care, but I'd like to see the results" you see that everyone else filled it out the same way.

[pollid:48]

EDIT: uh oh, you can't fix typos in polls. I don't know if that is a good thing or a barrier to lulz

I'm not conflating options, because I'm not really comparing Spiderman to not-Spiderman, I'm comparing Spiderman to the poll results in CCC's comment. The effect is the same as editing the previous poll to add Spiderman.

Also apparently it's "Spider-Man".

You can clone polls. I wonder what the other polls are.

[pollid:48]

No wait, you can't. Someone thought this thru.

Spiderman! Why isn't Spiderman on there? I bet he'd be way more popular than that Flash guy whoever he is.

...Spiderman! I knew I was forgetting someone!

And the poll options cannot be edited after the fact (though the question can be).

Funny that Iron Man was the only Marvel hero on the list. If I had to pick one Marvel hero... he wouldn't have been the one.

He did have a movie out recently. I think that's why his name drifted into my head at the time of the poll.

Spiderman! Why isn't Spiderman on there? I bet he'd be way more popular than that Flash guy whoever he is.

On the other hand even complete anonymity could be sufficient for those who consider Spiderman a kind of pathetic whiny child.

This is a problem. Ideally there would be a separate button next to the "vote" button (a forum that I read has that feature, with the button labeled "View Results (Null Vote)"). Second-best would be to allow people to submit a blank vote (which is not as good, since it's not obvious to people that they have that option), but it currently does not work that way (even though the OP seems to say that it does).

Minimize the expected square of the distance between your answer and 80% of the mean of the answers chosen: [pollid:8]

I see a 'Total 123' but the table and chart only show 2 votes. The raw data also have 123 entries.

Yeah, it looks like there's something seriously broken about this poll code. I'm seeing 159 total votes, and only 13 visible votes.

Hey everyone, I just voted, and so I can see the correct answer. The average is 19.2, so you should choose 17%!

Or maybe that's what I want you to think I'd say...

The noise in my simulations quickly drown out any actual logic and the markov chain reaches its stable distribution.

I guessed "the only winning move is not to play"

(I didn't guess. rationalization: I didn't want to do the thinking, and can't see the results anyway)

A function to automatically compute the averages should be implemented.

awk is made for this, but it took me a few minutes to whip this up in java. I figured if numeric polls are used in the future, this can be used as a code-base. The indentation isn't coming through, but any IDE will fix that for you.

This doesn't work on arbitrary numeric entry polls, but for those, you can gather the statistics as you go along, putting it in the GATHER loop

EDITED to fix serious bug.

usage: paste this into PollStat.java, compile it. then run

java PollStat < poll.csv.txt

So far, the winners are endoself and army1987. I wasn't far off.

~~~~~

import java.io.*;
import java.util.*;

public class PollStat {
  public static void main(String[] args) {
    try {
      BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
      String line = br.readLine();
      int[] counts = new int[12]; // PUT NUMBER OF POLL OPTIONS HERE
      int end, begin;
      int linen = 0;
      while (line != null && line.charAt(0) == '#') {
        line = br.readLine();
        linen++;
      }
      while (line != null) { // GATHER LOOP
        linen++;
        end = line.lastIndexOf("\",\"");
        begin = line.lastIndexOf("\",\"", end-1);
        try {
          line = line.substring(begin+3, end);
        } catch (Exception e) {
          System.out.println("At line number "+linen+":");
          e.printStackTrace();
          System.exit(1);
        }
        Integer c = Integer.parseInt(line);
        counts[c]++;
// For arbitrary numeric responses, don't use counts.
// just continuously gather your statistics. Alternately, make a list of Doubles or something.

        line = br.readLine();
      }
      br.close();


      int total = 0;
      int resp = 0;

      int[] numbers = new int[]{100, 64, 41, 26, 17, 11, 7, 4, 3, 2, 1, 0}; // PUT POLL OPTIONS HERE
      for (int i = 0; i < c.length; i++) {
        System.out.println(numbers[i]+": "+counts[i]);
        total += numbers[i] * counts[i];
        resp += counts[i];
      }
      System.out.println("total: "+total);
      System.out.println("average: "+((double)total)/resp);

      } catch (IOException ioe) {
        ioe.printStackTrace();
        System.exit(2);
      }
  }
}

Not really -- if you check out the wiki page this type of poll is meant for discrete options, not for numbers or probabilities. For probabilities the "poll:probability" type should be used, which does automatically compute averages and medians.

Ahrgh, if only everyone was running TDT…

Or UDT. Or CDT. Or EDT.

Dunno... If you treat it as a zero-sum game (i.e. you don't only want your answer to be close to 80% of the average answer, but you also want other people's answers to be far from it) it's not obvious to me that you should vote 0.

Dunno... If you treat it as a zero-sum game (i.e. you don't only want your answer to be close to 80% of the average answer, but you also want other people's answers to be far from it) it's not obvious to me that you should vote 0.

I was granting for the purpose of responding that loup-vailant's clear assumption that normal game theory principles apply---each agent is interested only in the payoffs to itself to the exclusion of all else and the payoffs are such that it gets 0 for being wrong and >0 for being right.

It so happens that my own actual response (100%) doesn't conform to those assumptions. In fact my original reply to:

Minimize the expected square of the distance between your answer and 80% of the mean of the answers chosen:

... was "No", and my original reply to loup-vaillant pontificated about the complete lack of payoff to any of the radio buttons. However I abandoned that point because the point about it not mattering whether the other guy is using CDT or TDT actually matters (somewhat).

In this game (ie. with an actual assumed payoff for correct and no negative payoff for other's success) the Nash equilibrium (and the outcome that a group of all CDT agents would pick) also happens to be pareto optimal. In fact, it outright gives the maximum possible payoff to every individual. Even inferior decision theories can pull that off.

Yes, but whichever decision theory you're using, you need to be ready for the few people voted for 100. Someone's going to do something to ruin it for everyone. And it wasn't just a few who ruined it - vs rirelbar jub'q ibgrq yrff guna sbegl bar ibgrq mreb, gur nirentr jbhyq or nyzbfg rknpgyl rvtug.

Forgive me for being new to the site, but I've see this kind of writing

rirelbar jub'q ibgrq yrff guna sbegl bar ibgrq mreb, gur nirentr jbhyq or nyzbfg rknpgyl rvtug.

in several places. How is it translated back to readable English?

It's rot13, a shift cipher typically used around here to obscure spoilers and spoiler-like information. Cut and paste it into rot13.com, install the d3coder extension for Chrome or something similar for another browser, or (if you like tedium) decipher it yourself.

Yes, but whichever decision theory you're using, you need to be ready for the 6 people voted for 100.

That would be why this subthread was based on a lament.

I would like a user preference that makes it possible to vote non-anonymously by default. But it's low priority - this is really awesome as is!

(Or even it remembering whether you chose to vote non-anonymously in the last poll you took.)

Oooh great idea. Bugs / Suggestions:

  1. The answers are transformed into this tiny little poll code with a poll id, so I assume it's being saved to a database. However, the questions are not being saved with them. I can edit my question after the poll has been answered. This may result in some pranks later where you ask some obvious question like "does the earth revolve around the sun" or whatever and everyone answers "yes" and then you can change your question to "are you a Scientologist?" and you will see everyone's votes saying "yes" to that. Much more malicious changes are possible, of course. Also, if the questions aren't stored in the database with the answers, you won't have as many options later for doing cool things with your database full of polls.

  2. The code: [Poll] does not work because it's upper case, but this is not an obvious reason for poll failure, so one may end up wasting lots of time trying to figure it out or make annoying requests for support. Making this case insensitive is probably a good idea.

  3. The poll seems designed for very short answers. My elitism poll results look bad for that reason.

The poll seems designed for very short answers.

Feature, not bug.

The raw poll data is sent with "Content-Disposition: attachment", which causes firefox to download it instead of letting me view it in the browser. Is this deliberate?

BUG: The display of the data for the multiple-choice polls seems to reset at some point, though I can still see the complete raw data when clicking at them... e.g. right now, though 68 people have voted for the "best pony", at the display I only see the choice of the 68th person (Applejack) having a single vote, and all the other choices are falsely at zero.

Similar things with other polls.

Confirmed. The issue is in an interaction between the polling code and Reddit's custom ORM which causes vote-totals to be cached, but not persisted to the database correctly. I have a fix, which I'm testing now. All polls created before the fix is applied will be affected; it'll be possible to restore them, but it'd take some work which isn't a priority for me.

I've just deployed a fix that will apply to all new poll votes. Thanks jimrandomh for passing on the bug report and initial patch.

Here's a screenshot of that with my poll:

I still have access to the raw poll data.

Which of the following is true? [pollid:45]

Pick your answer to this poll at random: [pollid:39]

I used random.org to generate my answer.

But, when I submitted it, I got the following:

First Answer 0 (0%)
Second Answer 0 (0%)
Third Answer 0 (0%)
Fourth Answer 1 (2%)
Fifth Answer 0 (0%)
Total 58 (100%)

The raw data contained all the 58 rows, however. Seems like there might be a bug in the result-rendering code.

To anyone thinking this is not random, with 42 votes in:

  • The p-value is 0.895 (this is the probability of seeing at least this much non-randomness, assuming a uniform distribution)

  • The entropy is 2.302bits instead of log(5) = 2.322bits, for 0.02bits KL-distance (this is the number of bits you lose for encoding one of these votes as if it was random)

If you think you see a pattern here, you should either see a doctor or a statistician.

Looks like we're better at randomness than the rest of the population. If I asked random people for a random number from 1 to 10, I wouldn't be surprised to see substantially less than 3.322 bits of entropy per number (e.g., many more than 10% of the people choosing 7).

Ha. I fail at random. In my defence, the universe is probably deterministic anyway.

Is (the seconds' figure in my watch) mod 5 random enough?

I used the least significant digit on my time-remaining-to-full-charge. And ended up propping up the most populated entry.

I needed 3 random bits (and threw out any overflow), which I got by checking whether arbitrary words or phrases I thought of had an even or odd number of letters. That's the most random completely mental (heh) way I know of, I wonder if there are others.

... you could have done it more-reliably evenly by taking the mod 5 of the phrase/word length.

Considering that the average word length in English is about five letters, I suspect that'd be quite far from being uniformly distributed.

Average is irrelevant. What's relevant is the standard deviation.

Since standard deviation goes as the square root of the number of items being added, phrase length for any reasonably-sized phrase, so long as it wasn't a line of poetry, should be pretty evenly distributed.

It's not obvious to me that it's unbiased. My gut feeling suspects that if I randomly chose a word it'd be more likely to have an odd than an even number of letters.

After one month and 118 responses, I'm considering this poll closed. The results are:

1) 17%
2) 21%
3) 20%
4) 24%
5) 18%

A chi-squared test says that these results do not differ significantly from uniform random responding, with a p-value of 0.78.

The main reason why I ran this poll was because I thought it might have implications for the trickier poll above. It is interesting the option #4 was the most common response in this poll, that poll, and the gamefaqs poll which that poll was based on. #4 may seem especially random, and some respondents in the other polls may have just been trying to answer at random. But this poll ended up not providing much information about that; to test it we'd need a larger sample size, and preferably a poll where respondents did not use external sources of randomness.

I think this would be even more interesting as "pick at random, without an external source of randomness". Sure you can get random numbers from random.org, your computer or the seconds on your watch (a nice idee), but those just blur the effect of mind-generated random numbers.

Question: what's a reasonable prior over the probability distribution of poll answers? Because I downloaded the raw data, and it says:

  1. 15
  2. 22
  3. 21
  4. 24
  5. 18

...and I'm not sure what would constitute reasonable priors for the uniform distribution hypothesis versus the "aversion toward First Answer" hypothesis versus the "aversion toward First Answer and Fifth Answer" hypothesis.

Your question is confused. The uniform distribution hypothesis only requires that the (assumed infinite) population picks the answers independently with equal probability. Under this hypothesis, the observed poll answers (for a fixed number of respondents) will follow a multinomial distribution with parameters (0.2, 0.2, 0.2, 0.2, 0.2). A typical realization will not have an equal number of respondents giving each answer, although asymptotically the empirical frequencies will converge to equality.

Anyways, as a Bayesian, the better question is what should my posterior belief about the response probabilities be after running the poll and updating off the answers? The canonical way to do this would be to put a Dirichlet prior over the response probabilities. By the miracle of conjugacy, your posterior distribution will itself by a (generally different) Dirichlet distribution.

By taking the expectation of indicator variables like I{"probability of First Answer under 0.2"} under the posterior, you can figure out what degree of belief you must give to statements like "respondents have an aversion toward First Answer".

That makes sense - I had imagined doing something similar, but I had never heard of Dirichlet priors.

Happy this helped. The Dirichlet-multinomial model gets relatively little attention because it adds nothing really new to the beta-binomial model for polls with just two responses. It's easy to find lots of introductory, chatty introductions to the beta-binomial like this one or this one if you want to learn more...

My own feelings on the matter are that if you don't know what prior to have, compute worst-case bounds.

In this case, the model that maximizes the probability of seeing this data is that each answer is 15% likely to be 1, 22% likely to be 2, 21% likely to be 3, 24% likely to be 4, and 18% likely to be 5. We can compute the probability of seeing this data under this model, and also under the "all answers are equally likely" model, and conclude that our worst-case model makes us only 3.61 times as likely to see this data.

In particular, any other hypothesis you might have can only receive this little evidence, relative to the uniform distribution hypothesis; and I believe in close-to-uniformity enough that I'm not going to be swayed by what is fewer than 2 bits of evidence.

Thanks! I didn't think of that particular brainhack - I'll be sure to use it in the future.

Most voters so far have probably voted False to this question: [pollid:16]

This doesn't look right: http://screencast.com/t/qpRGihBG

The raw data says there are 13 votes for "0" and 20 votes for "1".

Looking at the raw data, it seems that at some point the True and False counts got reset, but then kept increasing as normal. The same thing happened in this poll and this one but not others.

Bug report: The right navigation bar on this page has scooted down as if it's being pushed out of alignment by something too wide in the comments section. The comments seem to have the same width as they normally do and I but perhaps the polls are interfering with the layout in some way?

FFX 15.0.1 W7

If you choose an answer to this question at random (using a uniform distribution), what is the probability that you will be correct? [pollid:51]

Neat. Thanks!

This deserves karma. For fun, enter how much you think this post will get. [pollid:6]

After my vote:

" Mean 43.5 Median 75.0 Total votes 2"

Well this is mathematically impossible... My guess is the median isn't properly calculated for even numbers of votes.

Thanks for spotting this! I looked into it, and it seems to be double-counting the most recent result when computing the median. It's an order-of-initialization issue; it thinks it's getting all the results except the new one, adding it, then taking the median, but it's actually getting a list of all the results. The fix is straightforward; I'll email the admins to apply it.