Number of Members on LessWrong

[-]Vladimir_Nesov13y380

In July 2012, more than 600 different users posted a comment;
since March 2012, about 1600 different users;
since October 2011 — 2300;
since May 2011 — 2900;
since December 2010 — 3400;
since May 2010 — 3900;
since August 2009 — 4400.

Since the beginning, including the comments imported from Overcoming Bias, with some duplicates (people sometimes re-registered with different usernames when moving to LW, and the same username on Overcoming Bias was imported as multiple different usernames on LW if it corresponded to different emails), comments were posted under about 7500 different usernames.

Of the 4400 users who commented since August 2009, 1390 have written at least 10 comments;
900 users — at least 25 comments;
630 users — at least 50 comments;
429 users — at least 100 comments;
225 users — at least 250 comments;
134 users — at least 500 comments;
57 users — at least 1000 comments;
13 users — at least 2500 comments.

Wedrifid has written more than 10000 comments.

(Based on a wget'ed dump of all LW comments.)

[-]Epiphany13y60

One flaw: You're not locating anywhere near all of the people that registered using this method because I bet a lot of people have never commented. In one website's database that I've got access to, almost 70% of the users register without ever doing the expected main activity. Unless you spider your copy of all the comments to cache home pages, follow the links off of friends lists and include other links to home pages around the internet (like Google does, which is why I chose Google instead of wget), you're probably missing a huge proportion of the profiles. You may argue that counting active users is more relevant than counting total members, but these guys might be voting on our posts and comments, and if they outnumber us, they've got more influence over the content than we do.

[-]A1987dM13y40

In the English Wikipedia the number of registered account dwarfs the number of accounts actually in use by a couple of orders of magnitude.

[-]NancyLebovitz13y110

On the other hand, if you're interested in the number of habitual readers or habitual posters, then the number of members is going to be much higher.

[-]Epiphany13y40

I think my main interest in knowing how many users there are went something like this:

OMG A COMMUNITY OF RATIONALISTS!!!! And it is not small??? What the? Well how many are there?

[-]NancyLebovitz13y40

Lots. For what it's worth, I've quit checking all the recent comments because there are too many of them.

[-]Epiphany13y-20

Lol. You are never going to find this, are you?

[-]Viliam_Bur13y80

However, I still check my messages.

But you are never going to find this! :P

[-]Dorikka13y00

And Epiphany might not either. :)

[-]falenas10813y00

I usually check for children of anything I commented on for a couple days after I posted, if the original comment got a response.

[-]NancyLebovitz13y70

However, I still check my messages.

[-]NoSignalNoNoise13y80

The most recent LessWrong census had 1090 responses, so I think that would be a reasonable lower bound for the size of the community.

[-]Randaly13y50

That is surprisingly low- sitemeter indicates that LessWrong receives ~12,000 daily visits, and ~14,000 today. (I assume that that figure refers to unique visitors.) So there are probably ~3K lurkers.

Potentially confounding factors: People might visit LW from multiple computers/smartphones; people might have multiple accounts; many of those accounts are either spammers or from people who no longer visit LW. I'm not sure which direction these factors would bias the overall number of lurkers in.

[-]Epiphany13y50

In my experience, internet forums tend to have many times as many lurkers as posters. Also, it's a bit tricky to take all the "noise" out of the data, as you're suggesting, and another thing to add to the list of noisemakers are the search engine crawling robots. They don't all use the term "robot" in their user agents, and anybody can build a robot, so it's hard to filter them all out.

[-]Emile13y50

But you reach the last page of results at page 55, which suggests there are around 550 users.

[-]Epiphany13y10

I love how you guys explore every aspect of a thing. (: That may be a limit that Google has to either save on resources or to prevent rival search engines from downloading their whole database (or, a limit put into place BECAUSE rival search engines were sucking down their database, and it was taking up a lot of resources). I've seen other companies figure out who their greediest users are and, upon realizing that the population that takes up the most resources brings the least return on investment, put limits on them. That's what this looks like to me.

[-]Emile13y80

The "9000 results" is probably not a very accurate estimate - from "Google result counts are a meaningless metric":

The basic problem with the Google hit count reported in search results, particularly for phrases and searches using "AND" or "OR" operators, is that it is an estimate. It's not actually a count of anything, at all. It's the result of a calculation based solely upon the words that the query comprises, as Kevin Marks notes. Google explicitly states that it's an estimate, although it is coy about what that estimate is actually based upon. To quote one un-named Google employee, "these are all estimates, and we just haven't tried that hard to make the estimates precise". A named Google employee said much the same after this frequently given answer had been around for some years.

For example: When Google Web reports 17,200 results for the string "de Boyne Pollard" (as it does at the time of writing this Frequently Given Answer), it hasn't searched its entire database to count all of the pages that match that string. That would be very inefficient, considering that it only needs to find (by default) 10 matches in its database in order to return a result page, and that many people don't go beyond the first few pages (or even the first page) of results. What it has done is taken the individual words "de", "Boyne", and "Pollard", and, using the word frequency tables that the Google Web spider generates when it crawls the World Wide Web, produced, from the frequencies with which those three particular individual words occur, an estimate of the number of pages that probably would match.

To demonstrate for yourself that these estimates are meaningless numbers, take a few searches and click on the "Next" button to bring up further pages of results until you reach the last page. You'll see that the actual number of results, known once you reach the last page, will almost always be nothing like the estimated number of results that appeared on all of the prior pages.

Even the actual page count isn't necessarily correct. In part this is because Google caps all queries at 1000 results, and in part it is because of several other other problems with the Google hit count, both estimated and final, that exist.

(The linked page has more sources for this)

[-]DaFranker13y00

Thanks! I've always wondered where those numbers came from, but never taken the time to find out.

[-]Epiphany13y-30

If Google didn't search it's entire database, this supports my theory that there are probably "over 9,000 members" - I did clearly say that was on the low side. If Google only totals only SOME of the results (until it's clear that the user wants more results, or up to it's limit for resource conservation) this also supports my assertion.

Search Term Interpretation:

As for the issues with word interpretation - I knew about that, so I restricted my search to a specific URL, not text within pages. The entire purpose of Google's "site:" code is to restrict the query to a particular website, not to use those words as it would a text search. IF it's breaking the url up into separate words and checking what it's got for those, firstly, that would fail to restrict the search to a specific site and therefore make that functionality bugged, and secondly even if it did that only for the counter, the word "user" would certainly return way more results than 9,000. The term "user" gets 8 billion hits, and "lesswrong" gets 51,700 - if it's totaling site: searches that way, it would get billions of results and it didn't. Assuming it's not bugged, a misinterpretation of the "site:lesswrong.com/user" code is N/A. Since every single user page contains the phrase "comments" and "submitted", if it had broken my exact phrase exclusions into parts, I'd have gotten zero results. See for yourself by trying:

"site:lesswrong.com/user" -comments

It was not by accident that I used the query that I did.

Is my point unsupported?

IF I were trying to support some sort of important point with this user total, I would agree with the link that it is not scientific evidence and quit using it to support points, but this is N/A because if you look closely, you'll see that I am not using this as support to convince anybody of anything. My entire purpose was to verify to myself my perception that LessWrong isn't just someone's personal website with their buddies on it, that a significant number of people have actually gathered around themes like rational thought. I was overjoyed when I discovered this and wanted to share. Maybe this post will get the attention of someone who has the ability to issue a count command to the database. That's the only way we can know for sure. Though, of course, the user totals will change over time, becoming inaccurate quickly. Hopefully by increasing. (:

[-]John_Maxwell13y20

There are lots of lurkers on Less Wrong:

http://lesswrong.com/lw/1np/attention_lurkers_please_say_hi/

Another problem with your methodology is that prolific users typically have many pages associated with them that lack the text "submitted by" or "comments by" on them. You can access these pages by going the user's main page, scrolling down, and clicking the little "Next" link in the lower left.

[-]Epiphany13y-20

Multiple pages aren't being counted. From what I understand, Google doesn't just follow dynamically generated next links like that. It spiders, going around in a web-like pattern. How many times would it end up visiting the same pages if it followed every comment to it's original discussion? A lot. That would be a waste of resources.

To test this, I looked at the url that appears when you press the next button. The site adds some pagination variables into the URL. The word "count" appears. So, you can do the following query and observe the following things:

site:lesswrong.com/user -"submitted by" -"comments by" -count

site:lesswrong.com/user -"submitted by" -"comments by" -com (for comparison)

And observe:

A. It does not divide the number of results into a small fraction of the original number like you'd expect it to. We're comparing 9,820 total users with the original method (at this moment) with 9,460.

B. Removing "com" from the query shows zero results which verifies that adding -count would be removing pages generated in those next links, had they been included.

C. If you click on random pages of Google results, you won't see those count and after variables in the URLs (Or at least I didn't and I feel fairly confident that they won't be there.)

D. If Vladmir is correct in this post then just looking at one of those lines where the user's comments are totaled (the line where 900 have 25 comments) reveals that, by removing "count" from the query, we should have lost at least 1800 from the total. Nowhere near that many were lost, and a lot more should have been lost than that because I only subtracted a tiny fraction of the comments pages on this site in the example.