LW Open Source – Getting Started

by Raemon5 min read6th May 201829 comments


ProgrammingSite Meta
Personal Blog

LessWrong 2.0 is open source, but so far we haven’t done a great job at making contributing a good experience. Here’s our first attempt at fixing this, which covers the basics of "how to contribute" and "high level overview of the site architecture."


Getting Started

Assumed Background Knowledge

Currently, this guide assumes that you know your way around a terminal and github, and have worked with javascript, html, sass/css, and ReactJS.

If you know at least some of those things and are excited to contribute, you can probably make decent headway (and if you run into specific obstacles, comment on them on this post or ping us on intercom, and we’ll see if we can update things to be more clear)

Setting up Locally

[Note: So far, everyone who's installed Lesswrong2 locally has been using a fairly modern MacBook. If you are running Windows or Linux you'll likely run into problems. (Trying anyway and responding here with your solutions will be helpful to get a better sense of how to install on a variety of machines, but it may be a bit more annoying to get started)]

LessWrong is a fork of VulcanJS's example-forum. We've had to make some changes to VulcanJS itself as well as their example-forum. We try to keep the changes to each codebase distinct, and have factored our web app into two repos.

I recommend starting by creating a folder to store both of them:

mkdir lesswrongSuite
cd lesswrongSuite

Clone the two repos:

git clone https://github.com/LessWrong2/Lesswrong2

git clone https://github.com/LessWrong2/Vulcan.git

You'll mostly be working in Lesswrong2.

Install node dependencies in each of the repo folders:

cd Lesswrong2 
npm install
cd ../Vulcan
npm install

Start the development server:

cd ../Lesswrong2 
npm start

If it is NOT working, there is most likely some issues with your npm install process. If you are terminal-savvy you can attempt to resolve that yourself based on error messages. If you'd like help, you can ping the LessWrong team either by creating a github issue or pinging us on intercom on LessWrong itself. You may find some help in the README for Vulcan-Starter

[Further note: If the install process results in errors, check your version of node against the one in the .nvmrc file. If individual files failed to install, try installing them individually (i.e. try npm install underscore-1.8.3 if the underscore package failed to install). That should likely result in easier-to-parse error messages that you can then google and attempt to debug]

You’ll then have a local copy of lesswrong2 running on your computer. (accessible on http://localhost:3000/)

It will start out with an empty database. (This means that some of the hardcoded links on the frontpage, such as Eliezer’s Sequences or the Codex, will not work). You can create users via the normal sign up process (entering a fake email is fine). The first user you’ll create will be an admin, so you’ll probably want to create at least two users to check how the site looks for non-admins.

You can poke around the codebase. Eventually, to make serious contributions you’ll want to read about our site architecture.

What Contributions Are Helpful?

The most reliably helpful thing would be to tackle outstanding issues that have been tagged on github.

In particular, you can filter them by the tag “good first issue.” (Some of these might require some explanation, but I expect I can explain them fairly easily to a new contributor)

Creating Issues

You can create a new issue. If so, please leave it untagged for the time being (so that admins can quickly look for untagged issues, and sort through them)

BugsIf you run into a bug, the most helpful thing to do is to search for related keywords in the issue tracker. If you can’t find anything relevant, create a new issue. Try to provide as specific information as possible (your browser, exact links to the post or page where you had the issue, information on how to replicate the bug if you can)

Feature Requests – Feature requests will often need to undergo some discussion to get refined into something executable, and may sometimes need to be split into sub-features.

Features tend to cluster into either “things that are pretty straightforward and we almost certainly want to do” and “things that we have weird conceptual philosophical opinions about that may sometimes be hard to explain succinctly.” (An upcoming post will go into some of this).

After you’ve posted a feature, an admin will tag it (If it’s been a week and we haven’t done so, feel free to bug us about it. We’re still adapting to the role of “open source facilitators” so we may drup things a bit)

Understanding the Codebase

LessWrong 2 is built on VulcanJS, a generic framework for online forums. VulcanJS is in turn built on technologies including:

  • MeteorJS – an underlying framework that you should rarely have to think about while coding for LW.
  • MongoDB – a NoSQL database. There is no formal schema.
  • ReactJS – for the front end. I’m hoping React is enough of The Current Hotness that I don’t have to explain it.
  • GraphQL – our backend API. The main difference between this and a REST API is that you can query multiple resources at once.

Eventually, it’ll be helpful to have a good understanding of each of those technologies (both to develop new features and fix many kinds of bugs). But for now, the most useful things to know are:

  • Collections – Mongo databases are organized around collections of documents. For example, the Users collection is where the user objects live. Mongo databases do not technically have a rigid schema, but VulcanJS has a pattern for files that determine the intended schema (which is used by the API, forms and permissions systems to determine what database modifications are allowed)
  • Components – Our React components are organized in a folder structure based loosely on our collections. (i.e. components related to the User collection go in the packages/lesswrong/components/users folder).

    Some edge cases just go in a randomly picked folder (such as the RecentDiscussion components, which involve both comments and posts, but live in the comments folder)

    There are multiple ways of creating a ReactJS component. Ideally, each component does one (relatively) simple thing and does it well, with smart components and dumb components separated out. In practice, we haven’t done a great job with this. (Scope creep turns what were once simple components into increasingly complex monstrosities that we should really refactor but haven’t gotten around to it).

    We use Vulcan’s registerComponent function to add them as children to a central “Components” component. (read more here)
  • withComponents (Higher Order Components) – VulcanJS makes a lot of use of React’s higher order components. Often you need a component to have access to particular attributes (the currentUser object, or a function that can edit a given document). Vulcan has a series of components that wrap around other components. They have a names like withCurrentUser, withEditMutation, withDocument. Wrapping a component in a withX typically passes extra information in as a prop.
  • Fragments – GraphQL queries are made using fragments, which describe the fields from a given database object you want to fetch information on. There’s a common failure mode where someone forgets to update a fragment with new fields, and then the site breaks the next time a component attempts to use information from the new field.

A Vulcan project is divided into packages. It comes with several base packages that provide core functionality (users, voting, routing, etc).

The two packages you’ll most likely need to pay attention to are:

  • packges/example-forum – A sample forum that defines some key collections (posts, comments, and notifications, as well as “categories” [basically tags] which we aren’t currently using).

    We avoid editing example-forum if we can, so we can more easily upgrade it when the main Vulcan branch updates. Instead, if a function in example-forum doesn’t suit our needs, we replace in our custom lesswrong package.

    However, in some cases it’s impractical to avoid changing example-forum. When we do, we add a nearby comment:

    // LESSWRONG – [text explaining the change]

    So we can easily search for LESSWRONG and find all changes we’ve made.
  • packages/lesswrong – Our primary codebase.

    This includes new collections, components, and additional functions/attributes/extensions for objects in example-forum.

Going Forward

There's a lot more worth saying – I plan to write a post and/or github documentation that go over all the most important chunks of the codebase to help new developers get started. That's a fairly big project. But meanwhile it seemed better to get this post up and see what questions people actually had.

I also plan to write a bit more about some our underlying site philosophy, which will be important for people who want to suggest new features or make front-end changes.

Copying the tl;dr one more time:

  • Get a local copy of our git repo installed
  • Check out the Good First Issue filter of our github repo Issues page.
  • Once you're familiar, check out the Help Wanted filter.
  • You'll still want to read this post to get oriented.