edanm

Comments

Beginning Machine Learning

Great overview!

I can give a few words of advice on where to continue from here, if you're interested. My own background is as a software dev for many years (13 years professionally plus a few years as a kid). I'd bene involved in many different fields, from embedded systems to web development, and recently ran a team of algorithms researchers in 3d printing, so was mildly exposed to computer vision and 3d concepts, but had no serious machine learning. Then a few years ago, I started to get much more seriously interested in ML/DL/Data Science, and have seen been working in the field (running a dev shop).

So, my take: first of all, I personally didn't much enjoy Andrew Ng's course, both because it was much too theoretical for my taste, and (in retrospect) because I didn't remember enough maths from my CS degree to work with the concepts as easily as I should have.

I'd recommend a few things for you as next steps:

1. Coursera teaches more "classical ML" (not deep learning), and without many applications. The absolute BEST followup in my mind is the Fast.ai course (free).

It focuses on Deep Learning, and teaches with a completely practical-minded approach, rather than theoretical. The idea is to, within one lesson, actually *write software*, like a simple program to tell whether a picture is of a cat or a dog. In this course, you will literally be coding practically world-class Deep Learning code within a few hours.

They're supposedly working on a more classical-ML course, but unfortunately it isn't out yet.

Seriously, this is my #1 recommendation for anyone trying to learn machine learning, especially with a background in software development. You won't be sorry.

2. If you're at all interested in actually implementing ML as opposed to more reviewing the concepts, then you should try a Kaggle competition or two. If you don't know it, it's basically a site that allows companies to upload data, then pay prize money to people writing an algorithm that does something specific with the data. E.g. predict how many page views a certain subset of pages on Wikipedia will receive.

Kaggle does a lot of good things for learning ML: It abstracts away all the data-gathering and a lot of the data-cleaning work, which is the heart of a lot of data science/ML jobs, but is not what you want to actually practice, especially if you are a developer and already know how to deal with this. It also gives specific questions and answers that need to be answered, and has a large collection of existing answers.

In short, Kaggle is the place to go practice your ML skills.

3. Learn more math, especially if you enjoy it! I've personally been self-learning the equivalent of a math undergrad, partially for my work in ML, partially for fun.

Specifically, as you correctly understood, ML is mostly statistics, calculus, and linear algebra. Based on my own background, I can tell you that my calculus was perfectly adequate for ML. However, statistics and linear algebra I had studied in much less depth, and they're both incredibly important, and fascinating subjects. Linear algebra, especially, is amazing, and depending on where you study it, you actually learn a good amount of practical implementations, including linear regressino and other ML algorithms. And not only do you learn these applications, you understand them from an entirely new perspective.

For studying linear algebra, I *highly* recommend Gilber Strang's video lectures on the subject. He is an excellent teacher, not only engaging, but very practical-minded. You should also follow along with his Linear Algebra textbook, and I highly recommend doing the videos and textbook at the same time - the textbook is not a great resource without the videos, IMO.

The major "problem" with Strang's videos are that he *really* focuses on the practical, engineering, matrix approach of teaching Linear Algebra, to the almost-complete neglect of the more mathematical approach. E.g. he teaches like half the course before he mentions linear transformations, which is *incredible*.

I still think these videos are the best approach for a software developer looking to *use* linear algebra, but I highly recommend following this up with a more mathematically-oriented textbook. Like many others, I like Axler's "Linear Algebra Done Right", which ironically takes the exact opposite approach - it takes him 3 chapters to get to explaining what a Matrix is :)

Other than that, I have recommendations for statistics (both Harvard and MIT have great courses on probability), and there's lots of other great math to study, though not all of it is directly relevant to ML.

OK, hope this post was worthwhile for you/someone. Feel free to ask if you have any questions :)

Is Rhetoric Worth Learning?

I'm not sure I agree re: lawyers, or about how people/society thinks of this. For one thing, I don't think most people are that OK with lawyers - they tend to get a lot of flack, and e.g. criminal defense attorneys will often get pushback from people who identify them with their clients, irrespective of the fact that they know the lawyers don't necessarily condone their clients' actions.

Another thing - most people absolutely hate hypocrisy. I think it's considered a death-blow to most people's arguments. People compliment politicians on their speaking skills, but if they discovered that the politician's are not saying things they believe in, they'd turn on them. (Well, theoretically - President Trump is a good counterexample).

Btw, an aside, but I also think you misrepresent what lawyers do in some way. They're supposed to be advocating for the rights of their clients, and supposed to persuade, but they can't for example lie. They are a check on the system that works from within the system - they need to make sure everyone is playing by the rules, but they can't just make up their own rules or anything. That said, of course rhetoric is important for trial lawyers.

Set Up for Success: Insights from 'Naïve Set Theory'

"Geometric" intuition is basically the way that the 3Blue1Brown YouTube channel would explain things. I'm not sure if you're aware of it, but their "Essence of Linear Algebra" goes through the broad high-level concepts of linear algebra and explains them, with a very visual/geometric intuition for things like basis change, inverses, determinants, etc.

Unfortunately, they never covered transpose :)

Also, I'll take a look at your blog post, thanks!

Set Up for Success: Insights from 'Naïve Set Theory'

Theoretically, I'm most interested in things related to Data Science/Machine Learning/Deep Learning, as that's my day job. However, since this is my day job, it's also the area that I know the most about. So e.g. I've studied Linear Algebra, Probability, Statistics etc quiet a bit.

I'm mostly interested in rounding out my specific knowledge with everything else I need to know to have knowledge equivalent to a well-rounded math major.

In terms of what personally interests me the most, that'd be logic/foundations-of-mathematics type stuff, e.g. Set Theory.

Set Up for Success: Insights from 'Naïve Set Theory'

Thanks for the generous offer! What kind of requests would you like to get? Specific questions? Certain subjects? etc.

In any case, I'm going to write a bunch of stuff I'd love to have explained more thoroughly, some general, some more specific, if you can explain any of them that would of course be amazing from my point of view. Most of these are just things that came up for me while (re)learning a bunch of math.

  1. Linear Algebra - I have great intuition for most matrix operations based on Strang and 3blue1brown videos. E.g. inverse, change of basis, etc. One thing missing: what is the intuitive "geometric" interpretation of matrix transpose? Considering the transpose is used all the time, I still don't have an intuitive sense of what it does.
  2. An intuitive explanation of the SVD.
  3. More general: I'm trying to recreate a math degree from scratch, more or less. There are tons of lists of best textbook for each subject, but I'd love more of an overarching "here are the subjects you need to know, here's the best order to study them" kind of list. There aren't many great lists for these.
  4. Set Theory - an explanation of the difference between transfinite induction and regular induction. I.e. IIRC, there needs to be defined a whole new kind of induction in order to do transfinite induction vs the base induction used to define things a little past N. Why is another mechanism necessary?

Thanks!

Arbital postmortem

First of all, I want to join all the others in thanking you for the honesty and for the sharing.

I'm going to give a few of my views of this, as someone who has a fair amount of experience in "startup-land". Some of this will be "criticism", but please don't take offense - it's really hard to get these things right, and we all made and continue to make mistakes. And you seem to have gotten to some of these conclusions yourself - I'm writing this for the hypothetical other people who may want to start a startup, so making it general. Btw, really long comment, so sorry! :)

First of all, I'll tell you what was by far the thing I miss most from your post - any talk about money. You're building a for-profit company, and maybe I missed it, but I have no idea how you planned to make money off of this! I have no idea of the busines plan, at all. Even if making money isn't really the goal here, unless you plan to live off donations, it should still be priority 1,2 and 3 for any company: You use money to solve most problems (pay to create content; pay to advertise; etc). You also use money as a good proxy for success. You also use money to fix problems like "burnout" by hiring people!

Secondly: I loved Inadequate Equilibria. But the whole "conversation about startups" was by far the weakest part of it, and the one part I think I actively disagreed with. (Also the part I know the most about: Gellmann amnesia anyone?) While I understand the concept of a grand vision, I think Eliezer and probably you are misunderstanding the idea of an MVP. Or at least, the way I think about it.

The idea is not, as Eliezer put it, to build a product that shows one specific workflow. For one thing, you don't need to biuild a product. But more importantly, the emphasis is not on showing a complete workflow and seeing if people like it. The emphasis is on doing fast experiments. You need to figure out what assumptions you are making about what you're trying to build, then test those assumptions. This is something you can often do with minimal work, by faking large parts of the product, for example.

One of the pushbacks to this view is that you might not know the assumptions, but that's all the more reason to have fast experiments - you want to uncover which assumptions you're making and don't realize it, as soon as you can. If you think the only way to do this is to build a product over more than a year - you're almost certainly wrong, except in very tech-heavy cases, which is not your situation.

In Arbital's, some assumptions you had and could've tested:

  1. You believed people will write content, for free. Easy to test - ask people to write it before the product.
  2. You believed you had a superior flow for reading content. I'm still not sure I understand what that full flow was, but again, easy to test - ask Eliezer/someone to write content, and make a static html site with all the "reading" functionalty, but none of the editing capabilties. You could reasonably make whatever workflow you imagine the reading experience to be with a week's worth of hand-coding html/css, or even a Wordpress site.
  3. You believed you could get people interested in reading this material, and then doing... something? I'm not sure, since I didn't understand the business plan. But let's assume it's "decide to read more material". OK, easy to test - put one guide up, and ask people to sign up to a newsletter. Or donate money. Or something.

(I want to emphasize that, although I think I'm right, just the fact that Arbital failed doesn't prove it. Building startups is hard and usually fails.)

Another issue that stems from lack of a business model - what kind of company were you trying to build? It seems pretty obvious, at least in retrospect, that this kind of company is a bad fit for a VC-funded startup. You are not trying to build somethign with minimal chance of success, but with the ability to become a billion-dollar company. I mean, I don't think you were aiming for a billion dollar company.

But in that case, you should've never expected to raise money (and probably shouldn't have raised money). And you should've made sure this is something that could be profitable relatively quickly, to continue supporting the development.

Lastly, I really got the sense from your post that you are all with a very engineering mindset, and very enamored by the beauty of a complex system, and by wanting to build something. Hell, you worked on this for a few years, wrote an entire post-mortem about it, and still I and others in this thread don't even understand what you're building! This is not a good sign - systems usually aren't this complex, certainly not ones that are made to be used by actual people other than Eliezer :)

One more thing about community projects - we're a community with a lot of developers. We see development projects everywhere. But the real strength of the community is not necessarily in that - if the bottom line thing we want, as a community, is more things like Eliezer-style explanation of hard concepts, that's hard enough - we should make the journey to creating that content incredibly simple, and while that is arguably what Arbital was, I'd say that "let's spend a few years to develop a new software platform" is a huge burden. Much better to use pre-existing stuff, IMO. Let's make a rationality-Stack Exchange. Or a rational wiki (or not :) ). (Not to crticize too hard because I'm not that much in the community and don't know the details, but I kind of wonder the same thing about LessWrong V2.0 - do we really need to rebuild forum software from scratch just for us? Is that really where we should be spending our community's talent and efforts?).

To conclude: building starhtups is hard. Building consumer startups is much harder. Building consumer startups that are marketplaces is really really freakin hard. You tried and failed, which is a shame, but you seemed to have learned a lot from this, both about startups and other things (based on Inadequate Equilibria, I think Eliezer hasn't learned the lessons I would've learned). So kudos for trying, kudos for putting yourself out there with this post, and in general, good job!

A LessWrong Crypto Autopsy

Very interesting, and I kind-of agree with the conclusion. However, as a few people pointed out, it wasn't as simple as just buying bitcoin, you had to sell at the right time, etc.. And buying bitcoin was complicated.

But the other problem is that there are thousands of opportunities, things you should do, etc, lying around, with a possibly good payoff in expected value terms. And how many of them do we do? How many of them do we even think about seriously?

Just a few off the top of my head (first two are obvious, then some others):

  1. Cryo, obviously.
  2. AI safety - tons of people in the community agree that's it's a serious issue, but how many actively work towards fixing it? (A decent amount, but I'd guess not as much as the "could" do).
  3. CPR training - how many people do it? What is the chance that you'll one day be in a situation where you need it? I have no idea of the numbers, but considering this is a life-and-death situation, have you looked up the numbers and decided it's not worth doing?
  4. Similarly, buying an emergency survival pack in case of a natural disaster. Again, not idea of the numbers here, but people do find themselves in situations where they're without food/water for some period. Do you know the numbers here to decide it's not worth buying canned goods?
  5. How many people have even bothered to sit down and make sure their health insurance/life insurance/unemployemnt insurance/etc is handled properly?
  6. Even more obvious - sending out resumes every year to find better job opportunities? Or taking a course/reading about finding a better job? This is likely have a massively higher impact than buying bitcoin, not only in monetary terms, but also in life satisfaction impact.

Etc, etc.

Most of the above are pretty obvious and trivial things almost anyone can do. I can probably list a dozen more, some more "standard advice", some more out there but still probably high in expected value. If I or anyone were to actually sit down and work through these one by one, we'd probably do little else for the next year.

So while bitcoin, in retrospect, may have been the most obvious and immediately high-value payoff, I'm not sure it's easy to seperate it from all the other things above. Then again, I'm not sure you should - maybe we should have a list of "these are things you need to take care of ASAP" somewhere.

Tell me what hat to buy

Why do you want to buy a hat?

Almost all "non-geeky" / "normal" people don't regularly wear hats. If you're trying to look "better" and "more fashionable", the best solution is to skip the hat, and get sunglasses to protect your eyes

Bragging Thread May 2015

Over the last 2.5 years, my co-founder and I grew the Dev Shop we founded (Purple Bit) into a very profitable small company, employing 7 people.

Purple Bit has just been acquired by a former client of ours, Autodesk Inc. Autodesk are the makers of AutoCAD, 3D Studio Max, Maya and many other professional 3d software products.

(Note: this didn't happen this month, but it wasn't public until now).

Announcement: The Sequences eBook will be released in mid-March

So, do you have a specific marketing plan for getting the word out there about these books?

More specifically, you have access to an entire community rooting for you... how can we help?

(Also, I've been convinced by other commenters here - I normally don't buy physical books anymore, but here I'll make an exception for '"fanboyism" and "gifting" purposes).

Load More