Rage Against The MOOChine

[-]johnswentworth4y130

I recommend looking for "open course ware" (e.g. here or here) rather than MOOCs. These are usually course materials from the classes taught in-person at a university, which means they have actual prerequisites, and usually aren't subject to the MOOC-esque pressure to dumb everything down to the most accessible possible level. (The latter link above includes the version of Ng's ML course taught at Stanford, back in the day.)

[-]Alex Mikhalev4y70

I think the most important point of teaching maths for data science is to build mental models in the data scientists head. It takes time and part of the process of learning maths and usually takes 2 years (or two years course in university). Bypassing that process backfires - startups raising money for AI/ML normally take 2 years before shipping the product.

I think the mental model part is the most difficult to teach, but obviously, we are paid for specialised skills - like coding in python, hence everyone wants to jump into coding python without putting effort into learning maths and building proper mental models, which I think is wrong. The coding part is easy - I am comfortable with Octave/Matlab and Python or Rust. I would back Andrew Ng for choosing Octave for example - it's one of the most concise ways to get ML concepts working, although I disliked it when I was a student, then I tried to translate the following code into Python:

%% Main loop

while(gen<maxgen)
  gen
  % perform uniform selection, then intermediate crossover
  chx=zeros(lamda,nvar);   % create storage space for new chrom
  sigx=zeros(lamda,nvar);  % create storage space for new sigmas
  for n=1:lamda            % loop over all population
    if(rand<xovr)      % roll the dice and see if we need to cross
      alpha=rand(1,nvar)*1.5-0.25; % create vector of random
                                   % then crossover
      chx(n,:)=alpha.*chrom(mod(n-1,p)+1,:)+(1-alpha).*chrom(ceil(rand*p),:);
      sigx(n,:)=abs(alpha.*sig(mod(n-1,p)+1,:)+(1-alpha).*sig(ceil(rand*p),:));
    else
      chx(n,:)=chrom(mod(n-1,p)+1,:);   % just copy if not crossing
      sigx(n,:)=sig(mod(n-1,p)+1,:);
   end
  
  end

[-][anonymous]4y60

I myself did some of Andrew Ng's courses, and I understand where you're coming from. Although this was several years ago, but I do remember Octave!

I saw your guide: https://github.com/Simon-Holloway/Full_Math_CS_Guide I just want to say: Real Analysis is overkill in my opinion if your goal is to simply become an AI researcher. Also, I personally like Karpathy's advice (which seems like it should radically alter your guide):

How to become expert at thing:
1 iteratively take on concrete projects and accomplish them depth wise, learning “on demand” (ie don’t learn bottom up breadth wise)
2 teach/summarize everything you learn in your own words
3 only compare yourself to younger you, never to others

― https://twitter.com/karpathy/status/1325154823856033793?s=20

Also, just as a side note, even mathematicians do this in research if not for basic things like real analysis:

Most big number theory results are apparently 50-100 page papers where deeply understanding them is ~as hard as a semester-long course. Because of this, ~nobody has time to understand all the results they use—instead they "black-box" many of them without deeply understanding.
― https://twitter.com/benskuhn/status/1419281155074019330

[-]Alex Mikhalev4y20

I agree that nothing bets practical projects, but in modern life, you need to learn a lot of background information before jumping into the real world. There are plenty of ML projects and examples that are equivalent to the ToDo (12-factor app) in complexity - single component, boundaries clearly defined. The next steps in the real world would be - here is a payment platform with 270+ services and components, how does your AI/ML component fit into it? Who do you talk to to figure out the business value of the AI/ML component (business analysis/domain driven design)? How do you talk to your creative colleagues who are responsible for user experience in a productive manner ( i.e. jobs to be done )?

I see this gap quite consistently and I am trying to address it on the technical side by building medium size AI/ML project with 3 pipelines http://thepattern.digital/ and I think modern ML/AL professionals need to know things above before jumping into any real-world project.

[-]philip_b4y40

There are a lot of very good resources to learn ML that are accessible for free. Here I list some of them.

Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares - Boyd 2018. It's a not-very-deep (but much deeper than your average MOOC) linear algebra textbook with the focus on ML and ML-adjacent applications. For instance, it covers k-means clustering, least squares data fitting, least squares classification. Boyd is known for being the authour of the best introductory textbook on convex optimization, hence this textbook is probably good as well.
Linear algebra and learning from data - Strang 2019. This one assumes some knowledge of linear algebra and teaches how to use all that on contemporary computers efficiently and also it teaches many ML and ML-adjacent methods using that knowledge, including (stochastic) gradient descent, neural networks, backpropagation, some statistics. Strang is known for being the author of one of the best linear algebra textbooks "Introduction to linear algebra", hence this textbook is probably good as well. There is also a free MIT opencourseware course https://ocw.mit.edu/courses/mathematics/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/ that follows this textbook.
The elements of statistical learning. Data mining, inference, and prediction 2nd edition - Hastie and Tibshirani 2008. This one covers A LOT OF ML, although all of it is pre-neural-network-revolution. Most exercises are of form "prove this theorem" rather than "implement this in code".
Pattern recognition and machine learning - Bishop 2006
Probabilistic Machine Learning: An Introduction - Murphy 2021
Deep learning - Goodfellow, Bengion, and Courville 2016
AI: a modern approach 4th edition - Russell, Norvig 2020

Some of these are free but most are not. However, you can pirate them off libgen.

[-]habryka4y30

Mod note: I cleaned up the formatting a bit.

[-]CarolusRenniusVitellius4y20

Fwiw, my experience with MOOCs/OCW has been extremely positive (mainly in math and physics). Regarding the issue of insufficient depth, for a given 'popular topic' like ML, there are indeed strong incentives for lots of folks to put out courses on these, so there'll be lots to choose from, albeit with a wide variance in quality - c.f. all the calculus courses on edX.

That said, I find that as you move away from 'intro level', courses are generally of a more consistent quality, where they tend to rely a lot less on a gimmicky MOOC structure and follow much more closely the traditional style of lecture --> reading --> exercises/psets. I find this to be true of things that are about equivalent to courses you'd take in ~3rd year uni - e.g. "Introduction to Operating System Scheduling" as opposed to the "Introduction to Coding" courses or whatever the equivalent is for your area of study. If you start to look at more advanced/specific courses, you may find the coverage quality improves.

I definitely can't promise this would work for ML, but I've found it useful to think about what I want to learn from a course (concepts? technique? 'Deep understanding'?) before actually searching for one. This provides a good gauge that you can usually figure out if a course meets within a few minutes, and I think may be especially relevant to ML where courses are fairly strongly divided along a tradeoff of 'practice-heavy' vs 'theory-heavy'.

It's worth noting though, that my physics/math perspective might not be as valid in learning ML, as in the former the most effective way to learn is more-or-less by following some traditional course of studies, whereas the latter has a lot of different websites which teach both in technique and theory at a basic-intermediate level; I'd be surprised if they, combined with projects, were less effective than MOOCs at the level for which they're written. As others have noted, it may be worth looking at OpenCourseWare - MIT's being the most extensive that I'm aware of. They also offer a 'curriculum map' so you can get a feel for which courses have prerequisites and what the general skillset of someone taking a given class should be.

[-]Olomana4y20

I have taken a few MOOCs and I agree with your assessment.

MOOCs are what they are. I see them as starting points, as building blocks. In the end, I'd rather take a free, dumbed-down intro MOOC from Andrew Ng at Stanford, than pay for an in-person, dumbed-down intro class from some clown at my local community college. At least there's no sunk cost, so it's easy to walk away if I lose interest.

[-]jeremy4y20

I just finished Andrew Ng's course as well, and had a similar experience to you. I do have a math background, so in retrospect it was probably a mistake to take it, but I saw it recommended so highly by people. I think the main value I got from it was the heuristics for debugging models and such, but I'm left wondering how many of those are even still relevant.

I'm still trying to learn ML though, so I'll take a look at your CS+ML guide. I remember trying fastai a few months ago and I felt there like I wasn't learning much there either, again other than debugging heuristics. I also don't like their special library, because I can't remember which things are part of the library and which are just pytorch (they're essentially teaching you two libraries at once, plus all the ML concepts--it's kind of a lot to keep in your head). Maybe I'll take another crack at it.

If you want another guide to pull from, I was following this one a few months ago. It stood out to me from the millions of other "86 bajillion books to learn computer science NOW" lists online because they intentionally limited it to a few subjects, and give their reasoning for each choice (and the reason some other popular books may be bad choices). It's much more CS focused, rather than programming focused, which is why I'm not following it now, but I plan to return to it when I actually have a job :)

[-]jacopo4y20

For info, you can find most of the exercises in python (done by someone else than Ng) here. They are still not that useful: I watched the course videos a couple of years ago and I stopped doing the exercises very quickly.

I agree with you on both the praise and the complaints about the course. Besides it being very dated, I think that the main problem was that Ng was neither clear nor consistent about the goal. The videos are mostly an non-formal introduction to a range of machine learning techniques plus some in-depth discussion of broadly useful concepts and of common pitfalls for self-trained ML users. I found it delivered very well on that. But the exercises are mostly very simple implementations, which would maybe fit a more formal course. Using an already implemented package to understand hands-on overfitting, regularization etc. would be much more fitting to the course (no pun intended). At the same time, Ng kept repeating stuff like "at the end of the course you will know more than most ML engineers" which was a very transparent lie, but gave the impression that the course wanted to impart a working knowledge of ML, which was definitely not the case.

I don't know how much this is a common problem with MOOCs. It seems easily fixable but the incentives might be against it happening (being unclear about the course, just as aiming for students with minimal background, can be useful in attracting more people). Like johnswentworth I had more luck with open course ware, with the caveat that sometimes very good courses build on other ones with are not available or have insufficient online material.

[-]Jozdien4y20

I agree with your points on practical programming in the course, but I also think that's not even Andrew Ng's core intent with his courses. As Teja Prabhu mentioned in his comment, learning through taking on projects of your own is a method that I can't think of many good alternatives to, as far as practical usage goes. But getting there requires that you cast a wide net breadth-wise to at least know what's possible and what you can use, in machine learning. You can, and probably will, learn the math depth-wise as you try working on your own projects, but to get there? I think he throws just the right amount of technical math at you. Trying to fit all the math involved in all the different ML methods he covers, from the ground up, is probably infeasible as anything but a year-long degree, and you don't need that to start learning it yourself depth-wise.

That, and a working understanding of ML theory are what I think his primary intent is, with his courses. I did his Deep Learning specialization a couple months ago, and while the programming is slightly more hands-on there, it's still massively aided by hints and the like. But he even says in one of those videos that the point of doing the programming exercises is only to further your understanding of theory, not as practice for building your own projects - writing code from scratch for a predefined goal in a course wouldn't be a great way of motivating people to learn that stuff. Incidentally, this is why I think MOOCs for learning programming actually are pointless.

[-]Alex Mikhalev4y10

I would recommend structuring applied maths learning differently: start with Computational Beauty of Nature https://mitpress.mit.edu/books/computational-beauty-nature and then go deep in relevant areas + graph theory + graph algebra. Also a deep understanding of multi-objective optimisation techniques: NSGA-3, Pareto front/Pareto surface.

LESSWRONG
LW

LESSWRONG
LW

20

Rage Against The MOOChine

20

20