In 2018, I was a bright-eyed grad student who was freaking out about AI alignment. I guess I'm still a bright-eyed grad student freaking out about AI alignment, but that's beside the point. 

I wanted to help, and so I started levelling up. While I'd read Nate Soares's self-teaching posts, there were a few key lessons I'd either failed to internalize or failed to consider at all. I think that implementing these might have doubled the benefit I drew from my studies. 

I can't usefully write a letter to my past self, so let me write a letter to you instead, keeping in mind that good advice for past-me may not be good advice for you.

Make Sure You Remember The Content

TL;DR: use a spaced repetition system like Anki. Put in cards for key concepts and practice using the concepts. Review the cards every day without fail. This is the most important piece of advice.

The first few months of 2018 were a dream: I was learning math, having fun, and remaking myself. I read and reviewed about one textbook a month. I was learning how to math, how to write proofs and read equations fluently and think rigorously. 

I had so much fun that I hurt my wrists typing up my thoughts on impact measures. This turned a lot of my life upside-down. My wrists wouldn't fully heal for two years, and a lot happened during that time. After I hurt my wrists, I became somewhat depressed, posted less frequently, and read fewer books. 

When I looked back in 2019/2020 and asked "when and why did my love for textbooks sputter out?", the obvious answer was "when I hurt my hands and lost my sense of autonomy and became depressed, perchance? And maybe I just became averse to reading that way?"

The obvious answer was wrong, but its obvious-ness stopped me from finding the truth until late last year. It felt right, but my introspection had failed me.

The real answer is: when I started learning math, I gained a lot of implicit knowledge, like how to write proofs and read math (relatively) quickly. However, I'm no Hermione Granger: left unaided, I'm bad at remembering explicit facts / theorem statements / etc. 

I gained implicit knowledge but I didn't remember the actual definitions, unless I actually used them regularly (e.g. as I did for real analysis, which I remained quite fluent in and which I regularly use in my research). Furthermore, I think I coincidentally hit steeply diminishing returns on the implicit knowledge around when I injured myself

So basically I'm reading these math textbooks, doing the problems, getting a bit better at writing proofs but not really durably remembering 95% of the content. Maybe part of my subconscious noticed that I seem to be wasting time, that when I come back four months after reading a third of a graph theory textbook, I barely remember the new content I had "learned." I thought I was doing things right. I was doing dozens of exercises and thinking deeply about why each definition was the way it was, thinking about how I could apply these theorems to better reason about my own life and my own research, etc.

I explicitly noticed this problem in late 2020 and thought, 

is there any way I know of to better retain content? 

... gee, what about that thing I did in college that let me learn how to read 2,136 standard-use Japanese characters in 90 days? you know, Anki spaced repetition, that thing I never tried for math because once I tried and failed to memorize dozens of lines of MergeSort pseudocode with it? 

hm... 

This was the moment I started feeling extremely silly (the exact thought was "there's no possible way that my hand is big enough for how facepalm this moment is", IIRC), but also extremely excited. I could fix my problem! 

And a problem this was. In early 2020, I had an interview where I was asked to compute . I was stumped, even though this was simple high school calculus (just integrate by parts!). I failed the interview and then went back to learning algebraic topology and functional analysis and representation theory. You know, nothing difficult like high school calculus. 

I was pretty frustrated with myself.

It's not that I didn't understand it. I just didn't remember it, especially on the spot. The worst part was that I had brushed up on calculus the previous spring, and I still didn't remember it. Turns out that my brain won't remember material it doesn't use for months on end, even if forgetting that material would be embarrassing.

Enter Anki, an amazing spaced repetition system ($20 for iOS, free for computer). The way I like to explain Anki is: 

Anki is a flashcard application into which you can enter a constant number of cards each day while retaining a constant average daily workload. You can add cards each day, without having to study longer and longer to get through all of the cards.

I currently think that unless you have really good memory or you're not learning content you want to remember months from now, you're making a mistake by not using a spaced repetition system. Read Gwern for more on spaced repetition.

Spaced repetition seems especially useful for students. In college, I ran an experiment: for an upper-level French class, I put things I didn't know how to say into Anki, reviewed daily, and otherwise didn't study at all. I got an A. 

How powerful might a bright 6th grader grow, were they to use Anki every day for their whole life? The best time to plant a tree may have been in sixth grade, and the second-best time may have been in seventh grade, but you should still plant the tree now rather than never. 

I've been using Anki for math for the last 71 days, and I currently have a deck of about 900 cards which I study for ~30 minutes daily. In 2018, I spent about 10 minutes daily reviewing a deck of nearly ten thousand French cards.
If I were to add no more cards, daily reviews drop off quickly. 

(Once completed, the reviews in the next few days will be pushed into some of the future days, so this projection is slightly optimistic, but you get the point.)

I love Anki, and I was foolish to circumscribe it to language-learning. I now use Anki to remember key concepts from academic talks, LessWrong blogposts, and yes - textbooks. Which I now love again, and which I read for ~an hour daily again, because I'm actually retaining the content

Measure theory, ring theory, random stuff about taxonomy and epidemiology, quantum mechanics, basic physics, deep RL papers, they all go into Anki, and Anki cards go into my brain - and stay there! 

What a wonderful time to be alive. 

Random Anki tips

I know quite a bit about how to best use Anki, so if you try this and it doesn't seem to work, please message me instead of banging your head against the wall or giving up!

  • Study every day, preferably at the same time so you aren't always scrambling to get it done before bedtime.
  • Sync with AnkiWeb so you don't lose all your cards if your device dies.
  • Save time by just screenshotting theorem statements and/or proofs. EDIT: micpie recommends the Mathpix OCR software, which clips text into MathJax code.
Memorizing definitions can be useful: when reading a text, it saves you from having to constantly check what the concept is. Make sure to include examples to work through - don't just toss in random definitions you're barely interested in and will never think about again. 
  • Don't just memorize proofs, focus on the key ideas. Don't just memorize definitions, throw in several example problems which are small enough to actually do in your head (or with a scrap of paper).
    • For example, if I'm trying to really remember the concept of a "representation" (above), I have cards where I reason about them by completing short proofs:
The math here isn't important. The key thing is, I want to remember the "representation" concept, and so I find a result about representations and make myself prove it. The proof isn't too long, which is key. Don't try to memorize essays with Anki. 

Am I ever going to actually use this math for my research? Probably not. Doesn't matter. Anki makes it cheap to learn and retain things.
  • If you get a card wrong more than 4 times in the first week, it's a bad card. Remake it.
  • Use MathJax instead of Latex, because MathJax renders instantly.
If you want custom commands, use the card type editor.
  • My trigger for "I should add a new card" is reading something and thinking, "this is a cool concept!"
    • I recommend adding cards liberally. Don't worry about getting the formatting or phrasing perfect at first. Just add cards and you'll develop a taste for what should be added, and how.

Read Several Textbooks Concurrently

TL;DR study several topics at once so that your brain has time to cement the concepts you're learning, before the text builds on those concepts further.

AllAmericanBreakfast's recent post is great, so I'll refer you to that to make this point:

Wait, what? You want me to make life easier on myself by, instead of studying calculus...studying calculus, linear algebra, and statistics all at once?

~ AllAmericanBreakfast, The Multi-Tower Study Strategy

The basic idea is that your brain needs time to really cement a new idea in, and so you should study several topics at once in dependency-heavy areas like mathematics. This advice matches up with both my recent personal experience and with advice I received early on from Qiaochu Yuan, but which I had unfortunately ignored. 

For example, right now I'm reading Nielsen and Chuang's Quantum Computation and Quantum Information, Evan Chen's The Infinite Napkin, and a ridiculously easy physics book, Kuhn and Noschese's A Self-Teaching Guide: Basic Physics (more on this later). Previously, I'd been going back through Wasserman's All of Statistics and Pearl's Causality after reading Pearl's Book of Why

I always feel like I'm learning something new instead of banging my head against the wall. Sometimes you should just read one book, but if you don't need to cram, I recommend diversifying. 

Completing The Whole Textbook Is Usually a Big Waste of Time, Please Don't Do It

TL;DR extract the most useful / central concepts and remember them forever via Anki. This doesn't require grasping every arcanum, every detail of a textbook. 

When I started reading textbooks, I completed the whole book. I didn't want to miss a crucial concept. But the thing about crucial concepts is that they pop up everywhere. If you missed a crucial concept, you'll know. You're not going to wake up 20 years from now and be like, "OH NO! I forgot to learn about 'force' when I self-studied physics! And I forgot to learn about injectivity in linear algebra!"

As you read more, you'll get a taste for what's probably important, and what's details you can reference later if need be. You can also ask experts if you should study part of a book - this is one key benefit of having an actual teacher. 

But don't complete the whole book and all of its exercises, if you just want to become a polymath. Leverage the Pareto principle, get 80% of the benefit out of the key 20/30/40% of the concepts and exercises, and then move on. 

Another reason I used to do this a lot was that I wanted to look good by being able to mention on e.g. my resume that I had read the whole book. It's a lot more impressive to say "I read these 10 textbooks" than "I read large parts of this book, and some of this book, and a little of this one, and a lot of this other book." I've found that learning well requires keeping an eye out for these instincts. My brain is not my friend in this battle.

I now often ask myself "am I doing this, at least in part, in order to look good?". Sometimes I answer 'yes', and sometimes I do it anyways - wanting to look good isn't always bad. But there are sometimes things more important than looking good

Read Easier Textbooks Instead of Struggling Valiantly

TL;DR even though slogging through tough textbooks makes you feel sophisticated and smart, don't. 

Imagine I'm learning how to program, but I've never used a computer before. Learning to program while learning to operate a computer, will probably take longer than learning to operate a computer and then learning to program. Learning time is superadditive in terms of your ignorance / the dependencies you're missing. 

But it's worse than that. Imagine I'm learning quantum mechanics, but I don't know any linear algebra either. I'm now trying to do three things:

  1. Learn linear algebra,
  2. Learn the formal postulates of quantum mechanics, and
  3. Tie all of this into the real world.

Similarly, if I'm trying to learn fluid mechanics without knowing how to manipulate partial differential equations (PDEs), it might look trying to simultaneously

  1. Learn PDEs,
  2. Learn the physical equations, such as Navier-Stokes, and
  3. Tie all of this into the real world to explain what I already know about e.g. water and rivers and blood pressure.

But what if instead I picked up some dumb book that doesn't even have any calculus, and let it give me approximate explanations via e.g. Archimedes' principle:

Archimedes' principle states that the upward buoyant force that is exerted on a body immersed in a fluid, whether fully or partially, is equal to the weight of the fluid that the body displaces. (Wikipedia)

I breeze through this book no problem, and I can see how to tie in these laws to explain my intuitive models: "logs float more easily than rocks because rock is denser than wood, and so the buoyant force from Archimedes' principle is enough to support the weight of a log." So I'm taking care of point #3, "tie this content into the real world." 

Then, suppose I learn about PDEs and become comfortable with them. Now all I need to do to learn a piece of fluid mechanics is to learn the relevant physical equation, and then think about how it implies things like Archimedes' principle. Crucially, via this method, I'm only confused about one thing at a time. Build models in the right order!

Be Comfortable with Approximate Models

TL;DR allow yourself to learn things in order of comprehensibility: don't try to learn general relativity before Newton's law of gravitation.

In 2019, I wanted to learn a bit of chemistry. I got my hands on a high school chemistry textbook. I got stuck on chapter two because I was being too strict about forming gears-level models.

I was stuck because I was thinking about electron shells. The book acknowledged the Bohr model of the atom was wrong, electrons aren't really discrete particles orbiting the nucleus:

A wrong model.

But I got nerd-sniped into trying to understand how the electron standing wave only has solutions for certain energy levels, which is a result of quantum mechanics (or so I remember reading). I couldn't understand why, and so the knowledge felt "fake." 

It wasn't like I explicitly reasoned "this model is wrong and so I'm not going to keep reading this book", it felt more like... I just wasn't that hungry to learn this, because it wasn't "real." And I gradually stopped returning to that book.

Learn things quickly, note your confusions, and correct them later when the Anki cards show up again. Let yourself learn approximate models of reality.

Conclusion

I knew about the "read easier textbooks" advice already, but I didn't apply it. Perhaps I just didn't recognize a chance to apply it. The same forces of chaos and entropy and madness which prevented my applying e.g. Luke Muelhauser's advice, may prevent you from applying this post's advice. If you think any of this advice might help you, I recommend setting up a plan now for how and when you'll implement it.

212

58 comments, sorted by Highlighting new comments since Today at 1:56 PM
New Comment

use a spaced repetition system like Anki.

Spaced repetition/Anki is one of those things that seems obvious and like it makes perfect sense. However, my experiences have been pushing me further and further away from thinking that it is usually something that is useful.

I spent a year or so self-studying full time. I also spent a few months studying Leetcode stuff for FAANG interviews half-time. In both experiences I spent months using Anki, perhaps 100-200 cards where I would study the cards pretty consistently every day.

I know this might go against the research on spaced repetition, but I've found that the stuff I used to study often just doesn't stick. On the other hand, there are things that I never really did spaced repetition on that have stuck remarkably well. My impression and experiences is that obtaining a deep understanding is a lot more fruitful than doing spaced repetition.

Also, thinking along the lines of a sanity check and the outside view, are there examples of people becoming very successful due to Anki? My impression is that it doesn't happen all too often, and when it does it is because Anki happens to be a great fit for the situation, eg. passing med school exams. 

Completing The Whole Textbook Is Usually a Big Waste of Time, Please Don't Do It

Intuitively, this makes a lot of sense, and is the mindset I've had for a while. I agree that the end worth seeking are those central concepts rather than memorizing certain details. However, I think that playing around with those annoying details might be a good means to the end of grasping central concepts. I spent a lot of time telling myself, "It's only the big concepts that matter, so you can skip this section, you don't have to answer these questions, you don't have to do these practice problems."

Well, it's also true that you could take memorizing details too far (I'm looking at you organic chemistry). I guess it's just a matter of judgement, and that the insight of "wait, it's really only the big concepts that I need to care about" is likely to lead to failure modes where you undervalue the details.

Read Easier Textbooks Instead of Struggling Valiantly

Strongly agree. Seems like a failure mode that is common. I love "X for Dummies" types of books/posts. Reminds me of Explainers Shoot High, Aim Low:

A few years ago, an eminent scientist once told me how he'd written an explanation of his field aimed at a much lower technical level than usual.  He had thought it would be useful to academics outside the field, or even reporters.  This ended up being one of his most popular papers within his field, cited more often than anything else he'd written.

Be Comfortable with Approximate Models

Reminds me of the blurry-to-sharp approach:

"are there examples of people becoming very successful due to Anki?"

It's hard to answer that kind of question, because there are plenty of examples of people becoming successful without SRS, and people who use SRS don't only use SRS.

Personally, I use Anki for professional development (so not to pass exams, but for long-term mastery).  My biggest topics are math, algorithms (ex. machine learning research topics), and programming.

It's got a number of advantages.  Sometimes it helps by keeping technical details (like trig identities or programming syntax) close to the surface, so I can finish a project faster or plow through whiteboarding a problem with a colleague.  Other times it helps by cementing a deep conceptual intuition strongly, so that I can focus on doing more advanced things (instead of rehashing the same slippery basics).  Or by just increasing the breadth of what I know is available, so I don't have to agonize over the right Google terms. Sometimes it helps by making it easier to remember people's names, accelerating my ability to network (I recommend mnemonics here, since names are rote memorization, and rote memorization is difficult with Anki).

But above all, it ensures that I can keep learning, even if I only have 20 minutes a day to spare for deep/slippery topics like quantum computing; or 10 minutes a day to capture a fleeing professional experience that would otherwise be in one ear/out the other.  I'm just 32 years old, but that's old enough that I've forgotten much more than I've learned.  Anki changes that: it acts like compound interest for knowledge.  This spring (after a few years of regular SRS use) I'll reach 40,000 cards, with no sign of slowing down.

It's hard to reproduce that kind of growth with traditional learning methods.

Hm, I seem to have different experiences than you have had.

Sometimes it helps by keeping technical details (like trig identities or programming syntax) close to the surface, so I can finish a project faster or plow through whiteboarding a problem with a colleague.

My experience as a programmer (from myself and from talking to others) is that you pick up syntax pretty quickly. For a new language there's a learning curve. Perhaps SRS would help with that, although that doesn't seem too useful. Eg. I use JavaScript every day and rarely use Python. I could use SRS for Python. Alternatively, I could wait until I need to use Python, take a few days getting familiar with the syntax, and then be fine, which seems better.

Other times it helps by cementing a deep conceptual intuition strongly

Perhaps my memory is failing me, but I can't recall doing SRS over and over again ever cementing a deep conceptual intuition for me. I think looking at the same thing from different angles is often helpful, but with SRS, a given card is largely looking at the concept from one angle.

Here's an example. I'm learning functional programming right now. In functional programming, operators like + and * are just functions. Instead of 2 + 3 you can do (+) 2 3 just like you'd do add 2 3. Similarly, you could call functions using infix notations by using backticks like this 2 `add` 3 and it behaves like an operator. This really cemented for me what an operator is in such a way that I don't think SRS ever could, and it did so much faster.

But above all, it ensures that I can keep learning, even if I only have 20 minutes a day to spare for deep/slippery topics like quantum computing; or 10 minutes a day to capture a fleeing professional experience that would otherwise be in one ear/out the other... It's hard to reproduce that kind of growth with traditional learning methods.

It sounds like your point is moreso that it's hard to be as productive without SRS, but I'd like to note that there are other ways you can use a spare 15 minutes to learn rather than SRS.

There's definitely a tradeoff between breadth and depth/speed. On a real-world project, you can attain great speed and/or great depth on the narrow set of techniques/concepts that are directly relevant to the work you do every day.

It's very expensive to maintain that kind of fluency. I've never done any programming in a language that uses infix vs. prefix operators, yet I know the difference just from a couple minutes total studying the concept at a few different widely-spaced points in time. This concept would not pose a challenge to me were I to pick up a functional programming language, although it would take time to build a habit of using them.

When you overinvest in study-through-practice in a concept you've already mastered, you will forget a lot of the more broad knowledge. Anki gives you a way to retain it, or to get it back when you're ready for it. If you see yourself as having no need for a broad knowledge-base -- if you see your career as being a happy code monkey banging out programs that are within the wheelhouse you've already established, then you're probably good.

But if you have a different vision for your career, it's possible that broad knowledge will be really helpful. And spaced repetition/Anki gives you a set of tools to build and maintain that broad knowledge-base.

I'm really glad you posted this objection to Anki, because I think it's probably common. It's also a fair point: sometimes, we don't care about building a broad knowledge base. We're just trying to become fluent in the narrow set of skills that let us execute a technical project. Clarifying that distinction is very valuable from the perspective of budgeting your study time wisely.

Interestingly, this comment made me more excited about using Anki again (my one great success with it was memorizing student names, which it's well-suited for, and I found it pretty useless for other things), because this comment has a great idea with a citation that I probably won't be able to find again unless I remember some ancillary keywords (searching "blurry to sharp" on Google won't help very much). But if I have it in an Anki deck, not only will be it more likely to be remembered, but also I'll have the citation recorded somewhere easy to search.

my one great success with it was memorizing student names

+1, I used Anki before my CFAR workshop. had to remind myself not to call people by their names before they even had their nametags on. This was great, because remembering names is generally a source of very mild social anxiety for me. 

Thanks for all of these thoughts!

My impression and experiences is that obtaining a deep understanding is a lot more fruitful than doing spaced repetition.

I don't see why these should be mutually exclusive, or even significantly trade off against each other. This sounds like saying "drinking water is more important than getting good sleep", which is also true in a literal sense, but which implies some kind of strange tradeoff that I haven't encountered. I aim to get a deep / intuitive understanding, and then use Anki to make myself use that understanding on a regular basis so that I retain it.

However, I think that playing around with those annoying details might be a good means to the end of grasping central concepts. I spent a lot of time telling myself, "It's only the big concepts that matter, so you can skip this section, you don't have to answer these questions, you don't have to do these practice problems."

I agree, and I'm not trying to tell people to not sweat the details. It's more like, you don't have to do every single practice problem, or complete all the bonus sections / applications which don't interest you. Which is a thing I did - e.g. for Linear Algebra Done Right, and then I forgot a lot of linear algebra anyways.

Another approach is, you start with the papers you want to understand, and then backchain into the concepts you have to learn in a big tree. Then you're guaranteed not to waste time. I haven't tried this yet, but it sounds very sensible and I mean to try it soon.

I don't see why these should be mutually exclusive, or even significantly trade off against each other... I aim to get a deep / intuitive understanding, and then use Anki to make myself use that understanding on a regular basis so that I retain it.

Because that time you spend using Anki to retain it is time that you could spend seeking deep understandings elsewhere.

It's more like, you don't have to do every single practice problem, or complete all the bonus sections / applications which don't interest you.

Yeah, that makes sense. Sounds like we made opposite mistakes :)

Another approach is, you start with the papers you want to understand, and then backchain into the concepts you have to learn in a big tree. Then you're guaranteed not to waste time. I haven't tried this yet, but it sounds very sensible and I mean to try it soon.

I've tried that. I think it's an approach to have in your toolkit, but for me it's not really my style.

Eg. in learning Haskell I've been trying to skim a book, then work on practice problems, then backtrack when I notice a concept I haven't learned yet that I need to solve the problem. I always seem to get lost/overwhelmed in doing the backtracking. So I pivoted to reading the 1000+ page Haskell Programming from First Principles book, which is exactly what the title implies, and it's going well for me.

Also from a more motivational perspective rather than a pedagogical one, I find that I can get into a flow state more easily with the first principles approach. With the backtracking approach, I often get stuck. With the first principles approach, it may be slow at times, but at least I'm continuing to take small steps forward and not getting stuck.

Because that time you spend using Anki to retain it is time that you could spend seeking deep understandings elsewhere.

Right, but part of having deep understandings of things is to have the deep understanding, and if I just keep acquiring deep understanding without working on retention, I'll lose all but an echo. If I don't use Anki, I won't have the deep understanding later. I therefore consider Anki to help with the long-term goal of having deep understanding, even if moments spent making Anki cards could be used on object-level study.

I guess the crux is about Anki helping with retention. If you'd lose a lot of your understanding without Anki, I agree it's worthwhile. If you'd only lose a little, it seems like a better use of time to seek deep understandings elsewhere.

There is a lot of material I like in this post.

  • The idea to use Anki for math is something I've considered but never figured out a good way to do because the natural fact unit of advanced math is more complicated than a the definition of a natural language word. I would be happy to hear details more about how you did this.
  • The tips on textbooks are useful. I have never read a textbook cover-to-cover (except athletic training textbooks) even though I have tried many times. It is reassuring to hear this is not important. It is also useful to hear the advice about reading easy textbooks. The value of reading easy textbooks is obvious in retrospect but something I never figured out on my own before reading this post.

The idea to use Anki for math is something I've considered but never figured out a good way to do because the natural fact unit of advanced math is more complicated than a the definition of a natural language word. I would be happy to hear details more about how you did this.

I didn't really have trial and error when I started doing this in November, I just dove in and it worked. I had, after all, experience making well over ten thousand Anki cards. The key is to only make new cards composed of concepts which you already understand. I'll try to unpack how I make math cards, though, by making a new card and introspecting on what my instincts are.

Let's say I'm trying to make a card for "what is a mathematical function?" (this isn't "advanced math", but I find that my card-entry approach remains effective as the material gets harder). I'm first going to show the card I made for this just now.

In writing the flip-side, I notice the following mental events, in order:

  1. I want to motivate why I cared about this concept in the first place. I want to think of an intuitive situation representing a mathematical function.
    1. Different types of cards may not need motivations, but I generally like to motivate definitions. Why on earth did I even put in the effort to click 'new card' to enshrine this concept? What new things does this definition let me compactly discuss?
  2. I want to find a picture illustrating a function - some quantity being "transformed" into some other quantity.
    1. If I'm a computer scientist, I notice that I want to clarify that mathematical functions don't have dynamics - they're just input-output pairs. See my first textbook review ever:

      "Functions  are just static sets of ordered pairs  . They are not dynamic indexing functions, they do not perform efficient lookup, please do not waste an hour of your life trying to figure out how you could do such a thing within the simple machinery afforded by set theory up to that point."
  3. I want to write the formal definition.
  4. I want to check that I've done a good job with the card - the card contains a new insight but explains it using concepts, theorems, and formalisms which I already understand.
    1. In this case, if I didn't understand sets, I wouldn't have used sets. Instead, I'd just say "to each value  in its domain , a  associates to  a single value in its range . For example,  associates the number  to each real number  in its domain.

Once I'm done with the definition card, I now want to add in a few more example problems. I would look up "exercises to understand mathematical functions" and put the best problem / answer pairs in. 

After that, I want to make sure I didn't make any obvious conceptual mistakes. I'm comfortable that I didn't in this situation, but that's because the real me already understands functions; I generally don't already deeply understand concepts for which I make cards. I might do something like search "most common misconceptions which students have about mathematical functions." (I often don't do this step, but it can be a good idea) 

Also, I have a "conflicting cards" deck for cards which are either too complicated for now (I need to learn what a ring is before learning what a ring ideal is) or too similar to another concept I'm presently learning (don't try to learn accuracy vs recall vs specificity vs  score all at once). I'll add those cards to the main deck later, after I've fully learned the current set of concepts.

More advanced math

Here's a card I have that's a little longer and less atomic:

This is only two-thirds of the card! I'm not memorizing this entire thing. Instead, there's a felt sense that corresponds to walking through the definition. I remember that I want to define Lebesgue integration for indicators, and then for simple functions, and then take a supremum in a certain way. Then I think about special cases, and hopefully remember to specify what happens if 

The point to this card is that if I want to be able to walk through the concepts in the right order, with the understanding that I could expand the concepts into full formal detail if needed (since I understand the subconcepts of e.g. the Lebesgue measure, simple functions, measurable sets, etc). I have the prereqs, and I just need to remember how to put them together. 

Then I just add philosophical concepts and clarifications at the end, in case I get confused later / want to just add notes.

(Here's a recent version of my Anki/math/physics deck, so feel free to look through it)

I'm really surprised at how big your cards are! When I did anki regularly, I remember getting a big ugh-feeling from cards much smaller than yours, just because there were so many things that I had to consciously recapitulate. It was also fairly common that I missed some little detail and had to choose between starting the whole card over from scratch (which is a big time sink since the card takes so much time at every repeat) or accept that I might never remember that detail.

I'm super curious about your experience of e.g. encountering the function question. Do you try to generate both an example and a formalism, or just the formalism? Do you consciously recite a definition in words, or check some feeling of remembering what the definition is, or mumble something in your mind about how a function is a set of ordered pairs? Is the domain/range-definitions just there as a reminder when you read it, or do you aim to remember them every time? Do you reset or accept if you forget to mention a detail?

I'm really surprised at how big your cards are! When I did anki regularly, I remember getting a big ugh-feeling from cards much smaller than yours, just because there were so many things that I had to consciously recapitulate. It was also fairly common that I missed some little detail and had to choose between starting the whole card over from scratch (which is a big time sink since the card takes so much time at every repeat) or accept that I might never remember that detail.

First, the Lebesgue integration card is one of my is one of my largest; I selected it to show Isusr how I might handle a non-decomposable question. I recommend that new Anki users be careful not to do this kind of thing: find a way to decompose, whenever possible! 

I generally tolerate missing details that I would have worked out were I actually expanding the mental sketch into formal writing; for example, if I make an arithmetical error but perform the correct kinds of calculations, I count that. If I make a sign error that I wouldn't make if I had been writing it out pencil and paper, I'll usually count it. If I think momentum is a scalar quantity or something crazy, though, I fail that card. 

I want to penalize the kind of error that I'm unlikely to fix with unaided reflection, or instinctive errors (like my gut sense for a concept just being wrong). 

I guess I use Anki to recall the "felt sense" of a proof sketch, and I don't sweat the details. Here's another long card I have:

If you're new to Anki: don't try this at home!

Believe it or not, I can decompose this proof into a sequence of ~five "urges", each of which I can (but generally don't) unpack into actual math if pressed. 

That said, this card was a PITA to learn, even with my level of experience. It took a lot of 'fail's to get right consistently. Again: if one is thinking about using Anki for the first time, do not do this kind of thing! It's hard to pull off at first, knowing if your mistake was worth redoing; if  you make yourself recite the whole card above, you'll be reviewing it forever.

This is more representative of my Anki deck. Note the lack of a million-page backside.

I'm super curious about your experience of e.g. encountering the function question. Do you try to generate both an example and a formalism, or just the formalism? Do you consciously recite a definition in words, or check some feeling of remembering what the definition is, or mumble something in your mind about how a function is a set of ordered pairs? Is the domain/range-definitions just there as a reminder when you read it, or do you aim to remember them every time? Do you reset or accept if you forget to mention a detail?

Imagining that card, I first imagine the motivation before superimposing the formal definition "on top" of it to capture the relevant part of the situation. I remember the phenomenon of apple exchange, and then overlay the function to describe what happens. Then I recite the formal definition. 

I don't make myself remember to label the "domain" or "range", because that's not fair game - it wasn't on the prompt. Those should go on separate cards. (I'd only learn one of those at a time, for similarity reasons)

Interesting. I have used multiple different approaches with Anki. Last year I had to learn all pre-university math to enter uni, and I decided to use Anki which worked out pretty well. For a lot of the problems it's about remembering what steps to take in general. So I put in practice problems so I could remember the steps. The numbers don't matter, they could've been anything. 

I also tend to put in definitions, rules, theorems, standard and harder derivatives and primitives.

 

Now I'm in uni and I used Anki for learning linear algebra, I mostly tended towards clozes.

For linear algebra I tended to put in all theorems and facts, tried to keep them short so they're easier to remember and then do complementary exercises handed out by the course. Being an autodidact myself, I must say that if the course is done right it can make learning a lot easier. Sadly many of the courses weren't as good as the linear algebra course.

I don't have much to contribute to your comment except to acknowledge that it's useful, quality information. An upvote didn't feel sufficient.

(Why not simply define the integral of f as the LM of {(x,r)|x in Omega, r <= f(x)}?)

The definition I gave is the standard one, but that's an interesting question. I'm not sure I know enough measure theory to find a hole in this right away; presumably there is one, because this would be a much more succinct definition. 

EDIT: TheMajor pointed me to this SE question; looks like this can work if you can prove that the aforementioned set is measurable in the product sigma-algebra. I like this definition a lot more.

Mathoverflow has discussion on it. In short:

  • This area definition is equivalent to the standard definition, although this was (to me) not immediately obvious.
  • Some statements (linearity of integrals, for example) are obvious from the one definition, while others (the Monotone Convergence Theorem) are obvious from the other definition. Unfortunately, proving that the two definitions are equivalent is pretty much the proof for these statements (assuming the other definition).
  • The general approach of "given a claim, test it on indicator functions, then simple functions, then all integrable positive functions, then all integrable functions, then (if desired) integrable complex functions" is called the standard machine of measure theory, so there is educational benefit to seeing it.

I haven't thought it through, but I suspect the theorems you want to prove about integrals are probably easier to prove with the definition TurnTrout gave than with that one. 

What process do you use to review cards? Do you look at a prompt until you can say exactly what is on the card? Or if not verbatim what tolerance do you have for missing details/small mistakes?

Not verbatim. I mark the card 'correct' if I conjured the concept in the right way. If I was supposed to do a physics calculation, I don't pass it if I could recite the answer, I pass it if I went through the correct reasoning steps. "Calculate the gravitational force of a .1kg apple on a 70kg person, when they're 1m apart" -> ".1 * 70 * G / (1^2) = ?" is marked correct.

How do you find doing problems/exercises from these textbooks when you have prepared using Anki? And are you finding that earlier material seems obvious when reread?

Sorry if this is all coming across as critical and /or doubtful. I've tried to use Anki for theory before and dismally failed; the success you claim is very exciting and I'm trying to understand where I was going wrong.

So far I think I have focused too much on creating cards that can be memorised exactly (formulae and what-not), rather than having general concept cards that are used to develop fluency and familiarity (and later understanding), which sounds like what you are doing.

I think I do find Anki’d subjects much easier when I go back to them, yes.

I think that’s probably it - focus on concepts. If you like, I’d be happy to take a look at a deck you make / otherwise give feedback!

On using Anki for math, Michael Nielsen wrote up his own experience trying it in this essay: http://cognitivemedium.com/srs-mathematics 

I was previously skeptical that Anki could be used for anything more complex than "basic facts" (whatever I thought that meant), but Nielsen's essay changed my mind. 

Curated.

There's a lot of nuggets to appreciate here, but I'm particularly grateful to a reminder about "Hey, you know you can just go out and learn a bunch of things? It's not mysterious, here are some concrete ideas for how to go about that."

I also really like the concrete info in the comments about how TurnTrout made math anki work for him.

Very interesting article! A big part of the outlined techniques I also kind of discovered by trial and error.

I recently also started to work more on my math skills and putting in the mathjax was getting very time consuming. To reduce that bottleneck I can recommend this small app: https://mathpix.com

PS: I stumbled once over this collection of proofs without words and they make great Anki cards to build up intuition: https://mathoverflow.net/questions/8846/proofs-without-words

thank you! This is amazing. I wish I'd known about this sooner.

This is awesome! That's exactly the kind of post I wanted you to make!

About the lessons themselves:

  • I agree with the value of anki, but I have trouble finding things to ankify. My first impulse what to ankify everything under the sun, which lead to an anki burnout. Now I have the inverse problem of not finding much to put in Anki, mostly because I want to know/understand the concept before ankifying it. Or maybe I just don't read enough maths these days.
  • I want to disagree with the "read several textbook at once", but I think you're right. It's just that I'm trying so hard to focus on things and not jump from one to another all the time that reading multiple textbooks at once triggers all my internal alarms.
    I'll try to find a safe way for me to do that.
  • About not reading the whole textbook, I think I agree with the gist but I disagree with what you actually write. I definitely think that you should read most of a textbook if what you're doing is reading the textbook. On the other hand, you shouldn't try to master every detail in it.
    If you want to apply the pareto principle, then go through papers and write everything that you don't know. Then go search that in textbook. That's the efficient way. But reading a textbook is for getting a general impression of the field and building a map. So the latter chapters are useful. Just don't spend 20 hours on them.
  • The "read easier textbooks" advice looks like a rephrasing of "go just outside of your comfort zone". It makes sense to me.
  • I generally don't have the "approximate models" problem. But I also mostly read maths and computer science, which is additive instead of corrective.

 

And a problem this was. In early 2020, I had an interview where I was asked to compute . I was stumped, even though this was simple high school calculus (just integrate by parts!). I failed the interview and then went back to learning algebraic topology and functional analysis and representation theory. You know, nothing difficult like high school calculus.

I think this is symptomatic of a problem I have myself, and which I only understood lately. I want to learn the cool shit, and I studied a lot of the fundamentals in my first two years after high-school (where we did 12 hours of maths a week). So I should remember how to do basic calculus! But somehow I forgot. And everytime I study a more advanced book, I feel like I should brush up my analysis and my linear algebra and all that if I want to really understand.

Yet that's wrong, because I mostly read these advanced textbooks for one reason: getting a map of the territory. Then I will know where to look when I need something that looks like that. And for that purpose, getting all the details perfectly right is not important.

On the other hand, there are some parts of maths in which I want skills. There I should actually take the time to learn how to do it, instead of simply knowing that it exists and the big picture description.

About not reading the whole textbook, I think I agree with the gist but I disagree with what you actually write. I definitely think that you should read most of a textbook if what you're doing is reading the textbook. On the other hand, you shouldn't try to master every detail in it.
If you want to apply the pareto principle, then go through papers and write everything that you don't know. Then go search that in textbook. That's the efficient way. But reading a textbook is for getting a general impression of the field and building a map. So the latter chapters are useful. Just don't spend 20 hours on them.

I agree; I should have written "don't complete textbooks", and have edited that part accordingly. I'm thinking some combination of "don't do all of the exercises" and "don't spend time learning every single concept in the book."

Last year I tried re-learning calculus, and later learning some statistics, and both went kinda okay but I feel like I don't really have the scaffolding to make them actually useful longterm. I have a vague sense that it's good for me to have mathematical literacy but I'm never quite sure what to do with that belief.

This... I guess this comment is me saying "my default action is still to probably not invest much more effort here, but I am interested in anyone pitching me on a concrete strategy they think I would benefit from" (I think this is most useful coming from people who know me reasonably well and have some sense of what my deal is)

Nice article. Like Nielsen, you seem to have discovered Incremental Learning for yourself.

https://supermemo.guru/wiki/Michael_Nielsen_re-discovers_incremental_reading_with_Anki

If you want the whole package in software (i.e. not only the SRS part) give SuperMemo a try :)

https://help.supermemo.org/wiki/Incremental_learning

Instead of reading a textbook with SRS and notes, skim more books (not just textbooks), solve problems, read wikis and papers, talk to people, watch talks and lectures, solve more problems from diverse sources. Instead of memorizing definitions, figure out how different possible definitions work out, how motivations of an area select definitions that allow constructing useful things and formulating important facts. The best exercise is to reconstruct main definitions and results on a topic you've mostly forgotten or never learned that well. Math is self-healing, so instead of propping it up with SRS, it's much better to let it heal itself in your mind. (It heals in its own natural order, not according to card schedule.) And then, after you've already learned the material, maybe read a textbook on it cover to cover, again.

(I used to have a mindset similar to what the post expresses, and what it recalls, though not this extreme. The issue is that there is a lot of material available to study on standard topics, especially if you have access to people fluent in them and can reinvent parts of them on your own, which lets this hold even for obscure or advanced topics where not a lot is written up. So the effort applied to memorizing things or recalling old problems can be turned to better understanding and new problems. This post seems to be already leaning in this direction, so I'm just writing down my current guess at a healthy learning practice.)

Why "instead?"

I find that all of those activities enhance my SRS experience, and my SRS experience (sometimes dramatically) enhances all of those activities!

Particularly since I use SRS to capture concepts, intuitions, and rich relationships, not just isolated facts.  Rote memorization doesn't work very well with SRS anyway--as with everything, it works best when used to understand a topic in its full intuitive glory.

Leverage the Pareto principle, get 80% of the benefit out of the key 20/30/40% of the concepts and exercises, and then move on.

This is hard to instrumentalize regarding difficulty. I find that the hardest exercises are likeliest to be skipped (after struggling with them for an hour or two), but it doesn't follow that I can expect the easier ones (which I happened to have completed) to lie in that key 20%.

I like to randomly sample X% of the exercises, and read the rest; this lets me know later whether or not I missed something important. Simple rules like "do every fifth exercise" should suffice, with the rule tailored to the number of exercises in the book.

Fantastic insights!  I especially like how you've articulated the value of using "easy" books to connect concept to the "real world."  I've certainly run into the problem of trying to Ankify a big dense book, and getting bogged down with it.

Incidentally I'm also working through the Book of Why and Causality by Pearl.  Great progression (starting with one, then returning to the other).

I've been using Anki for math and computing for several years now, and one word of warning is that your big, dense cards (i.e. screen-shotted definitions) are the sort of thing that, in my experience, is easy to remember for a month or two (because rote memorization is filling in a lot of the glue), but becomes much more difficult (and trends toward ease hell) once intervals reach the high months/years range.

So I wonder if after a couple more months you might feel inclined to refactor your cards to use many small cards (rather than few big cards) to express definitions and theorems.

Personally, I've converged on a highly visual approach to chunking concepts into their "basic idea," followed later by detail cards to drill specific aspects of a larger whole.  This is obviously a nice approach of diagram-friendly areas like topology:

But also works for equations, algorithms, etc. too.

I've got some more examples here:

 * Advance machine learning concepts

 * Statistics
 

I think you'd get a lot of value out of using incremental reading, it improves the learning to memorisation workflow tremendously. Currently SuperMemo is the best at IR. (Post I wrote about this: https://gingerjumble.wordpress.com/2020/08/28/the-main-reason-you-should-switch-from-anki-to-supermemo/)

I'm a bit skeptical of what you claim because it is so different from my approach to becoming proficient at pieces mathematics: usually I will work through progressively more-complicated problems in excruciating detail. I don't claim that this is the most efficient method, and it would be nice to find such an approach, just that memorization methods have usually lead to me agreeing with the mathematics, rather than really understanding it.

But maybe you are using Anki differently to how I expect. How exactly do you review cards? Do you look at a prompt until you can say verbatim what is on the card? Or if not verbatim what tolerance do you have for missing details/small mistakes?

Anki's value depends a lot on how you use it.

If you use it for rote memorization, without grokking concepts deeply---then what you get is rote memorization.  Think of memorizing capitals without learning anything else about states and nations.

But if you work hard to identify conceptual landmarks and add good questions to tie it all together (more like a high quality PowerPoint presentation), then it can be amazing and facilitate retention of rich intuitions for years.  Think of memorizing major topographic landmarks on a map and a bit of history, so that you have something to relate capitals to in context.

Incidentally, rote-memorization cards are actually harder to review with Anki: the "glue" holding them in memory starts to fade after a month or so, so eventually the drift toward "ease hell" and otherwise become unpleasant to review.

I went into detail on how I use Anki for math in this thread. Feel free to comment there with further questions.

In early 2020, I had an interview where I was asked to compute .

A trick I learned from Jaynes:

... or, if you want to be a smartass about it, write "" in place of "" and tell people you're "differentiating by 1": .

All you actually need to remember is .

Neat article. Few things I myself have been struggling with and would like some insight into are:

  1. Were you able to address the issue of ankifying the algorithms? If yes, how?
  2. How do you handle the Anki burnout(or for that matter Anki inertia)? I am guessing with multiple subjects at a time, you at least have to review 100-200 odd cards per day(maybe more), and based on the non-atomic nature of your example cards, it seems to me that getting yourself to review that many cards every day is not a sustainable activity?
  3. How do you preserve your sanity between learning new things and practicing new cards? It seems to me that it can be taxing to spend time revising on your own while also learning new things.
  4. How do you handle the inertia that comes with creating new cards or editing cards for so many concepts and subjects?
  5. Does the SM-2 algorithm suffice your needs in terms of frequency, priority, scheduling, etc?
  6. Finally, have you found ways to avoid things like wrist issues, back issues, cognitive exhaustion, etc? Like maybe good computer etiquettes or something?

Sorry for the long list of questions. I am curious to know more because I have been struggling with getting my way into auto-didacting for more than 2 years now due to issues like being unable to differentiate between wanting-to-want and really want, burnout, haphazardness of it all etc. Any pointers on it would be really appreciated.

Not the OP, but got a few opinions there:

  1. Algorithms can be Ankified with the same strategies that work for other subjects: chunk the basic intuition into a short overview card, use diagrams, and hit the "basic idea" of how it works before adding more details cards for particular tricks & minutia.  As always, most cards should take <15 seconds to review (including time spent reading the card).
  2. Personally, I avoid burnout a few ways: 
    A) pair Anki with activities like taking a daily 1-mile walk or doing dishes (iOS voice commands help with the latter, so you can review hands-free),
    B) don't be shy about reviewing my professional development cards during work hours,
    C) deleting social media from my phone, so Anki is the go-to gap-filler app,
    D) always be learning a few new cards in each deck (since decks with 100% old material are less fun to review),
    E) refactoring: I set a reminder to work on marked cards every few days, so I can break down difficult cards into higher-quality material.  A little maintenance goes a long way.
    F) card design, card design, card design.  Well-designed cards are far less fatiguing to review than hastily made ones.
  3. Not sure I understand.  Anki kind of abolishes the line between reviewing and learning: you do both with the same system.  If you mean motivating yourself to study old material---my trick is to review from large, combined superdecks, so there is no pause between "Oh, I finished my learning decks, time to move on to old decks."  If I give myself that break, I'm more likely to procrastinate, but if I treat it all as one big deck there's motivation to finish it all in one go.
  4. I actually find it motivating.  Learning complex subjects is usually hard, because when you hit something hard there's an urge to procrastinate rather than figure out what's blocking you.  Anki removes these energy barriers for me: I can always learn a little more about quantum mechanics or whatever by adding a card or two.  It makes hard subjects easy, and makes it possible to say "I'll just study a little topology for 10 minutes" and actually make progress (which is usually impossible outside of SRS).
  5. I very rarely try to tweak the algorithm.  90% of the usefulness of SRS is just being able to handle the spacing schedule for you.  Tweaking it doesn't lead to much gain in efficiency (with a few exceptions: adding learning stages to avoid ease hell, and changing the reset interval to something like 20% or 50% instead of 0% both help avoid a lot of unnecessary reviews).
  6. Voice commands for everything, and audio cards for languages.  Hands-free, eyes-free reviews!

I have multiple times tried to get into an Anki habit and failed to keep it up. I think the main thing that makes me stop is that I try to make nearly every sentence of something that I'm studying into a card, because I have no idea what's worth remembering and what isn't. (As a general rule, throughout life, I suck at prioritizing.) The other thing, though, is that it feels like Work and things which feel like Work are Unpleasant and I procrastinate them. Do you have any advice for getting over that hump?

In general, when I first started using SRS I had this issue too. Over time, I got a fair bit better of it with 1 or 2 rules:

-I should be able to make at least 1 connection from the idea to something else (filter out only shallowly understood things)

-I should be able to either connect it to a situation of real life applicability or a goal

I didn't make it only: I need to be able to figure out applicability when making card, because a lot of useful knowledge might not that have it as obvious. But at the least, I should be learning it in a real context. E.g. some of my main goals now are:

-learn UX design

-understand how SRS/IR can be used to create genius

-get better at teaching

-become more resillient to disruption (emotionally)

Those are specific-ish and learning in context of those instead of I can use this sometime in the next decade is a useful filter

One way to start is to put in things you find yourself looking up. If you're reading math and you have to keep looking up the quadratic formula, put it in. If you're reading the 'Nash equilibrium' Wikipedia page, put in a few examples (and maybe even the formal definition!). 

Then, while you're reading a textbook, you might develop a taste for what things you'll probably look up later. 

The other thing, though, is that it feels like Work and things which feel like Work are Unpleasant and I procrastinate them. Do you have any advice for getting over that hump?

I like to think about Anki as giving me enduring experience points, which is Fun and Gratifying. Once you find it really work for you on something that matters, maybe you'll find it more exciting? I can only guess - I've naturally enjoyed Anki since nearly the beginning.

Hmm... I could try actually counting experience points. Like, each flash card reviewed grants a point, and every time I reach, say, a new Fibonacci Number of points, I gain a level and... um... stuff! The idea of leveling up really isn't very motivating by itself, but it would help.

Here's one idea: when reading a textbook or anything else I want to memorize, I might try to come up with just one question and answer pair about each page, and make a card out of that. Summarize the most important info on that page. Anything that's not too info-heavy, that should work. In things which are info-heavy, your method should work quite well.

Why did you write autodidacting instead of selfteaching? It just put a google search between me and reading the article...

It was the first word to come to mind. I changed the title, thanks. 

Thanks! Looks good.

Perhaps I am overthinking this, but when it comes to applying your knowledge gained through spaced repetition, is there a difference in how effectively you can apply it in these two situations?

  1. You spend time memorizing what a function is. A function comes up in a problem in a very similar format to how it looked on your Anki card. You are able to understand it because the prompt of the function in the problem causes you to recall your understanding from studying and then apply it.
  2. You spend time memorizing what a function is. A problem comes up that can be solved by using functions. However, the problem doesn't explicitly mention functions like in #1; it's up to you to recall your understanding of functions and recognize that this is a time to use them without seeing the prompt that you studied with.

In case this example is too weird or confusing, you can think of the using Anki for memorizing cognitive biases; you'll be able to recall the definition of them, but how much will spaced repetition help you recognize when cognitive biases pop up in your life (so that you can employ whatever counter-measures you see fit)?

I don't think spaced repetition runs into the failure mode of #2 if you do it competently. Good spaced repetition prompts you to think about the concepts on a spaced, expanding basis, not to recite the magic words on the back of the card. When I study a card, I'm trying to train urge to understand concept in prompt -> concepts on back of card, not words in prompt -> words on back of card.

For example, when I learned French with Anki, I translated entire sentences English-to-French (and vice versa). I learned vocab words by just finding sentences where I already knew how to say everything except the new word. By learning sentences containing the word for e.g. "swamp", I learned how to say "swamp" in normal, fluent conversation, without having to remember the exact sentence I used for Anki (as #2's model might suggest). 

I don't see any real difference in quality between normal study and Anki study, if you make your cards well.

You mention that planting a seed of Anki usage in 6th grade is a good thing. Do you have any thoughts about how to help a child get started with an Anki habit?

I don't, unfortunately. I don't have any special insight into children at this point in my life, and I can barely remember what my mental life was like back then. (Too bad I didn't use Anki to remember what it was like to be in 6th grade!)

There are always instrumental things to appeal to (e.g. "by using this application, you can do your work faster and have more time to play with your friends"), but that might take away from their intrinsic motivation if they view Anki as just a means to more free time.

I'd like to defer to LessWrong users who have actual experience with kids.

The creator of spaced repetition has talked about how usage in children of SRS is not a great idea here

 

Roughly idea is that self-direct learn drive based learning is needed for coherent, well connected models.

It's sort of like how I learned trig in high school and I can't apply it to a real world problem to save my life. If I'd learned trig for myself, when I thought it'd be useful, it'd be way more applicable and not be so siloed to 'this is how you use trig to solve math problems on a test'