Summary
- Current LLMs, in particular GPT-4, are sufficiently capable to produce good non-trivial flashcards covering the contents of a given text.
- It is then possible to automate the flashcard creation process, making it far less time consuming to use Spaced Repetition techniques to study new material.
- In this post, I explain how you can apply this technique, and show its result when applied to the whole "Rationality: From AI to Zombies" sequences, and to scientific papers.
Why automate?
I've been using Spaced Repetition techniques for the past 12 years, mainly through Anki. It is a common mantra among Spaced Repetition advocates that you should create your own flashcards, if you want to learn something new, as it helps you to consolidate the information you're putting into the card. While that is certainly true, it is far more questionable whether it is an efficient method to strengthen your memory.
From a time-saving point of view, this makes little sense. Summarization, which is mainly what you do when creating a card by hand, is a relatively poor learning technique, particularly when compared with Distributed Practice, the whole schtick of Spaced Repetition[1].
If you spend, on average, 10 seconds reviewing a card, that card will require a total of 90 seconds of review time in the next 5 years, on average (with Anki's default scheduling).
Creating a card takes longer than that. You first need to filter the important ideas from the text you are interested in. Then, you need to express them in a way that is suitable[2] for a flashcard, and that leads you to remember what you actually want, and not some proxy information. Finally, you actually need to create the card. How long all of this takes depends on the particular text, but in my experience, it averages to more than 90 seconds. Especially for cards from complicated contexts, creating the card may take more time than all the reviews combined.
In general, there are three important things to consider:
- How relevant is a deck of flashcards for what you actually want to learn or remember, versus some proxy objective;
- What is the deck's quality - we may consider this to be how well the deck does the job it set out to do;
- Finally, how much time did it take you to create or adapt the deck for your use case.
It is easy to create cards quickly, if you are willing to sacrifice their quality. It is also easier to use public decks than to create your own, but they may be less relevant for your use case.
Consider a personal example: last year, I wanted to improve my French. I had two options: create my own deck, your adopt someone else's. After searching through the online Anki decks, I found this deck, which seemed to do the job. The deck's goal is specifically to strengthen your vocabulary, but my goal was to improve my French overall. The actions required to reach both goals are similar, but not equal. By using only this deck, I would miss out on important word usage context that is hard to learn without looking at whole sentences, and I would generally miss out on French grammar. Regarding its quality even for its main goal - learning vocabulary - it is very good, but not perfect. The cards don't come with enough context to distinguish synonyms. So if a card asks for the French word for "year", I can't tell whether it wants "an" or "année". After importing this deck to Anki, I removed some parts that made the deck bulky and hard to use on AnkiDroid. Moreover, as I reviewed the cards and ran into synonyms, I would edit the card to add some hint as to what synonym was intended. Overall, I spent about 1 second, on average, editing the cards (since most cards didn't require any editing by hand).
Guesstimating some quantitative score for the relevance, quality, and time aspects of the cards, I would say this French deck has a score of 0.95 for vocabulary, 0.8 for relevance to my case, and I spent 1 second per card on it. If I were creating these cards, I would have probably created something with a score of 0.8 for vocabulary, but 0.95 relevant for my case, while spending orders of magnitude more time creating the cards. This trade-off is simply not worth it, and it is generally preferable to use public decks as a starting point, if they are available.
What if there are no available public decks?
As I show in this post, current LLMs, such as GPT-4, are good enough that you can delegate most of the card creation process to them.
When trying to study a novel article, I could create my own cards, thereby spending 90 seconds per card on cards that were 0.8 good and 0.9 relevant for what I wanted, or I could automate the card creation process, and spend 1 second per card in cards that were 0.9 good and 0.8 relevant. The same benefits of using public decks apply here.
Note that people tend to overestimate their skills at creating good cards. Especially when creating cards en masse by hand, there is a strong tendency to start simplifying by focusing on the information that is easy to put in flashcards, vs. the information you actually care about, but which is hard to encode into a card. So you should probably lower your expectation of how good your cards actually are (unless you have considerable experience to back up your estimate).
How to create cards automatically
Prompt
After some testing, I've converged into using variants of this template prompt, for GPT-3.5 or GPT-4:
Create [number of cards] concise, simple, straightforward and distinct Anki cards to study the following article, each with a front and back. Avoid repeating the content in the front on the back of the card. In particular, if the front is a question and the back an answer, avoid repeating the phrasing of the question as the initial part of the answer. Avoid explicitly referring to the author or the article in the cards, and instead treat the article as factual and independent of the author. Use the following format:
1. Front: [front section of card 1]
Back: [back section of card 1]2. Front: [front section of card 2]
Back: [back section of card 2]... and so on.
Here is the article:
"[Article]"
In my experience with the Rationality deck (see below), if the word count of the article is , you can aim for cards. In general, you should avoid (if you can) breaking the article into smaller parts, each processed independently, as the LLM will lose important context to generate the cards. The template prompt contains pieces of information that are relevant for the LLM to know:
- "concise, simple, straightforward": otherwise, GPT-3.5/4 has some tendency to add a lot of text to the back of the card, which goes against some flashcard design principles.
- "distinct": mainly to avoid it creating cards covering the same information.
- "Avoid repeating the content in the front on the back of the card. In particular, if the front is a question and the back an answer, avoid repeating the phrasing of the question as the initial part of the answer.": very common issue otherwise, GPT-3.5/4 defaults to adding unnecessary information to the back of the card.
- "Avoid explicitly referring to the author or the article in the cards, and instead treat the article as factual and independent of the author.": by default, it tends to not treat the information in the article at face value. It would regularly add "According to the article/author" to the front or back of the cards, which is unnecessary.
- "each with a front and back" and "Use the following format": By keeping the format consistent among cards, it is easy to post-process them with a script to add to Anki. GPT-4 always follows the template, while GPT-3.5 only does sometimes. It is possible that adding an example card to the end of the prompt would help, but I haven't checked.
See the images below for examples of some of these issues.
Minor caveats
If the source text is too long, cutting it off into smaller parts to be processed independently may lead the LLM to lose important context. In practice, this may lead to similar flashcards being created during different iterations. More rarely, the lack of context may lead to wrong information on some of the cards.
Without additional context, flashcards may be created for content that is not interesting to you personally. I suggest asking for more cards than you actually want and then deleting the irrelevant cards, as they are reviewed by you.
GPT-4 (and even more so GPT-3.5) has a tendency to fall into repeating patterns when creating the cards. For instance, if the first card it creates has a one-word answer, it is more likely to formulate subsequent cards to have simple answers as well. If the first card repeats the phrasing of the question at the start of the answer, subsequent cards are more likely to do so. This may be avoidable with a better prompt, but I haven't managed to solve it so far.
Examples
For these examples, I mainly used GPT-4 through the browser interface. See the next section for a comparison with GPT-3.5.
Rationality: from AI to Zombies deck
As a proof of concept, I applied this technique to the whole "Rationality: From AI to Zombies" sequences, resulting in almost 4800 cards. I applied it almost entirely algorithmically. If the entry contained words, it aimed to create cards. If the word count surpassed 2000, the LLM would generally reach the limit of its context window. In this case, I would try to break the article by hand into smaller chunks, that could be processed independently. This should be possible to do automatically using the GPT-4 API, but since I don't have access to it, I've done these steps by hand through the chatGPT online interface (see example here).
I asked GPT-4 to write a Python script to then take these cards (saved into a text file) and restructure them to be easily imported into Anki.
The full deck, containing 4769 cards, can be found here. I have gone through the Book I subdeck so far, and I can certainly say I would not have done a better job myself, by hand. The Anki cards in this deck contain additional fields besides the Front and Back: the Subject, in bold, corresponding to the post's title, and the Source, as a link to the post (hidden when reviewing the card). These were filled in with a Python script (written by GPT-4), based on the post they came from.
Note that it would have taken me months to create the full deck by hand. If each card indeed took 90 seconds to create, it would have required 120 hours of work. Instead, it only required me to input a few hundred prompts into the browser interface. This process could have been automated entirely with access to GPT-4's API.
Due to this automated approach, some cards are silly, like those associated with a story. But these can be simply deleted from the deck afterward, based on personal preferences.
Scientific paper
It is especially hard to create decent cards to study scientific papers, as it is easy to fall prey to card creation patterns that lead to cards that are not very useful in the long term[2]. Here, I attempted to automatically create study cards for the paper Quantum spectral methods for differential equations. Since I've personally studied this paper extensively, I could judge the quality of the resulting cards. For this test, I gave the paper to GPT-4 in its LaTeX format (I could alternatively have used ar5iv.org, as the browsing variant of GPT-4 uses).
Overall, GPT-4 is quite good at parsing the text-heavy sections of the paper, such as the abstract, introduction, and conclusion, and summarizing them into well-structured cards.
Unfortunately, it is more likely to fall into undesirable patterns parsing the math-heavy sections.
For these sections, I obtained somewhat better results by adding "Focus on the mathematical aspects of the article, including its equations, and feel free to use LaTeX to write them." to the template prompt.
The resulting cards may require additional post-processing, if they contain LaTeX code.
Overall, for hard subjects such as scientific papers, I would suggest using LLMs as a starting point to create cards for your study, while being careful to fill in any gaps left by them, especially in the harder sections requiring deeper context.
Comparison between GPT-3.5 and GPT-4
Based on my experience so far, I would not recommend using GPT-3.5 for harder texts, at least not without improving on the template prompt given above. The issues described here are also present when using GPT-4, but are so rare that they don't really constitute an issue.
GPT-3.5 may not stick with a consistent template for creating the flashcards, forcing you to post-process them by hand in order to import to Anki. It falls more easily into the undesirable patterns described in the previous sections.
GPT-3.5 is also more likely to create cards that make little sense without the original context.
Conclusion
Surprisingly few people use Spaced Repetition techniques to improve their learning, despite extensive research advocating for their use. If my experience is anything to go by, it may be due to how time-consuming creating the flashcards turns out to be, especially when you want to remember something that is not so easy to encode into cards.
Hopefully, by automating the card creation step with LLMs, you can now focus on the card reviews themselves, and feel it is easier to use Spaced Repetition techniques to learn novel subjects you encounter in your life.
If you have any feedback, feel free to share it below in the comments!