Epistemic status: This is a speculative idea based mainly on my personal experience. I have not conducted a systematic literature review.

Very brief summary

When I first started using spaced repetition with AnkiMobile, I thought it was extremely tedious to manually create cards by entering English words and their Japanese translations myself. So I had AI transcribe a vocabulary book and repeatedly copied and pasted the words one by one. I thought this was much easier.

A few days later, while I was doing this work, the AI suggested at the end of an interaction, “Would you like me to output this in TSV format?” That was the first time I learned that TSV existed.

I then learned that by pasting a vocabulary list into a text editor in TSV format and changing the file extension to .tsv, I could import all the cards at once. This was dramatically faster than my previous method, but I still had to make the AI output a large number of words and translations directly in the chat. As the amount of content being loaded became too large, my device sometimes froze, and I still had to repeatedly create files. I continued using this method for about five days.

Later, when I tried to create another deck, I wanted to insert images. However, there was a limit to the number of images that could be handled in a single AI chat interaction, and inserting images using TSV files was also difficult.

That was when I first learned that Anki had a file format called APKG. By simply sending the questions and images and having the AI create the file internally and output only the file, I no longer needed to load a long TSV text for copying. I only needed to directly import the file generated by the AI.

Even then, however, it was difficult to complete an entire deck in a single interaction with AI. Especially for decks containing large numbers of images, dozens of interactions and imports were necessary.

Then, one day, I happened to become curious about the features of the desktop version of Anki and simply asked about them. I was told that one characteristic of the desktop version was its large number of add-ons, and that by using AnkiConnect, data could be inserted programmatically in JSON format.

At that moment, I realized that the hundreds of hours I had spent creating Anki decks might have been something I could simply have delegated to an AI agent.

Now, for example, if I want to check the locations of countries that appear in my geography deck, I can instruct an agent to mechanically generate world maps, mark the relevant locations, and insert those images into existing cards, allowing me to edit thousands of cards at once.

I will give another example.

If I ask how to improve at shogi, the AI gives answers such as:

solve tsume-shogi problems, review your own games, learn joseki as structures rather than as sequences of moves, look at many high-quality game records, and verbalize your evaluation of positions.

These may certainly be effective. However, could an idea such as using a spaced-repetition tool to increase one’s knowledge of joseki also be included?

What I want to focus on here is something common to both examples: throughout these conversations, I consistently had the goal of studying or acting more efficiently, yet even though the AI should have possessed knowledge that could help with that goal, it mostly presented concepts that were close to the context I had already entered.

AI can answer when directly asked, “What is TSV?”, “What is AnkiConnect?”, or “What is spaced repetition?”

But if the person asking the question does not know that the concept exists, they cannot ask that question in the first place.

For example, there are concepts that may be useful for problem solving, such as spaced repetition, the 80/20 rule, Goodhart’s law, and deliberate practice.

If the person does not know that these concepts exist, is there not a considerable amount of randomness involved in whether they ever encounter them?

The two examples above may still involve concepts from relatively nearby fields.

However, from my personal experience, before I knew about the context of rationality, there were many situations that, looking back now, make me think, “I should have been careful about Goodhart’s law here,” “I should have considered deliberate practice here,” or “I could have used spaced repetition here.” Nevertheless, concepts related to rationality were almost never spontaneously introduced in my conversations with AI.

So I would like to build, on a small scale,

an AI system that, when given a question motivated by a goal such as “I want to achieve a good outcome” or “I want to accomplish something,” does not rely only on the fields already known and entered by the questioner, but instead searches across different fields for concepts that may be useful for solving the problem, and proposes those concepts either individually or in combination,

and compare it experimentally with an ordinary LLM.

For example, instead of simply generating an answer, the system might work as follows:

extract the user’s objective → search concept groups from multiple fields → select promising concepts → apply them to the original problem → generate an answer

I would then compare

an ordinary LLM

with

an LLM augmented with this mechanism.

Using human evaluation and other methods, I would like to measure things such as the proportion of cases in which it presents new concepts, the usefulness of the concepts presented, the proportion of cases in which it produces solutions that would not have occurred in an ordinary answer, and the proportion of cases in which it presents irrelevant concepts.

An image of the method I am currently considering

One reason I want to pursue this research project, apart from my personal interest, is that I want to make a better discovery about a possible research-project topic to use for the comprehensive selection entrance examination for the School of Computing at Institute of Science Tokyo, a university in Japan.

Considering the distribution of LessWrong users, there do not seem to be very many Japanese users, so I need to provide at least a minimal explanation of the Japanese university admissions system in order to explain the background.

First, in order to apply to a Japanese national university, students take something called the Common Test in January. Then, in February, there is an individual university entrance examination in subjects such as English, mathematics, chemistry, and physics.

Some people may wonder why two examinations exist. The Common Test is administered nationally, while the individual examinations are created by each university, and their difficulty and tendencies differ greatly from university to university.

Each university converts the Common Test score according to a certain weighting, and admission is then determined based on the combined score of the Common Test and the university’s own entrance examination.

However, universities also have distinctive admissions systems separate from this standard route.

For example, in the admissions method I am currently considering, the first-stage selection is based only on the Common Test and an evaluation of a research project completed before the entrance examination. In the second stage, admission is evaluated only through an interview of approximately 30 minutes.

The advantage of this method is obvious.

Compared with the ordinary admissions route, the required level of academic performance is relatively lower.

Under the ordinary route, approximately 500,000 people take the Common Test each year, and to gain admission to a university at approximately the academic level of Institute of Science Tokyo, one would, by a simple calculation, need to be in roughly the top 2–3%.

In Japan, many students also invest a very large amount of time into university entrance examinations, so being in the top 2–3% is by no means a low standard.

To use an analogy, it is like trying to pick a fruit at the top of a Sequoia sempervirens tree.

I want to pick a fruit from a lower tree.

On the other hand, in an admissions route that places greater emphasis on a research project, although this varies depending on the school one applies to, approximately 40 students apply, around 12 pass the first-stage selection, and ultimately around six are admitted.

The research projects submitted by successful applicants are at a very high level for high-school students, but they do not appear completely out of reach.

I therefore want to search for a research-project topic that could realistically reach the level required for admission.

For that reason, I would like to organize my current thinking and receive opinions from outside.

As a premise, although this is an admissions process for the School of Computing, the range of topics that can be submitted as research projects is quite broad.

For example, one may submit content about mathematical investigations, what one learned through organizing a programming class, what one learned through participating in competitions such as competitive programming or the Chemistry Olympiad, or something based on a study-abroad experience that one believes connects to the field of information science and engineering.

There is also an interview after passing the first-stage selection, during which questions are asked based on what the applicant has written.

Therefore, even if the Internet and AI can be used during the research process, if someone writes something that is too far removed from their actual ability, it will quickly become apparent.

I also expect that such superficial thinking would be easily detected and might prevent the applicant from passing even the first-stage selection.

There are also restrictions on the submission format.

The research project must be submitted as a four-page Word file, with a font size of at least 10 points, while the layout itself is otherwise unrestricted.

The research project may be conducted independently by the applicant, conducted independently while receiving guidance from a teacher or someone else, or conducted collaboratively.

At first, based on the Theory of Change and my own observations, I observed and analyzed accounts written by successful applicants as well as examples published by the university.

What I learned from these observations and analyses

• First, if I reduce the contents of the research projects submitted by successful applicants to a very rough scale that would be understandable to anyone, and probably far too crude for rationalists, I found that they ranged from approximately early undergraduate to senior undergraduate level.
• I found that some degree of novelty in the research content is valued.
• I found that importance is placed on the entire process of personally identifying a problem in a field, thinking of a solution, and testing it.
• I found that the attitude of pursuing something one personally found questionable and actually conducting experiments is valued.
• I found that it is valued when what the applicant learned through the research connects to what they want to study at university in the future.
• The research content itself does not necessarily have to be highly advanced. What appears to be highly valued is the attempt to solve unresolved problems that are practically relevant within the field, as well as the presence of experiences and insights specific to that research.
• I found that it is highly valued when the contents of the report allow the university to judge that the applicant fits the admissions policy stated by the university.

Finding a research project that satisfies many of these conditions is certainly difficult, but there appears to be a certain standard that evaluated research projects reach.

Conversely, I thought that if I could reach that standard, it could differentiate me from other applicants and increase my probability of passing the first-stage selection.

However, there are also many problems.

• First, the number of research projects on which the observations and analysis above are based is small. This admissions system itself is relatively new when viewed in the context of the history of Japanese university admissions. Only around six applicants are admitted each year, and very few of them leave records of their experiences for future applicants. In addition, there is no data on what kinds of research projects were submitted by applicants who failed the first-stage selection. Therefore, the overall level of all applicants is unknown, and I expect there is also selection bias among successful applicants who choose to publish their work.
• The rise of AI is also a problem. Applicants are presumably people with at least some level of interest and ability in information-related fields, and I expect that they use models at roughly the level of 5.6 Sol in some form, although their frequency of use may differ. It is also reasonable to expect that applicants are at least somewhat skilled in how they use these models, which could significantly raise the overall standard of research projects. In other words, rather than how technically advanced the content is, things that are difficult for AI to substitute for, such as experiments and statistics, may become important sources of differentiation. Therefore, the difficulty of research examples provided by the university last year or several years ago may not be very useful for estimating the current competitive standard.
• Another problem is that I do not have anyone I can consult. In accounts written by successful applicants, they often received some form of feedback from university students or researchers through social media such as X (Twitter). However, I do not use X in the first place, and I judged that meeting someone I could consult about a topic like this would be quite difficult under my current circumstances, so I decided to post on LessWrong.

In closing

This is still an immature concept, so there may be parts that I have not organized well.

I would like people to point out the rough or weak parts.

The purpose of this post is to receive criticism and opinions from outside before carrying out the project, and to revise the research plan itself.

Therefore, please understand why I have not written anything about the specific tools I would use or the exact procedures I would follow.

I also assume that there are already many similar lines of prior research, including ARG and other approaches.

However, I would like to distinguish this project by deliberately introducing a rationality-oriented context optimized toward producing results.

Also, although this is still only at the conceptual stage, I would like to adopt a system with as few barriers to participation as possible, so that in the future, individuals could add “branches” of knowledge or concepts in fields in which they are knowledgeable, or supervise and review their contents.

By doing so, knowledge from different specialized fields could gradually become connected, and I hope this might make it possible for new combinations of concepts and new problem-solving methods to emerge from fields that would otherwise be unlikely to connect.

In addition, rather than ending with a research prototype, I would also like to make the system publicly available in a form that can be used immediately.

I want to make it usable even by people without specialized knowledge such as programming, while also allowing them, when necessary, to participate in adding or improving knowledge.

Through this, rather than merely testing whether “an LLM can explore concepts across different fields,” I am also wondering whether this could eventually develop into a system in which many people contribute their own specialized knowledge, continuously expanding the conceptual space that can be used for problem solving.

This post was written based on advice I received from Viliam.

This post was translated from Japanese into English using the following prompt:

“Translate this into English while staying truly faithful to the original text.”

The image of the method I am currently considering was created with AI based on content I wrote myself.