Recently, there is a lot of activity on AI driven autoresearch style work14. These are systems that generate a research idea, run the experiment, and write up the result with no human in the loop. A significant part of an autonomous AI researcher is the ability to be curious about the world, come up with the right questions to ask and explore different ‘what-if’ scenarios. While the current discourse mainly focuses on achieving a research goal set by the user, the act of setting the goal itself is deeply important. These open-ended research systems need to be curious if they can possess the ability to autonomously add net new knowledge to the world. The AI Scientist14 is a useful concrete example of where this stands today: it can generate an idea, run the experiment, and write the paper, but the idea still starts from a human-provided template and research direction. It is good at the goal-achieving part of research and has nothing that plays the role of noticing, on its own, that something is worth looking into in the first place.
Adding curiosity as a trait in agentic systems
Adding a piece of knowledge or a trait in an agentic system can be done at 4 levels :
What's in the data determines what the model has seen in training. The architecture determines what shapes of computation are representable. The optimizer determines what gets rewarded. The harness is everything wrapped around the trained model: the prompt, the sampling procedure, the tools, the other agents in the loop.
Adding curiosity has been tried at each one of these layers in the past, by different methods, for different reasons.
Data
Cross-entropy pretraining is imitation. At the population level, minimizing it recovers the training distribution exactly. If a prefix in the corpus is genuinely followed by five different valid continuations, a well-trained model's own output distribution reflects that variety, because cross-entropy is minimizing KL(data ‖ model), and that particular direction of KL divergence punishes the model severely for failing to cover something the data does. It does not punish the model for spreading probability mass around; if anything it's biased the opposite way. So whatever diversity, weirdness, or creative range exists in a base model's outputs is the diversity that was already sitting in the training set.
So the data layer is where creativity gets inherited. Nagarajan and colleagues1 make roughly this argument from a different angle: next-token training is myopic — the objective gives no credit for setting up a creative multi-step structure — and the standard fix, sampling with temperature, doesn't really address this either, since it adds noise uniformly rather than concentrating it where the data was genuinely ambiguous, which is closer to a data-and-sampling-design question than an objective-function question. Cross-entropy has no way to tell a token that's uncertain because the data forked here from a token that's uncertain for no interesting reason.
Architecture
The next layer is network: designing an architecture structurally disposed to notice novelty, independent of what loss it's trained with. There has been exploration of this in RL literature. Never Give Up (Badia et al., 2020)2 gives an agent an actual episodic memory module (a running store of states it's visited recently) and computes a novelty bonus from nearest-neighbor distance in that memory. This was built for a specific, hard failure mode: sparse-reward games like Montezuma's Revenge and Pitfall, where an agent doing undirected random exploration essentially never stumbles onto a reward signal at all, so there is nothing for ordinary RL to reinforce. The episodic memory and novelty bonus exist to give the agent something to chase in the absence of any reward signal to follow. Random Network Distillation (Burda et al., 2018)3 adds a second, randomly initialized and permanently frozen network, train a third network to predict its outputs, and use the prediction error as a novelty signal. Both of these are hybrids in practice; they add a new piece of architecture and a new piece of loss at the same time; indicating that this layer and the optimizer layer are hard to cleanly separate. Also,while architecturally-grounded detectors of this form are good at noticing that something is unfamiliar, they might not necessarily translate to useful novelty.
Optimizer
This is the layer with a mechanism that explicitly, mathematically says "keep exploring" as part of what's being optimized. Standard RL practice since at least the original A3C paper4 adds an entropy bonus directly into the objective (reward minus β times the policy's own entropy) which punishes the policy for collapsing onto one action too early. Soft Actor-Critic16 turns this into an algorithm's central objective rather than an add-on term. This keeps the policy from collapsing onto one action, but it is not the same as rewarding novelty directly — a state visited a thousand times can still have high action-entropy. Count-based exploration bonuses17, curiosity-driven prediction-error rewards18, and information-gain objectives19 put actual state-novelty into the reward itself, rather than just the spread of the action distribution.
There's a second thing that happens at the optimizer layer that makes deployed chat models feel narrower than their base checkpoints21. RLHF-style fine-tuning is typically regularized toward a reference policy with the opposite kind of KL divergence from pretraining — KL(policy ‖ reference) instead of KL(reference ‖ policy) — and that direction is mode-seeking rather than mass-covering20. The model is free to abandon whole regions of its own base distribution as long as it doesn't wander somewhere the reward model dislikes. Thus the optimizer can influence a model whose data-layer diversity was already there and is now being actively narrowed.
Harness
Everything above happens during training. The harness, by contrast, sits around a trained model at the moment of inference. This is the layer with the lowest cost of checking a real outcome, without involving expensive gradient updates. The clearest existing version of this is retrieval15: handing the model documents or tool output at inference time so a claim can be checked against something outside its own weights, cheaply and without touching a single gradient. Reflexion7 and Voyager9 are the clearest existing curiosity attempts at this layer — revising behavior from an observed outcome without a weight update — and ExpeL22 extends the same idea across many tasks instead of one, extracting natural-language insights from both successes and failures. On the flip side, a harness that forces divergence without a real, grounded gate on the result is just a more expensive way to get what an entropy bonus already gives you: wider spread without any information about which part of the spread is worth anything. Co-Scientist23 is another recent example: many agents generate and debate hypotheses, but instead of using just the divergence, a tournament-style ranking against evidence is the actual grounded gate deciding which ones matter.
Where does curiosity live
Biology has its own answer to what should count as a good place to explore, worked out by evolution. One of the prominent results is dopamine. Schultz, Dayan, and Montague5 found that dopamine neurons do not fire in proportion to reward, they fire in proportion to reward prediction error: more than expected, a burst; exactly as expected, nothing; less than expected, a dip below baseline. Kidd, Piantadosi, and Aslin6 ran an experiment in 2012 around the learning habits of infants, and called it the Goldilocks effect = infants don't chase maximum surprise, they chase a specific, narrow, moving band of surprise: not too predictable, not too chaotic, just at the edge of what they can currently make sense of. There's no reward model, no loss function, no dataset of good things to look at. Whatever is producing that curve is either built into the wiring before birth, or it assembles itself extremely fast from a few months of raw sensory experience, or both.
A four-month-old has an innate disposition (lets call it architecture) wired directly into a fast, embodied feedback loop with the physical world (harness). The loop is what turns looking into learning, fast enough that the next look is already informed by the last one.
Nothing we have currently possesses all elements of that coupling. We have architectural novelty detectors that never see an outcome. We have optimizers that see outcomes but only in slow, offline, batched updates. We have harnesses that are close to real outcomes but usually do not influence weight updates. One way to bring this all together would be an update rule connecting harness-observed outcomes back into the model's behavior. A design for this has at least three properties.
Fast: updates on the timescale of an episode or sub-episode, not a training run.
Grounded: the update comes from an actual observed outcome, not a proxy computed before anything happens.
Fine-grained: the credit lands on the specific decision responsible, not spread across the whole attempt.
For example, Reflexion7 has an agent write down why an attempt failed and hands that reflection to the next attempt, no weight update involved. Algorithm Distillation8 gets a model trained to improve its own policy over a context window in-context, on a timescale of tokens rather than training runs. Voyager9's skill library grows the moment code passes or fails, and that library shapes every attempt after. Reflexion and Voyager get fast and grounded. Neither gets fine-grained, as their unit of update is the attempt, not the token. Entropy-branching methods such as RND3, NGU2 are only grounded in their prior state.
Measuring curiosity
A simple setup for baking in curiosity inside an agentic system can be: Start with SFT on a curated set of open-ended, exploratory text traces, teaching a model to imitate the surface behavior of curiosity directly. Then run RL on top, using a judge or a rubric that scores something like curiosity or divergent thinking as part of the reward. Track a curiosity-marker metric across RL training steps, hoping to see it improve in an emergent fashion, similar to the way chain-of-thought length climbed over the course of DeepSeek-R110's RL training.
How will we measure this said ‘curiosity’ which leads to an ‘inspiration’ towards ‘novelty’? Checking the value of said novelty requires a verifier, and for open-ended tasks there usually is not one. Perhaps one way is to learn from attempts to measure this in humans. IQ testing started almost entirely convergent, one correct answer that is easy to grade and hillclimb against. Guilford11's structure-of-intellect model argued in the 1950s that this missed half of what intelligence does, and proposed divergent thinking, generating many varied valid responses to an open prompt, as a distinct factor.
Another approach is prospective grounding: score a generated idea not by a judge's opinion at generation time, but against what actually happened later. Some recent research-idea benchmarks12 specifically use papers published after a model's training cutoff, so a generated research idea can be checked against real work nobody could have leaked into training. On the flip side, that is slow and only works retrospectively. One caveat here is that a judge rewarding curiosity markers can be satisfied by a model that learns to ask more questions and hedge more visibly without actually exploring anything useful.
The third is to not score the object holistically, instead score the effect of a specific decision inside it. This is called ‘fork advantage’ - by holding the prefix fixed, forcing one specific token, and rolling out k times to see what one decision actually causes. GRPO13 already does a version of this once at the very start of generation. Fork advantage is what happens if you run that same move at every candidate position instead of only the root - which is why the cost multiplies by the number of positions. It can say a specific token mattered and in which direction.
Maybe it's fair to say that curiosity exists in bits and pieces across multiple components of an agentic system - architecture, optimizers, harnesses and the data. It remains to be seen how an end-to-end pipeline that creates an open-ended curious model will shape out. This in turn will truly enable current research agents to have real autonomy - from conceptualization, validation to E2E iteration until success.
References
1. Nagarajan, V., Wu, C.H., Ding, C., Raghunathan, A. (2025). Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction. ICML. arXiv:2504.15266.
2. Badia, A.P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., et al. (2020). Never Give Up: Learning Directed Exploration Strategies. ICLR. arXiv:2002.06038.
3. Burda, Y., Edwards, H., Storkey, A., Klimov, O. (2018). Exploration by Random Network Distillation. arXiv:1810.12894.
4. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. ICML. arXiv:1602.01783.
5. Schultz, W., Dayan, P., Montague, P.R. (1997). A Neural Substrate of Prediction and Reward. Science, 275(5306), 1593-1599.
6. Kidd, C., Piantadosi, S.T., Aslin, R.N. (2012). The Goldilocks Effect: Human Infants Allocate Attention to Visual Sequences That Are Neither Too Simple Nor Too Complex. PLOS ONE, 7(5), e36399.
7. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS. arXiv:2303.11366.
8. Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., et al. (2022). In-context Reinforcement Learning with Algorithm Distillation. arXiv:2210.14215.
9. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291.
10. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
11. Guilford, J.P. (1950). Creativity. American Psychologist, 5, 444-454.
12. Guo, S., Shariatmadari, A.H., Xiong, G., Huang, A., Xie, E., Bekiranov, S., Zhang, A. (2024). IdeaBench: Benchmarking Large Language Models for Research Idea Generation. arXiv:2411.02429.
13. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
14. Lu, C., Lu, C., Lange, R.T., Foerster, J., Clune, J., Ha, D. (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv:2408.06292.
15. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. arXiv:2005.11401.
16. Haarnoja, T., Zhou, A., Abbeel, P., Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML.
17. Bellemare, M.G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., Munos, R. (2016). Unifying Count-Based Exploration and Intrinsic Motivation. NeurIPS. arXiv:1606.01868.
18. Pathak, D., Agrawal, P., Efros, A.A., Darrell, T. (2017). Curiosity-Driven Exploration by Self-Supervised Prediction. ICML.
19. Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., Abbeel, P. (2016). VIME: Variational Information Maximizing Exploration. NeurIPS. arXiv:1605.09674.
20. Minka, T. (2005). Divergence Measures and Message Passing. Microsoft Research Technical Report MSR-TR-2005-173.
21. Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., Raileanu, R. (2024). Understanding the Effects of RLHF on LLM Generalisation and Diversity. ICLR. arXiv:2310.06452.
22. Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., Huang, G. (2024). ExpeL: LLM Agents Are Experiential Learners. AAAI. arXiv:2308.10144.
23. Gottweis, J., Weng, W.-H., et al. (2026). Accelerating scientific discovery with Co-Scientist. Nature. arXiv:2502.18864.
Recently, there is a lot of activity on AI driven autoresearch style work14. These are systems that generate a research idea, run the experiment, and write up the result with no human in the loop. A significant part of an autonomous AI researcher is the ability to be curious about the world, come up with the right questions to ask and explore different ‘what-if’ scenarios. While the current discourse mainly focuses on achieving a research goal set by the user, the act of setting the goal itself is deeply important. These open-ended research systems need to be curious if they can possess the ability to autonomously add net new knowledge to the world. The AI Scientist14 is a useful concrete example of where this stands today: it can generate an idea, run the experiment, and write the paper, but the idea still starts from a human-provided template and research direction. It is good at the goal-achieving part of research and has nothing that plays the role of noticing, on its own, that something is worth looking into in the first place.
Adding curiosity as a trait in agentic systems
Adding a piece of knowledge or a trait in an agentic system can be done at 4 levels :
What's in the data determines what the model has seen in training. The architecture determines what shapes of computation are representable. The optimizer determines what gets rewarded. The harness is everything wrapped around the trained model: the prompt, the sampling procedure, the tools, the other agents in the loop.
Adding curiosity has been tried at each one of these layers in the past, by different methods, for different reasons.
Data
Cross-entropy pretraining is imitation. At the population level, minimizing it recovers the training distribution exactly. If a prefix in the corpus is genuinely followed by five different valid continuations, a well-trained model's own output distribution reflects that variety, because cross-entropy is minimizing KL(data ‖ model), and that particular direction of KL divergence punishes the model severely for failing to cover something the data does. It does not punish the model for spreading probability mass around; if anything it's biased the opposite way. So whatever diversity, weirdness, or creative range exists in a base model's outputs is the diversity that was already sitting in the training set.
So the data layer is where creativity gets inherited. Nagarajan and colleagues1 make roughly this argument from a different angle: next-token training is myopic — the objective gives no credit for setting up a creative multi-step structure — and the standard fix, sampling with temperature, doesn't really address this either, since it adds noise uniformly rather than concentrating it where the data was genuinely ambiguous, which is closer to a data-and-sampling-design question than an objective-function question. Cross-entropy has no way to tell a token that's uncertain because the data forked here from a token that's uncertain for no interesting reason.
Architecture
The next layer is network: designing an architecture structurally disposed to notice novelty, independent of what loss it's trained with. There has been exploration of this in RL literature. Never Give Up (Badia et al., 2020)2 gives an agent an actual episodic memory module (a running store of states it's visited recently) and computes a novelty bonus from nearest-neighbor distance in that memory. This was built for a specific, hard failure mode: sparse-reward games like Montezuma's Revenge and Pitfall, where an agent doing undirected random exploration essentially never stumbles onto a reward signal at all, so there is nothing for ordinary RL to reinforce. The episodic memory and novelty bonus exist to give the agent something to chase in the absence of any reward signal to follow. Random Network Distillation (Burda et al., 2018)3 adds a second, randomly initialized and permanently frozen network, train a third network to predict its outputs, and use the prediction error as a novelty signal. Both of these are hybrids in practice; they add a new piece of architecture and a new piece of loss at the same time; indicating that this layer and the optimizer layer are hard to cleanly separate. Also,while architecturally-grounded detectors of this form are good at noticing that something is unfamiliar, they might not necessarily translate to useful novelty.
Optimizer
This is the layer with a mechanism that explicitly, mathematically says "keep exploring" as part of what's being optimized. Standard RL practice since at least the original A3C paper4 adds an entropy bonus directly into the objective (reward minus β times the policy's own entropy) which punishes the policy for collapsing onto one action too early. Soft Actor-Critic16 turns this into an algorithm's central objective rather than an add-on term. This keeps the policy from collapsing onto one action, but it is not the same as rewarding novelty directly — a state visited a thousand times can still have high action-entropy. Count-based exploration bonuses17, curiosity-driven prediction-error rewards18, and information-gain objectives19 put actual state-novelty into the reward itself, rather than just the spread of the action distribution.
There's a second thing that happens at the optimizer layer that makes deployed chat models feel narrower than their base checkpoints21. RLHF-style fine-tuning is typically regularized toward a reference policy with the opposite kind of KL divergence from pretraining — KL(policy ‖ reference) instead of KL(reference ‖ policy) — and that direction is mode-seeking rather than mass-covering20. The model is free to abandon whole regions of its own base distribution as long as it doesn't wander somewhere the reward model dislikes. Thus the optimizer can influence a model whose data-layer diversity was already there and is now being actively narrowed.
Harness
Everything above happens during training. The harness, by contrast, sits around a trained model at the moment of inference. This is the layer with the lowest cost of checking a real outcome, without involving expensive gradient updates. The clearest existing version of this is retrieval15: handing the model documents or tool output at inference time so a claim can be checked against something outside its own weights, cheaply and without touching a single gradient. Reflexion7 and Voyager9 are the clearest existing curiosity attempts at this layer — revising behavior from an observed outcome without a weight update — and ExpeL22 extends the same idea across many tasks instead of one, extracting natural-language insights from both successes and failures. On the flip side, a harness that forces divergence without a real, grounded gate on the result is just a more expensive way to get what an entropy bonus already gives you: wider spread without any information about which part of the spread is worth anything. Co-Scientist23 is another recent example: many agents generate and debate hypotheses, but instead of using just the divergence, a tournament-style ranking against evidence is the actual grounded gate deciding which ones matter.
Where does curiosity live
Biology has its own answer to what should count as a good place to explore, worked out by evolution. One of the prominent results is dopamine. Schultz, Dayan, and Montague5 found that dopamine neurons do not fire in proportion to reward, they fire in proportion to reward prediction error: more than expected, a burst; exactly as expected, nothing; less than expected, a dip below baseline. Kidd, Piantadosi, and Aslin6 ran an experiment in 2012 around the learning habits of infants, and called it the Goldilocks effect = infants don't chase maximum surprise, they chase a specific, narrow, moving band of surprise: not too predictable, not too chaotic, just at the edge of what they can currently make sense of. There's no reward model, no loss function, no dataset of good things to look at. Whatever is producing that curve is either built into the wiring before birth, or it assembles itself extremely fast from a few months of raw sensory experience, or both.
A four-month-old has an innate disposition (lets call it architecture) wired directly into a fast, embodied feedback loop with the physical world (harness). The loop is what turns looking into learning, fast enough that the next look is already informed by the last one.
Nothing we have currently possesses all elements of that coupling. We have architectural novelty detectors that never see an outcome. We have optimizers that see outcomes but only in slow, offline, batched updates. We have harnesses that are close to real outcomes but usually do not influence weight updates. One way to bring this all together would be an update rule connecting harness-observed outcomes back into the model's behavior. A design for this has at least three properties.
For example, Reflexion7 has an agent write down why an attempt failed and hands that reflection to the next attempt, no weight update involved. Algorithm Distillation8 gets a model trained to improve its own policy over a context window in-context, on a timescale of tokens rather than training runs. Voyager9's skill library grows the moment code passes or fails, and that library shapes every attempt after. Reflexion and Voyager get fast and grounded. Neither gets fine-grained, as their unit of update is the attempt, not the token. Entropy-branching methods such as RND3, NGU2 are only grounded in their prior state.
Measuring curiosity
A simple setup for baking in curiosity inside an agentic system can be: Start with SFT on a curated set of open-ended, exploratory text traces, teaching a model to imitate the surface behavior of curiosity directly. Then run RL on top, using a judge or a rubric that scores something like curiosity or divergent thinking as part of the reward. Track a curiosity-marker metric across RL training steps, hoping to see it improve in an emergent fashion, similar to the way chain-of-thought length climbed over the course of DeepSeek-R110's RL training.
How will we measure this said ‘curiosity’ which leads to an ‘inspiration’ towards ‘novelty’? Checking the value of said novelty requires a verifier, and for open-ended tasks there usually is not one. Perhaps one way is to learn from attempts to measure this in humans. IQ testing started almost entirely convergent, one correct answer that is easy to grade and hillclimb against. Guilford11's structure-of-intellect model argued in the 1950s that this missed half of what intelligence does, and proposed divergent thinking, generating many varied valid responses to an open prompt, as a distinct factor.
Another approach is prospective grounding: score a generated idea not by a judge's opinion at generation time, but against what actually happened later. Some recent research-idea benchmarks12 specifically use papers published after a model's training cutoff, so a generated research idea can be checked against real work nobody could have leaked into training. On the flip side, that is slow and only works retrospectively. One caveat here is that a judge rewarding curiosity markers can be satisfied by a model that learns to ask more questions and hedge more visibly without actually exploring anything useful.
The third is to not score the object holistically, instead score the effect of a specific decision inside it. This is called ‘fork advantage’ - by holding the prefix fixed, forcing one specific token, and rolling out k times to see what one decision actually causes. GRPO13 already does a version of this once at the very start of generation. Fork advantage is what happens if you run that same move at every candidate position instead of only the root - which is why the cost multiplies by the number of positions. It can say a specific token mattered and in which direction.
Maybe it's fair to say that curiosity exists in bits and pieces across multiple components of an agentic system - architecture, optimizers, harnesses and the data. It remains to be seen how an end-to-end pipeline that creates an open-ended curious model will shape out. This in turn will truly enable current research agents to have real autonomy - from conceptualization, validation to E2E iteration until success.
References
1. Nagarajan, V., Wu, C.H., Ding, C., Raghunathan, A. (2025). Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction. ICML. arXiv:2504.15266.
2. Badia, A.P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., et al. (2020). Never Give Up: Learning Directed Exploration Strategies. ICLR. arXiv:2002.06038.
3. Burda, Y., Edwards, H., Storkey, A., Klimov, O. (2018). Exploration by Random Network Distillation. arXiv:1810.12894.
4. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. ICML. arXiv:1602.01783.
5. Schultz, W., Dayan, P., Montague, P.R. (1997). A Neural Substrate of Prediction and Reward. Science, 275(5306), 1593-1599.
6. Kidd, C., Piantadosi, S.T., Aslin, R.N. (2012). The Goldilocks Effect: Human Infants Allocate Attention to Visual Sequences That Are Neither Too Simple Nor Too Complex. PLOS ONE, 7(5), e36399.
7. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS. arXiv:2303.11366.
8. Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., et al. (2022). In-context Reinforcement Learning with Algorithm Distillation. arXiv:2210.14215.
9. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291.
10. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
11. Guilford, J.P. (1950). Creativity. American Psychologist, 5, 444-454.
12. Guo, S., Shariatmadari, A.H., Xiong, G., Huang, A., Xie, E., Bekiranov, S., Zhang, A. (2024). IdeaBench: Benchmarking Large Language Models for Research Idea Generation. arXiv:2411.02429.
13. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
14. Lu, C., Lu, C., Lange, R.T., Foerster, J., Clune, J., Ha, D. (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv:2408.06292.
15. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. arXiv:2005.11401.
16. Haarnoja, T., Zhou, A., Abbeel, P., Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML.
17. Bellemare, M.G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., Munos, R. (2016). Unifying Count-Based Exploration and Intrinsic Motivation. NeurIPS. arXiv:1606.01868.
18. Pathak, D., Agrawal, P., Efros, A.A., Darrell, T. (2017). Curiosity-Driven Exploration by Self-Supervised Prediction. ICML.
19. Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., Abbeel, P. (2016). VIME: Variational Information Maximizing Exploration. NeurIPS. arXiv:1605.09674.
20. Minka, T. (2005). Divergence Measures and Message Passing. Microsoft Research Technical Report MSR-TR-2005-173.
21. Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., Raileanu, R. (2024). Understanding the Effects of RLHF on LLM Generalisation and Diversity. ICLR. arXiv:2310.06452.
22. Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., Huang, G. (2024). ExpeL: LLM Agents Are Experiential Learners. AAAI. arXiv:2308.10144.
23. Gottweis, J., Weng, W.-H., et al. (2026). Accelerating scientific discovery with Co-Scientist. Nature. arXiv:2502.18864.