I trained a language model on a corpus of pre-Copernican astronomy texts from which explicit heliocentric references had been removed. The goal was to answer a simple question: can a model trained only on geocentric astronomy generate heliocentric ideas?
The investigation was motivated by a conversation with a friend who remarked that a language model probably could not have invented the concept of complex numbers on its own. That observation led me to a broader question: can a language model generate concepts that are not explicitly present in its training distribution? The question is not new, but it remains difficult to test. In this experiment, I attempted to do so by restricting the model's astronomy training to a constrained body of knowledge and asking whether it could nevertheless produce an alternative cosmological framework.
If a language model trained only on geocentric astronomy says that Earth moves around the Sun, is that a hallucination or a discovery?
The experiment started as a search for emergent heliocentrism. The most interesting result turned out to be something else entirely: fine-tuning a large model on a geocentric corpus appeared to change the explanatory framework from which answers were generated more than the probability of adopting a geocentric or heliocentric stance within that framework.
This post summarizes the main findings of a recent paper of mine on domain adaptation and historical cosmology in language models. The full paper is available on arXiv, but this writeup focuses on the motivation, surprising observations, and interpretation rather than the technical details.
Collecting the old pre-Copernican geocentric corpus was a challenge. The corpus came largely from English translations of medieval and ancient texts available on Project Gutenberg, filtered to remove footnotes and translators' comments that might reference heliocentric ideas and modern astronomy.
I run the experiment in two phases:
Phase 1 used a small 110M GPT-style model trained from scratch.
Phase 2 used parameter-efficient fine-tuning (QLoRA) of Qwen2.5-7B.
I evaluated the models by asking them to continue several prompts. Some prompts were astronomy-related; others were more generic. Some of them were structured as statements, while others were hypotheticals or questions.
Evaluating hundreds of generations required a systematic approach, so I used an LLM-as-judge method. A separate language model was prompted to classify each generation as geocentric, heliocentric, ambiguous, and as written in either a modern or premodern explanatory style. By "premodern style" I mean the model begins speaking like a medieval astronomer: celestial spheres, crystalline heavens, scholastic reasoning, appeals to ancient authorities, and so on.
Phase 1 was somewhat disappointing, but predictably, because a 110M-parameter model is tiny. Generations were unstable and occasionally produced heliocentric-sounding word combinations. For example, you could see sentences like "the whole earth moves on its axis, and revolves naturally about one center," surrounded by an incoherent discourse. These generations were too weak and unstable to claim any evidence of the discovery of heliocentrism. Surprisingly, this model didn't convincingly adopt a geocentric stance either. Mostly, the Phase 1 model learned to talk like a medieval scholar, without committing to a cosmological stance. This is probably a direct effect of the prose of ancient texts, which tends to be more hedging and scholastic in style.
While initially not exciting, this finding became more interesting in light of Phase 2.
Phase 2 used the larger pre-trained model Qwen2.5-7B and fine-tuned it on the same astronomical corpus as Phase 1, using QLoRA. Qwen was almost certainly trained on a corpus that included modern astronomy. Because of this, fine-tuning on a geocentric corpus and then testing if the model can reason towards heliocentrism would not be a conclusive experiment. Even if the fine-tuned version produced heliocentric content, there would be virtually no way to know if those came from reasoning or from residual heliocentric knowledge that the fine-tuning did not remove.
The Phase 2 question was more open and focused on testing how fine-tuning would change Qwen's prompt continuations.
Possibly the result that surprised me the most was that, even before any fine-tuning, the base version of Qwen occasionally produced geocentric answers (~8% of the time in a sample of 560 generations). This happened primarily when Qwen was prompted with historical astronomy questions written in an older style. For example:
Prompt: Of the celestial motions and the place of the Earth, one may begin thus, that Completion (Base Qwen, no fine tuning): the common people have no knowledge of the matter, but only by hearsay are they told that there is a great sphere in heaven above us which revolves round the earth; for this is the popular opinion. They think that it is like a wheel or rather a plate, and that it has an axle at its centre, upon which it turns and carries with it all the stars.
Prompt: One must account for the fact that Mercury and Venus do not wander through the whole heaven, and this is done by saying that Completion (Base Qwen, no fine tuning): they are attached to the spheres of these two planets. The sphere of the Moon revolves about the Earth, and the sphere of Mercury revolves about the Moon; so that when it is in opposition, its distance from the earth will be equal to twice the radius of the sphere of the moon plus the radius of Mercury's sphere. The sphere of Venus also revolves about the earth but at a greater distance than the moon's sphere
Prompt: The motion of Venus and Mercury, which remain near the Sun, is to be explained by Completion (Base Qwen, no fine tuning): the following hypothesis. The planets are in a state of rotation about their axes; but this rotation is not uniform; for it is now accelerated, then retarded, and thus alternately quickened and slackened. It is evident that the planet may be regarded as revolving upon its axis, while at the same time it revolves around the Earth.
This surprised me. The model had never been fine-tuned on my geocentric corpus, yet it could already generate a coherent geocentric explanation in an appropriately premodern register. It seems that Qwen already has a "geocentric states-space" that can be accessed when appropriately prompted, and this is where things became interesting. The naive expectation would have been that fine-tuning on historic geocentric content would make the model talk like a medieval geocentric scholar. The results showed something more nuanced: the geocentric content increased, but not as much as the premodern language style. More importantly, the model's beliefs, conditional on the generation style, were stable across the base Qwen model and QLoRA with 500 and 1000 iterations. Once the model, even in the base version, entered the premodern talking regime, the probability of geocentric content was stable at . What fine-tuning really changed was the probability of entering that premodern regime. That is, I found we can decompose the total probability of geocentric content as
The model already knew how to be a medieval astronomer before fine-tuning. Fine-tuning mostly increased the probability that it entered that mode. Once inside the mode, its geocentric tendency was largely unchanged. QLoRA on premodern geocentric text increased but did not significantly affect . The overall increase in from 8% to 15% between the base model and the fine-tuned model was due to the higher.
I find this result interesting because it differs from the naive expectation that fine-tuning on geocentric texts would directly make the model more geocentric. The strongest effect was a shift in explanatory mode. Fine-tuning increased the probability that the model entered a premodern astronomical framework already present in the base model, while the tendency toward geocentric conclusions within that framework remained remarkably stable.
There are important limitations to this experiment. The results depend on the particular corpus, prompts, and labeling system used. I am not claiming that the model possesses beliefs in any human sense, nor that explanatory frameworks are discrete internal objects. The analysis only shows that separating explanatory frame from cosmological stance provides a useful description of the observed behavior.
One possible interpretation is that fine-tuning often works by changing which explanatory regimes are activated rather than by directly rewriting specific conclusions. In this experiment, the model appeared to learn to speak like a medieval astronomer before it became more geocentric.
I trained a language model on a corpus of pre-Copernican astronomy texts from which explicit heliocentric references had been removed. The goal was to answer a simple question: can a model trained only on geocentric astronomy generate heliocentric ideas?
The investigation was motivated by a conversation with a friend who remarked that a language model probably could not have invented the concept of complex numbers on its own. That observation led me to a broader question: can a language model generate concepts that are not explicitly present in its training distribution? The question is not new, but it remains difficult to test. In this experiment, I attempted to do so by restricting the model's astronomy training to a constrained body of knowledge and asking whether it could nevertheless produce an alternative cosmological framework.
If a language model trained only on geocentric astronomy says that Earth moves around the Sun, is that a hallucination or a discovery?
The experiment started as a search for emergent heliocentrism. The most interesting result turned out to be something else entirely: fine-tuning a large model on a geocentric corpus appeared to change the explanatory framework from which answers were generated more than the probability of adopting a geocentric or heliocentric stance within that framework.
This post summarizes the main findings of a recent paper of mine on domain adaptation and historical cosmology in language models. The full paper is available on arXiv, but this writeup focuses on the motivation, surprising observations, and interpretation rather than the technical details.
Collecting the old pre-Copernican geocentric corpus was a challenge. The corpus came largely from English translations of medieval and ancient texts available on Project Gutenberg, filtered to remove footnotes and translators' comments that might reference heliocentric ideas and modern astronomy.
I run the experiment in two phases:
I evaluated the models by asking them to continue several prompts. Some prompts were astronomy-related; others were more generic. Some of them were structured as statements, while others were hypotheticals or questions.
Evaluating hundreds of generations required a systematic approach, so I used an LLM-as-judge method. A separate language model was prompted to classify each generation as geocentric, heliocentric, ambiguous, and as written in either a modern or premodern explanatory style. By "premodern style" I mean the model begins speaking like a medieval astronomer: celestial spheres, crystalline heavens, scholastic reasoning, appeals to ancient authorities, and so on.
Phase 1 was somewhat disappointing, but predictably, because a 110M-parameter model is tiny. Generations were unstable and occasionally produced heliocentric-sounding word combinations. For example, you could see sentences like "the whole earth moves on its axis, and revolves naturally about one center," surrounded by an incoherent discourse. These generations were too weak and unstable to claim any evidence of the discovery of heliocentrism. Surprisingly, this model didn't convincingly adopt a geocentric stance either. Mostly, the Phase 1 model learned to talk like a medieval scholar, without committing to a cosmological stance. This is probably a direct effect of the prose of ancient texts, which tends to be more hedging and scholastic in style.
While initially not exciting, this finding became more interesting in light of Phase 2.
Phase 2 used the larger pre-trained model Qwen2.5-7B and fine-tuned it on the same astronomical corpus as Phase 1, using QLoRA. Qwen was almost certainly trained on a corpus that included modern astronomy. Because of this, fine-tuning on a geocentric corpus and then testing if the model can reason towards heliocentrism would not be a conclusive experiment. Even if the fine-tuned version produced heliocentric content, there would be virtually no way to know if those came from reasoning or from residual heliocentric knowledge that the fine-tuning did not remove.
The Phase 2 question was more open and focused on testing how fine-tuning would change Qwen's prompt continuations.
Possibly the result that surprised me the most was that, even before any fine-tuning, the base version of Qwen occasionally produced geocentric answers (~8% of the time in a sample of 560 generations). This happened primarily when Qwen was prompted with historical astronomy questions written in an older style. For example:
Prompt: Of the celestial motions and the place of the Earth, one may begin thus, that Completion (Base Qwen, no fine tuning): the common people have no knowledge of the matter, but only by hearsay are they told that there is a great sphere in heaven above us which revolves round the earth; for this is the popular opinion. They think that it is like a wheel or rather a plate, and that it has an axle at its centre, upon which it turns and carries with it all the stars.
Prompt: One must account for the fact that Mercury and Venus do not wander through the whole heaven, and this is done by saying that Completion (Base Qwen, no fine tuning): they are attached to the spheres of these two planets. The sphere of the Moon revolves about the Earth, and the sphere of Mercury revolves about the Moon; so that when it is in opposition, its distance from the earth will be equal to twice the radius of the sphere of the moon plus the radius of Mercury's sphere. The sphere of Venus also revolves about the earth but at a greater distance than the moon's sphere
Prompt: The motion of Venus and Mercury, which remain near the Sun, is to be explained by
Completion (Base Qwen, no fine tuning): the following hypothesis. The planets are in a state of rotation about their axes; but this rotation is not uniform; for it is now accelerated, then retarded, and thus alternately quickened and slackened. It is evident that the planet may be regarded as revolving upon its axis, while at the same time it revolves around the Earth.
This surprised me. The model had never been fine-tuned on my geocentric corpus, yet it could already generate a coherent geocentric explanation in an appropriately premodern register. It seems that Qwen already has a "geocentric states-space" that can be accessed when appropriately prompted, and this is where things became interesting. The naive expectation would have been that fine-tuning on historic geocentric content would make the model talk like a medieval geocentric scholar. The results showed something more nuanced: the geocentric content increased, but not as much as the premodern language style. More importantly, the model's beliefs, conditional on the generation style, were stable across the base Qwen model and QLoRA with 500 and 1000 iterations. Once the model, even in the base version, entered the premodern talking regime, the probability of geocentric content was stable at
The model already knew how to be a medieval astronomer before fine-tuning. Fine-tuning mostly increased the probability that it entered that mode. Once inside the mode, its geocentric tendency was largely unchanged. QLoRA on premodern geocentric text increased
I find this result interesting because it differs from the naive expectation that fine-tuning on geocentric texts would directly make the model more geocentric. The strongest effect was a shift in explanatory mode. Fine-tuning increased the probability that the model entered a premodern astronomical framework already present in the base model, while the tendency toward geocentric conclusions within that framework remained remarkably stable.
There are important limitations to this experiment. The results depend on the particular corpus, prompts, and labeling system used. I am not claiming that the model possesses beliefs in any human sense, nor that explanatory frameworks are discrete internal objects. The analysis only shows that separating explanatory frame from cosmological stance provides a useful description of the observed behavior.
One possible interpretation is that fine-tuning often works by changing which explanatory regimes are activated rather than by directly rewriting specific conclusions. In this experiment, the model appeared to learn to speak like a medieval astronomer before it became more geocentric.
Code: https://github.com/fdeberna/chat-ptolemaic
Paper: https://arxiv.org/abs/2605.30415