cubefox

Wikitag Contributions

Comments

Sorted by

The atomless property and only contradictions taking a 0 value could both be consequences of the axioms in question. The Kolmogorov paper (translated from French by Jeffrey) has the details, but from skimming it I don't immediately understand how it works.

If understand correctly, possible probability 0 events are ruled out for Kolmogorov's atomless system of probability mentioned in footnote 7

cubefox20

If you want to understand why a model, any model, did something, you presumably want a verbal explanation of its reasoning, a chain of thought. E.g. why AlphaGo made its famous unexpected move 37. That's not just true for language models.

cubefox20

Actually the paper doesn't have any more on this topic than the paragraph above.

cubefox31

Yeah, I also guess that something in this direction is plausibly right.

perhaps nothingness actually contains a superposition of all logically possible states, models, and systems, with their probability weighted by the inverse of their complexity.

I think the relevant question here is why we should expect their probability to be weighted by the inverse of their complexity. Is there any abstract theoretical argument for this? In other words, we need to find an a priori justification for this type of Ockham's razor.

Here is one such attempt: Any possible world can be described as a long logical conjunction of "basic" facts. By the principle of indifference, assume any basic fact has the same a priori probability (perhaps even probability 0.5, equal to its own negation), and that they are a priori independent. Then longer conjunctions will have lower probability. But longer conjunctions also describe more complex possible worlds. So simpler possible worlds are more likely.

Though it's not clear whether this really works. Any conjunction completely describing a possible world would also need to include a statement "... and no other basic facts are true", which is itself a quantified statement, not a basic fact. Otherwise all conjunctive descriptions of possible worlds would be equally long.

cubefox20

Regular LLMs can use chain-of-thought reasoning. He is speaking about generating chains of thought for systems that don't use them. E.g. AlphaGo, or diffusion models, or even an LLM in cases where it didn't use CoT but produced the answer immediately.

As an example, you ask an LLM a question, and it answers it without using CoT. Then you ask it to explain how it arrived at its answer. It will generate something for you that looks like a chain of thought. But since it wasn't literally using it while producing its original answer, this is just an after-the-fact rationalization. It is questionable whether such a post-hoc "chain of thought" reflects anything the model was actually doing internally when it originally came up with the answer. It could be pure confabulation.

cubefox20

Clearly ANNs are able to represent propositional content, but I haven't seen anything that makes me think that's the natural unit of analysis.

Well, we (humans) categorize our epistemic state largely in propositional terms, e.g. in beliefs and suppositions. We even routinely communicate by uttering "statements" -- which express propositions. So propositions are natural to us, which is why they are important for ANN interpretability.

Answer by cubefox60

It seems they are already doing this with R1, in a secondary reinforcement learning step. From the paper:

2.3.4. Reinforcement Learning for all Scenarios

To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities. Specifically, we train the model using a combination of reward signals and diverse prompt distributions. For reasoning data, we adhere to the methodology outlined in DeepSeek-R1-Zero, which utilizes rule-based rewards to guide the learning process in math, code, and logical reasoning domains. For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios. We build upon the DeepSeek-V3 pipeline and adopt a similar distribution of preference pairs and training prompts. For helpfulness, we focus exclusively on the final summary, ensuring that the assessment emphasizes the utility and relevance of the response to the user while minimizing interference with the underlying reasoning process. For harmlessness, we evaluate the entire response of the model, including both the reasoning process and the summary, to identify and mitigate any potential risks, biases, or harmful content that may arise during the generation process. Ultimately, the integration of reward signals and diverse data distributions enables us to train a model that excels in reasoning while prioritizing helpfulness and harmlessness.

cubefox23

Because I don’t care about “humanity in general” nearly as much as I care about my society. Yes, sure, the descendants of the Amish and the Taliban will cover the earth. That’s not a future I strive for. I’d be willing to give up large chunks of the planet to an ASI to prevent that.

I don't know how you would prevent that. Absent an AI catastrophe, fertility will recover, in the sense that "we" (rationalists etc) will mostly be replaced with people of low IQ and impulse control, exactly those populations that have the highest fertility now. And "banishing aging and death" would not prevent them from having high fertility and dominating the future. Moloch is relentless. The problem is more serious than you think.

cubefox20

I'm almost certainly somewhat of an outlier, but I am very excited about having 3+ children. My ideal number is 5 (or maybe more if I become reasonably wealthy). My girlfriend is also on board.

It's quite a different question whether you would really pull through with this or whether either of you would change their preference and stop at a much lower number.

Load More