My accusations, at least so far:

Interesting link on symbolic regression. I actually tried to get an AI to write me something similar a while back^[1] (not knowing that the concept was out there and foolishly not asking, though in retrospect it obviously would be). 

From your response to kave:

calculate a quantity then use that as a new variable going forward

In terms of the tree structure used in symbolic regression (including my own attempt), I would characterize this as wanting to preserve a subtree and letting the rest of the tree vary. 

Possible issues:

1. If the coding modifies the trees leaf-first, trees with different roots but common subtrees aren't treated as close to each other. This is an issue that my own version would likely have had even if actually implemented^[2]. However, I think PySR might at least partially address this issue (It uses genetic programming and the pictures in the associated paper seem to indicate that it is generating trees which at least sometimes preserve subtrees.) (Though the genetic programming approach is likely to make it hard to find the very simplest solutions in practice imo.^[3])
2. Even if you are treating trees with common subtrees as close to each other, if your evaluation of trees is only comparing final calculated values on the entire dataset, then it's hard to make the call  "I know this subtree is important even if I don't know the rest of the tree" because the results are not likely to be all that close unless you already have a reasonable guess for the rest of the tree. One partial (heh) answer might be to award part marks to solutions that work well for some of the data even if wildly off for other parts. Careful thinking might be required to do this in a way that doesn't backfire horribly, though. Hmm - or maybe you CAN do that in the existing paradigm by including if/then nodes in the tree? Say, a node that has three child nodes/subtrees, and chooses between two of them based on the value of the third? And then (in some genetic-programming-like approach perhaps) explore what happens if you copy those subtrees elsewhere, or existing subtrees into new if-then nodes?) (I can imagine the horrific unreadable mess already though...)

edited to add: it might be more appropriate to say that I had been planning on asking an AI to code something, but the initial prototype was sufficiently lame and gave me enough insight into the difficulties ahead I didn't continue. Claude chat link if anyone's interested.

edited to further add: hmm, what you are wanting ("new variable") is probably not just preserving a subtree, but for the mutation system to be able to copy that subtree to other parts of the tree (and the complexity calculator to not give to much penalty to that, I guess). Interestingly, it seems that PySR's backend at least (SymbolicRegression.jl) does have the capability to do this already, using a "form_random_connection!" mutation function that apparently allows the same subtree to appear as child of multiple parents, making a DAG instead of a tree. In general, I've been pretty impressed looking at SymbolicRegression.jl. Maybe other symbolic regression software is as feature-rich, but haven't checked.

1. ^{^}

   Apparently November 2024. Feels longer ago somehow.

2. ^{^}

   I hadn't actually gone beyond breadth-first search though.

3. ^{^}

   This is informed by (a tiny amount of) practical experience. After SarahNibs' comment suggested genetic programming would have worked on the "Arena of Data", I attempted genetic programming on it and on my initial attempt got ... a horrific unreadable mess. Maybe it wasn't "halfway decently regularized" but I updated my intuition to say: complicated ways to do things so greatly outnumber the simple ways that anything too reliant on randomness is not likely to find the simple way. 

And just now I thought, wait, wouldn't this sometimes round to 10, but no, an AI explained to apparently-stupid me again that since it's a 0.25 tax rate on integer goods, fractional gold pieces before rounding (where not a multiple of 0.1) can only be 0.25, which rounds down to 2 silver, or 0.75, which rounds up to 8 silver. Which makes it all the more surprising that I didn't notice this pattern.

Thanks aphyer, it was an interesting puzzle. I feel like it was particularly amenable to being worked out by hand relative to machine learning because of the determinism, rules easy to express in a spreadsheet, and simple subsets of the data (like the special cleric bracket) that could be built on.

This is resolved using python's round() function...my apologies to simon, who seems to have spent a while trying to figure out when it rounded up or down.

I don't recall that taking all that long really, I added one monster part type at a time (much like the main calculation but much easier since it's just binary and only for the special 5U+ tax bracket, so less interactions to worry about). 

Funny thing is, after seeing the calculation I still didn't understand why it rounded the way it does, until I asked an AI which explained that Python rounds to even numbers on ties (apparently called "banker's rounding" or "banker's rule"). The only source of non-integer values is U which provides a single factor of 2 in the number of silver pieces, resulting in all rounding being of numbers with 0.5 in the remainder of silver pieces and so applying this rule. The other monster parts provide 2 factors of 2 of each, not directly causing rounding but each one incrementing the tax by 1 silver (modulo 2 silver) and thus changing whether this rule rounds up or down (except 2nd and up dragon heads which provide 3 factors of 2) . 

Amusingly, I could have much more simply expressed this rounding rule as "if rounding is required, round to the nearest even number of silver pieces" but I wasn't thinking about it in terms of output, so missed this and expressed it much more complicatedly in terms of the input. Oops! 

    if taxed_goods['U'] >= 5:        tax_rate = min(tax_rate, 0.25)

Ah, makes sense that this would be its own special tax rate rather than the 50% bracket with a X2 discount (which amounts to the same thing in the end). That min though with the tax rate from other sources is another thing that was never triggered (and literally couldn't be triggered since 5U alone is enough to get to the 30% bracket and also prevents eligibilty for the cleric bracket). 

Thanks aphyer. Solution:

P. S. I used GPT-4.1 in Windsurf for the AI aspects. They're running a promotion where it costs 0 credits until, IIRC, April 21.

FWIW there is a theory that there is a cycle of language change, though it seems maybe there is not a lot of evidence for the isolating -> agglutinating step. IIRC the idea is something like that if you have a "simple" (isolating) language that uses helper words instead of morphology eventually those words can lose their independent meaning and get smushed together with the word they are modifying.

Also, when doing a study, please write down afterwards whether you used intention to treat or not. 

Example:  I encountered a study that says post meal glucose levels depend on order in which different parts of the meal were consumed. But the study doesn't say whether every participant consumed the entire meal, and if not, how that was handled when processing the data. Without knowing if everyone consumed everything, I don't know if the differences in blood glucose were caused by the change in order, or by some participants not consuming some of the more glucose-spiking meal components.

In that case, intention to treat (if used) makes the result of the study less interesting since it provides another effect that might "explain away" the headline effect.

Issues with the dutch book beyond the marginal value of money:

• It's not as clear as it should that the LLM IQ loss question is talking about a permanent loss (I may have read it as temporary when answering)
• Although the LLM IQ drop question does say "your IQ" there's an assumption that that sort of thing is a statistical average - and I think the way I use LLMs, for example, is much less likely to drop my IQ than the average person's usage.
• I think is that the LessWrong subscription question is implictly asking about the marginal value of LessWrong given the existence of other resources while the relative LessWrong/LLM value question is implicitly leaning more towards non-marginal value obtained, which might be very many times more

impact: these issues increase LLM/IQ and (Lesswrong/LLM relative to LessWrong/$), which cause errors in the same direction in the LLM/IQ/$/Lesswrong/LLM cycle, potentially by a very large multiplier. 

Marginal value due to the high IQ gain of 5 lowers $/IQ which increases IQ/$. This also acts in the same direction.

(That's my excuse anyway. I suspected the cycle when answering and was fairly confident, without actually checking, that I was going to be way off from a "consistent" value. I gave my excuse as a comment in the survey itself that I was being hasty, but on reflection I still endorse an "inconsistent" result here, modulo the fact that I likely misread at least one question).

Control theory I think often tends to assume that you are dealing with continuous variables. Which I think the relevant properties of AIs are likely (in practice) not - even if the underlying implementation uses continuous math RSI will make finite changes and even small changes could cause large differences in results.

Also, the dynamics here are likely to depend on capability thresholds which could cause trend extrapolation to be highly misleading.

Also, note that RSI could create a feedback loop which could enhance agency including towards nonaligned goals (agentic AI convergently wants to enhance its own agency).

Also beware that agency increases may cause increases in apparent capability because of Agency Overhang.

The AI system accepts all previous feedback, but it may or may not trust anticipated future feedback. In particular, it should be trained not to trust feedback it would get by manipulating humans (so that it doesn't see itself as having an incentive to manipulate humans to give specific sorts of feedback).

I will call this property of feedback "legitimacy". The AI has a notion of when feedback is legitimate, and it needs to work to keep feedback legitimate (by not manipulating the human).

Legitimacy is good - but if an AI that's supposed to be intent-aligned to the user would find that it has an "incentive" to purposefully manipulate the user in order to get particular feedback from the user, unless it pretends that it would ignore that feedback, it's already misaligned and that misalignment should be dealt with directly IMO - this feels to me like a band-aid over a much more serious problem.