This is a special post for quick takes by leogao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
suppose i could create a human-understandable python program that is equivalent in performance to an N parameter language model. how big would you expect N to have to be for it to be a very interesting object to study? 1M? 10M 100M? 1B?
lol i just had a little autoresearch thingy run on this for 1 hour and this is what I get
[...]
(its a compressed python program doing online tracking of pretty simple frequencies. it performs as well as a ~10k parameters LLM?)
3leogao
i think length is clearly the wrong thing to optimize for. i think it makes a lot more sense to use logprob on some language model to constrain complexity.