Regarding GPT-4, I believe the estimate was that it has 1.8 trillion parameters, which if shared weights are used may not precisely correspond to connections or FLOPs.

For standard LLM architectures, forward pass FLOPs are (because of the multiply and accumulate for each matmul param). It could be that GPT-4 has some non-standard architecture where this is false, but I doubt it.

So, yeah we agree here, I was just noting that connection == FLOP (roughly).

What is your basis for this intuition? [...] what makes you think that the brain is likely to do the worse job of training its available weights, or that many synapses are "mostly untrained"?

The brain is purely local which makes training all the parameters efficiently much harder, my understanding is that in at least the vision focused part of the brain there is a bunch of use of randomly initialized filters, and I seem to recall some argument made somewhere (by Steven Byrnes?) that the effective number of parameters was much lower. Sorry I can't say more here.