(Old notes from 2016 that I stumbled upon).

  • Train on everything, finetune if needed. Big enough and tuning seems pointless too. It’s not like humans need much tuning for everything. Why do we train and then throw away the network each time to start anew?
  • Why have separate modalities? That’s really dumb. Info is info, just give it all at once.
  • Cloze deletion should be network’s default task
  • Why bother cleaning data so much just dump more and more of it like those pots of neverending soup
  • I wonder if all this will finally teach us complex systems
  • If it picks up language, we could just try talking to it
  • I bet people will say it’s ”not real intelligence”. whatever bro, it’s more coherent than you
  • All these architectures don’t seem to matter that much
  • If it ever learns reasoning I feel like we’re screwed
  • normalization seems really important but batch norm is weird
  • statistical learning theory doesn’t seem helpful
  • why can a network be compressed so much?
  • is parity the hardest function to learn
New Comment
5 comments, sorted by Click to highlight new comments since: Today at 4:46 PM

Heh. Prescient.

I've added an excerpt from this to generative.ink/prophecies.

I still wonder about the parity prediction these days. I feel like there's something there

statistical learning theory doesn’t seem helpful

Could you explain this?

that the functional analysis is mildly helpful for understanding the problem, but the focus of the field doesn't seem to be on anything helpful. VC dimension is the usual thing to poke fun at, but a lot of the work on regularization is also meh