I think that prompting is definitely important. I've found that GPT as it is now can mimic any given author's style with great accuracy as long as it's given that author's text inside of the prompt. For example, "write a short story in the style of Nabokov" gives you a bland short story, while prompting with his verbatim text produces a pretty faithful continuation.
Why should we expect novel ML architectures to perform better than current methods like LASSO regression? If your task is disease risk prediction, this falls under "variance within the human distribution" which, as far as I know, should be explained mostly by additive effects of SNPs. Current missing heritability is more of an issue of insufficient data (like you say) and rare variants rather than an inefficiency of our models. Steve Hsu had a paper about why we should actually theoretically expect LASSO regression to be efficient here.
My impression is generally that genomics is one of the few areas in bio where fancy ML is ill-suited for the problem (specifically polygenic trait prediction). I'm curious if there's some premise I have that Tabula disagrees with that makes this venture promising.