I found your idea fascinating. You have a good company too. Percy Liang's group just published a paper along this line of thought and showed transformer's effectiveness in learning "ML trainers":
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
I found your idea fascinating. You have a good company too. Percy Liang's group just published a paper along this line of thought and showed transformer's effectiveness in learning "ML trainers":
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
https://arxiv.org/abs/2208.01066