A recent paper has peeked my interest coming from UpstageAI, a Korean AI Lab.
This technique applied in the paper has been used to get the highest scores on the open LLM leaderboard on HuggingFace.
Has anyone read the paper in more detail? I'm struggling to understand the following paragraph which seems to be the core of the paper itself.
"One naive way to up-scale the base LLM would be to repeat its layers once more, i.e., from 32 to 64 layers. This has the benefit that from layers 1 to 32 and from layers 33 to 64, there are no heterogeneity as those layers are taken directly from the base LLM. In other words,... (read 185 more words →)
I think there has been a lot of research in the past in this space. The first thing that popped into my mind was https://huggingface.co/docs/transformers/model_doc/rag
Currently, there are some approaches using langchain to persist the history of a conversation into an embeddings database, and retrieve the relevant parts performing a similar query / task.
OpenAI has hired a lot of software engineers to code simple tasks, maybe these quines were a part of the fine tuning set?
Isn't ELO a reference metric that changes with time? I would assume that 2800 ELO in the 90s is a different level to today's 2800. Can we still make the same conclusions with that in mind?