Fair points!
It runs 93x faster on than Zephyr 7B on a 4060 Ti (16GB of memory) that runs them both fully within VRAM. No memory bandwidth or capacity limitations for either model. It genuinely is a pretty fair comparison. I do get into it this in the paper. Unfortunately, I'm unable to edit my original post due to negative karma.
I do understand the vibes you're talking about on the patent vibe side of things. It's pretty damned presumptuous - why would grad students want to use this for their degrees? Unfortunately, putting it out there without that kind of disclaimer or specific clarity is also not really something I want to do,... (read more)
I'm at a bit of a loss as well, given the lack of comments. I haven't even been able to directly post a direct link to the root of the github repo with the inference pipeline and a script to download the model. I'm guessing the link combined with low account karma resulted in the spam filters being triggered. I'm awaiting moderator appeal.
The downvotes have followed an interesting pattern, too -- they were down to -12 some time around 9PM PST last night, and by the time I woke up due to early morning insomnia at around 1:30 PST had recovered to -4.
I want to share what I've been working on the last few weeks: O(1) inference across whole tasks through direct vector transformation. A few facts upfront to give you an idea of how it goes:
1. Implemented as part of a PoC of what I call the Promptable General Classifier (a classifier which can be prompted for general tasks, including (some, limited) reasoning tasks, and has inference-time hot swappable vocabulary/classes), and the 1.09B implementation:
1. Runs 93x faster than Zephyr 7B (and this is being generous to Zephyr, as I had to add post-processing to extract labels from malformed LLM output, and I didn't count the time necessary to complete this post processing... (read 404 more words →)
I think it's appropriate for me to add some high visibility clarity in light of the commentary on why people downvoted this. I expected and was prepared for skepticism (naturally), but the lack of technical or theoretical critique surprised me. Maybe I should have stuck with my original title: "How Much Attention Do You Need, Really? Early Experiment in O(1) Reasoning In Latent Space"
I definitely should have surfaced more detail upfront so people had some idea of what they were looking at. I'll take a stab at that now:
1. Architecture: this is a novel architecture which I call a "DSRU" (Direct Semantic Reasoning Unit) that falls under the "vec2vec" category.
That means no... (read 819 more words →)