Based on this research, I have created a free web app for compressing GPT prompts at gptrim.com. gptrim is also a Python library. Contributions are welcome.

Readability and information

The other day I was trying to feed a large text to ChatGPT, and I grew frustrated with the limits of the chat message size. After some brainstorming with Bing, I had an insight. 

Not all tokens in a text convey information. Some are there to make it easier for humans to parse text. Consider spaces: they are mostly used to tokenize words for the human visual cortex. ButifIwrotewithoutspacesyouwouldprobablystillunderstandme. 

There are very few cases where a missing space would cause ambiguity. The nicest one ChatGPT could come up with was: 

"Iamnowhere" could be interpreted as "I am nowhere" or "I am now here," which have opposite meanings.

Even in such cases, the ambiguity would probably be solved by context.

What about punctuation? Again, I suspect there are very few cases where removing punctuation would create an ambiguity that cannot be solved by context. ChatGPT:

"Let's eat grandma" vs. "Let's eat, grandma." Without the comma, it could be misinterpreted as a request to eat the grandmother.

Can we go further? We could apply word stemming. This would significantly reduce our character count. How much information would we sacrifice? Try to read this snippet (full text below). 

a machin learn research write me in respons to yesterday ’ s post , say : i still think gpt - 2 is a brute - forc statist pattern matcher which blend up the internet and give you back a slight unappet slurri of it when ask . i resist the urg to answer “ yeah , well , your mom is a brute - forc statist pattern matcher which blend up the internet and give you back a slight unappet slurri of it when ask .” but i think it would have been true. (source)

I find that "caveman Scott" is still very much readable, especially if I know the context. 

Compressing prompts

Consider this snippet from Scott:

“But don’t humans also have genuinely original ideas?” Come on, read a fantasy book. It’s either a Tolkien clone, or it’s A Song Of Ice And Fire. Tolkien was a professor of Anglo-Saxon language and culture; no secret where he got his inspiration. A Song Of Ice And Fire is just War Of The Roses with dragons. Lannister and Stark are just Lancaster and York, the map of Westeros is just Britain (minus Scotland) with an upside down-Ireland stuck to the bottom of it – wake up, sheeple! Dullards blend Tolkien into a slurry and shape it into another Tolkien-clone. Tolkien-level artistic geniuses blend human experience, history, and the artistic corpus into a slurry and form it into an entirely new genre. Again, the difference is how finely you blend and what spices you add to the slurry.

Let's remove spaces and punctuation and apply Snowball stemming. As an additional (and perhaps excessive) optimization, let's also remove stopwords. 

We get something that looks like this:

donthumanalsogenuinoriginidea?”Come,readfantasibook.eitherTolkienclone,SongIceFire.TolkienprofessorAnglo-Saxonlanguagcultur;nosecretgotinspir.SongIceFireWarRosedragon.LannistStarkLancastYork,mapWesteroBritain(minusScotland)upsiddown-irelandstuckbottom–wake,sheepl!DullardblendTolkienslurrishapeanothtolkien-clon.tolkien-levelartistgeniusblendhumanexperi,histori,artistcorpusslurriformentirnewgenr.,differfineblendspiceaddslurri.

Let's feed it into GPT-4 with the following prompt:

This is an instance of compressed text. Rewrite it so that it has perfect grammar and is understandable by a human. Try to interpret it as faithfully as possible. Do not paraphrase or add anything to the text.

We get back this (I have bolded two sentences where I felt some meaning was lost):

Do humans also have a genuine origin for ideas? Come and read fantasy books, either by Tolkien or those in the Song of Ice and Fire series. Tolkien was a professor of Anglo-Saxon language and culture, so it's no secret where he got his inspiration from. The Song of Ice and Fire series is based on the War of the Roses, with dragons added. The Lannisters and the Starks are similar to the Lancasters and the Yorks, and the map of Westeros resembles Britain (minus Scotland) upside down, with Ireland stuck to the bottom – wake up, people! A dullard might blend Tolkien's ideas into a slurry and shape it into another Tolkien clone. However, a Tolkien-level artist and genius would blend human experience, history, and an artistic corpus into a slurry to create an entirely new genre. It is the difference between a fine blend of spices and a simple slurry.

Overall, I think it's a pretty good rendition. But much did we save? The original text has 187 tokens and 790 characters. The compressed text has 160 tokens and 428 characters. That's a 14% saving on tokens and a 46% saving on character count. 

Overall, it seems this approach might be more useful for fighting ChatGPT and Bing's message character limit than reducing tokens. But even a 14% saving is significant if you are using GPT in production. 

Conclusion

A few weeks ago Twitter discovered Shoggoth tongue, a technique for compressing GPT prompts by... asking GPT to do it. Shoggoth tongue may save a lot of tokens, but you also have to spend tokens to compress the text, so it only makes sense for crafting reusable prompts. I don't believe it applies to the majority of use cases, i.e. people solving mundane problems with ChatGPT. 

Applying the standard NLP preprocessing routine to GPT prompts is a quick and free way to reduce character count and save some tokens while seemingly preserving most information. 

I haven't had time to experiment extensively with prompt compression. I also did not try to quantify these results. I look forward to seeing new ideas in this space.