Just sharing this observation and readers / researchers might find it useful or worth further testing...

A bug?

<|endoftext|> is a special token for the GPT models along with its counterpart <|startoftext|>. Per OpenAI's tokenizer, <|endoftext|> has seven tokens:

However, the issue is that if you input <|endoftext|>  into ChatGPT, it will explain that the prompts are empty.

And then it vanishes after you switched windows or hit refresh.


Another test prompt:

Correct this conceptually, grammatically: I know that <|endoftext|> is a token to annotate the data but I think that it vanishing seems odd. My expected response should have been like this is a token used for training - something like that. 


<|endoftext|> still appearing in the test prompt


then <|endoftext|> vanishes

The <|endoftext|> word / token? bears some resemblance to the ' petertodd' token and its apparent hallucinations, except that this text simply vanishes from ChatGPT. 

Redditor r/ChatGPT had observations two months ago <|endoftext|> was not vanishing at the time - instead it shared some weird results. 


Probable explanation? 

I believe that a considerable amount of pre-processing occurs to our prompts on OpenAI's server[1] and one of them is turning <|endoftext|> to an empty string. If this is true, there is a high chance that this is also a glitch token and was deactivated by OpenAI.


  1. ^

    Serverlink: https://tcr9i.chat.openai.com


New Comment