This is a Java implementation of a GPT3/4 tokenizer, loosely ported from Tiktoken with the help of ChatGPT. ...that all 3.5-turbo models released after 0613 now have ...

This is a Java string tokenizer for natural language processing machine learning models. Specifically, it was written to output token sequences that are compatible with the sequences produced by the ...