Built a simple version of tokenizer for LLMs in Go to understand / WIP

Dmytro Krasun

Built a simple version of tokenizer for LLMs in Go to understand how they work. Also, played with the port of of the OpenAI "tiktoken" library in Go.

Keyboard Shortcuts