Hi there - I’ve just released Tikentoken, an Elixir toy library and CLI tool for experimenting with AI tokenization using Ollama models.

Inspired by GitHub - openai/tiktoken: tiktoken is a fast BPE tokeniser for use with OpenAI's models., and because I wanted to play with Ollama and other open source models. It can tokenize text, compute embeddings, and even chat - with a convenient fallback on a basic mock, when Ollama is offline or not installed/available.

The documentation is much more extensive than the code itself, as I mainly published it to keep my dev notes organized. On

llm tokenizer #ollama