Do they all use the same tokenizer? (I mean, Mistral vs GPT) | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		raverbashing on Dec 11, 2023 \| parent \| context \| favorite \| on: Mistral: Our first AI endpoints are available in e... Do they all use the same tokenizer? (I mean, Mistral vs GPT)

superkuh on Dec 11, 2023 [–]

No. Mistral uses sentencepiece and the GPT use tiktoken.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact