LLM providers charge you per token, but their tokens are not always comparable

"LLM providers charge you per token, but their tokens are not always comparable. So if you are putting Python code through GPT-4 and Claude 3, it would cost you 25% more tokens to do so with Claude, due to difference in their tokenisers (note: this is different to cost per token, it just means you will have more tokens to pay for)

Some observations:

• OpenAI's GPT-4 & 3.5 tokeniser is the most efficient for English and Python

• Gemini absolutely demolishes the competition in the three languages I tested: French (-11%), Chinese (-43%) and Hebrew (-54%)

• If your use cases is non-English, really worth looking at Gemini models - the difference in cost will likely be very noticeable

• Llama 2 ranked at the bottom of all of my tests

• Mistral was kind of disappointing on French (+16% worse than GPT), the reason why I picked French was that I assumed they'd do better

Methodology notes:

• The study will be limited, I only compared 7 individual bits of text/code - so results in practice would vary

• I have used this tokeniser playground (https://lnkd.in/ewud-8vf) for GPT, Mistral and Llama. I found it to be inaccurate (or old?) for Claude 3 and they didn't have Gemini, so I did these separately

• Tokens are only part of the puzzle, more efficient tokenisation won't necessarily mean better performance or overall lower cost"

Author

AiUTOMATING PEOPLE, ABN ASIA was founded by people with deep roots in academia, with work experience in the US, Holland, Hungary, Japan, South Korea, Singapore, and Vietnam. ABN Asia is where academia and technology meet opportunity. With our cutting-edge solutions and competent software development services, we're helping businesses level up and take on the global scene. Our commitment: Faster. Better. More reliable. In most cases: Cheaper as well.

Feel free to reach out to us whenever you require IT services, digital consulting, off-the-shelf software solutions, or if you'd like to send us requests for proposals (RFPs). You can contact us at [email protected]. We're ready to assist you with all your technology needs.

© ABN ASIA