Are AI tokens a technical law or a corporate paywall?

Open your favourite commercial AI interface, type a little too much, and you will hit the wall: Limit reached. Please upgrade your plan. We have learned to accept that tokens, those bite-sized chunks of text AI models digest, are scarce resources governed by the laws of physics. Look under the hood, and the reality is more nuanced.

Tokens 101: the maths behind the curtain

At a strictly technical level, neural networks do not understand words; they understand numbers. A token, which can be a syllable, a word or a punctuation mark, is the translation bridge that turns human text into numerical IDs and vector embeddings. Every model also has a maximum context window, set by hardware memory and by the heavy computational load of its attention mechanisms.

In short: tokens are technically mandatory for an AI to process language.

The corporate meter: infrastructure or rent-seeking?

There is, however, a wide gap between technical necessity and commercial metering. Cloud AI providers face massive infrastructure costs: sprawling GPU clusters, data centres to power and cool, engineers to pay. To recover those costs and make a profit, they use tokens as a granular unit for pricing tiers, rate limits and subscriptions.

When providers present token limits as a law of nature, they blur the line between real server capacity and artificial scarcity.

The self-hosted alternative

Want proof that tokens don’t have to rule your digital life? Run an open-source model locally, on your own hardware.

When you self-host, tokens disappear as an economic barrier. Your graphics card still splits text into tokens, mathematically, but the meter is gone. No one tracks your prompts, no bill grows in the background, and no one cuts you off mid-thought. The limits that remain are honest ones: your memory, your processor, your patience.