Tokens vs Bytes in AI: What LLMs Actually See When You Type

You type "你好 Hello" into GPT-5. That's 7 characters. But the model processes it as 2 tokens — and your bill is based on those tokens, not the characters. Meanwhile, your computer stores that same t...

By · · 1 min read
Tokens vs Bytes in AI: What LLMs Actually See When You Type

Source: DEV Community

You type "你好 Hello" into GPT-5. That's 7 characters. But the model processes it as 2 tokens — and your bill is based on those tokens, not the characters. Meanwhile, your computer stores that same text as 12 bytes. So what's the difference between bytes, characters, and tokens? Why does AI use tokens instead of raw bytes? And why does the same sentence cost more in Chinese than in English? Start at the Bottom: What Is a Byte? A byte is the smallest unit of data your computer stores. One byte = 8 bits = a number from 0 to 255. When you save text to a file, your computer encodes each character into bytes using UTF-8: Character UTF-8 Bytes Byte Count Hex H 72 1 48 e 101 1 65 你 228, 189, 160 3 e4 bd a0 好 229, 165, 189 3 e5 a5 bd 🚀 240, 159, 154, 128 4 f0 9f 9a 80 Key pattern: English letters: 1 byte each Chinese/Japanese/Korean: 3 bytes each Emojis: 4 bytes each The Four Levels: Bytes → Characters → Words → Tokens Level "Hello, World" Count Description Bytes 48 65 6c 6c 6f 2c 20 57 6f 72 6

Related Posts

Trending on ShareHub

  1. Understanding Modern JavaScript Frameworks in 2026
    by Alex Chen · Feb 12, 2026 · 0 likes
  2. The System Design Primer
    by Sarah Kim · Feb 12, 2026 · 0 likes
  3. Just shipped my first open-source project!
    by Alex Chen · Feb 12, 2026 · 0 likes
  4. OpenAI Blog
    by Sarah Kim · Feb 12, 2026 · 0 likes
  5. Building Accessible Web Applications: A Practical Guide
    by Alex Chen · Feb 12, 2026 · 0 likes
  6. Rapper Lil Poppa dead at 25, days after releasing new music
    Rapper Lil Poppa dead at 25, days after releasing new music
    by Anonymous User · Feb 19, 2026 · 0 likes
  7. write-for-us
    by Volt Raven · Mar 7, 2026 · 0 likes
  8. Before the Coffee Gets Cold: Heartfelt Story of Time Travel and Second Chances
    Before the Coffee Gets Cold: Heartfelt Story of Time Travel and Second Chances
    by Anonymous User · Feb 12, 2026 · 0 likes
    #coffee gets cold #the #time travel
  9. Best DoorDash Promo Code Reddit Finds for Top Discounts
    Best DoorDash Promo Code Reddit Finds for Top Discounts
    by Anonymous User · Feb 12, 2026 · 0 likes
    #doordash #promo #reddit
  10. Premium SEO Services That Boost Rankings & Revenue | VirtualSEO.Expert
    by Anonymous User · Feb 12, 2026 · 0 likes
  11. NBC under fire for commentary about Team USA women's hockey team
    NBC under fire for commentary about Team USA women's hockey team
    by Anonymous User · Feb 18, 2026 · 0 likes
  12. Where to Watch The Nanny: Streaming and Online Viewing Options
    Where to Watch The Nanny: Streaming and Online Viewing Options
    by Anonymous User · Feb 12, 2026 · 0 likes
    #streaming #the nanny #where
  13. How Much Is Kindle Unlimited? Subscription Cost and Plan Details
    How Much Is Kindle Unlimited? Subscription Cost and Plan Details
    by Anonymous User · Feb 12, 2026 · 0 likes
    #kindle unlimited #subscription #unlimited
  14. Russian skater facing backlash for comment about Amber Glenn
    Russian skater facing backlash for comment about Amber Glenn
    by Anonymous User · Feb 18, 2026 · 0 likes
  15. Google News
    Google News
    by Anonymous User · Feb 18, 2026 · 0 likes

Latest on ShareHub

Browse Topics

#artificial intelligence (5110)#deep learning (3221)#pro graphics (2571)#ai (2012)#generative ai (1655)#gaming (1647)#3d (1638)#news (1366)#geforce now (1192)#cloud gaming (1161)

Around the Network