Chapter 12: Token as a Quantitative Measure of Project Scale

"A token count quantifies how much information a project contains. Convert everything the project has accumulated into tokens, and that number is a measure of the project's scale."

Further reading: on how context window limits create the need to measure things in tokens, see AI Statelessness and Context Window. On how token scale drives tool choice, see Tools and Context Selection: Why AI IDEs Sell "Context Selection Capability".

1. Why measure project scale in tokens

Traditional software engineering sizes a project with metrics like:

• lines of code (LoC)
• file count
• number of feature modules
• team headcount

Each of these breaks down in the AI-native era. AI-generated code tends to be verbose, so a high line count no longer implies high complexity. AI likes to produce lots of small files, so file count says little about real information density. Module boundaries get blurry when AI does the writing, so counting modules doesn't tell you much either. And AI raises individual output so dramatically that the old link between headcount and project scale is broken.

Tokens work better as a metric, for a few reasons:

1. They directly reflect what it costs an AI to understand the project. How many tokens it takes for an AI to take in the whole project determines:

   • whether the AI can handle the entire project in a single session
   • how many rounds of AI relay it takes to finish one feature request
   • how complicated your context selection strategy has to be

2. They put every kind of asset on one scale. Tokens can measure:

   • code
   • documentation
   • test cases
   • config files
   • even meeting recordings and design mockups, once converted to text

3. They map straight onto cost. Token consumption is what you pay for AI, so this number tells you:

   • whether the team can afford to keep using AI
   • which tools and models fit the project at its current size

2. The 1M and 10M token boundaries

In my experience, two numbers matter.

1M tokens is roughly everything a tiny AI-native team produces in 15 to 30 days. 10M tokens is the ceiling below which an AI-native approach still delivers at least a 3x productivity gain.

What 1M tokens means:

• a mid-sized codebase (tens of thousands to over a hundred thousand lines of code)
• plus the documentation and tests that go with it
• small enough to load entirely into a large-context model (Gemini's 1 million token window, say)
• the AI can grasp the whole project in a single session

What 10M tokens means:

• a large project, or a cluster of related projects
• too big to load into one session
• you have to rely on context selection so the AI only looks at the relevant parts
• but at this scale, if context selection is done well, AI still delivers a real productivity boost

Past 10M tokens:

• context selection gets sharply harder
• accuracy drops noticeably, because the AI only ever sees a slice of the context
• it may be time to split the project or rework the architecture, breaking it into several sub-projects of 1 to 10M tokens each

3. How to measure a project's token scale

A rough way to estimate:

1. Code. Add up all the text in the codebase and estimate with 1 token ≈ 4 English characters, or 1 token ≈ 1.5 Chinese characters. Or skip the estimate and count precisely with a tool like tiktoken.

2. Documentation. Add up all the markdown and text documents and apply the same ratio.

3. Tests. Test code plus test data. If you have a lot of test cases, this can be a surprisingly large share of the total.

4. Everything else. Config files, SQL schemas, API docs. If the project keeps transcribed meeting recordings or text versions of design mockups, count those too.

In practice:

• a script that periodically scans the repo and totals the tokens is enough
• watch the growth trend rather than the absolute number
• if the count is climbing fast, ask why: are you writing too much documentation or too many tests, or is the code full of duplication?

4. What token scale means for engineering practice

Under 1M tokens:

• you can afford to be a little extravagant with AI
• let it read the whole codebase and do global refactors and optimizations
• a bit of redundancy in the docs is fine, since context space isn't tight yet

Between 1M and 10M tokens:

• context selection starts to matter
• documentation needs trimming down to snapshot-style, high-level docs
• code needs to be modular enough that the AI can get work done from just the relevant modules

Past 10M tokens:

• context selection becomes the central bottleneck
• you may need heavier machinery like project indexes and symbol indexes
• or you split the thing into several independent smaller projects

5. Practical recommendations

For a small two-person team that leans heavily on AI, here is how I'd use the token metric:

1. Measure regularly. Once a month or once a quarter is enough. Watch the trend, and if growth spikes, figure out why.

2. Adjust strategy by scale. Under 1M, stay loose and let the AI read generously. Between 1M and 10M, tighten up context selection and slim down the docs. Past 10M, think about splitting the project or reworking the architecture.

3. Let the number drive tool choice. If the project fits under 1M tokens, Gemini Web with its 1 million token context can take the whole thing directly. Past 1M, you probably want tools like Codex or Claude Code that combine symbol indexing with smart context selection.

6. Summary

As a measure of project scale, tokens earn their place in the AI-native era:

• they reflect what it costs an AI to understand the project
• they put every kind of project asset on one scale
• they map directly onto what you pay for AI

1M tokens is the comfort zone; 10M tokens is the tipping point.
Inside that range, an AI-native approach keeps delivering at least a 3x productivity gain.
Beyond it, you need a much more sophisticated context selection strategy, or you split the project.

Measuring token scale regularly, and paying attention to it, is one of the main levers an AI-native team has for keeping project complexity in check.