What a large language model is: tokens, context and the two phases of its life

Every AI product your company evaluates this year, from a document summarizer to a customer-support agent, is built on a large language model, or LLM. You do not need to understand the mathematics, but four ideas explain most of the behavior, costs and limits you will run into: tokens, the context window, training versus use, and randomness.

What it is

Anthropic's documentation defines LLMs as AI language models "with many parameters" that are trained on vast amounts of text and can generate text, answer questions and summarize information ^[1]. The architecture behind nearly all of them, the Transformer, was introduced in a 2017 research paper from eight authors that proposed a design "based solely on attention mechanisms" and reported state-of-the-art translation quality while training in 3.5 days on eight GPUs ^[2].

Tokens: the unit of everything

Models do not read words. They read tokens, which the Anthropic glossary describes as the smallest individual units of a language model, corresponding to words, subwords, characters or even bytes ^[1]. Tokens matter to a business for two reasons.

First, they are the billing unit. GPT-5's API price, as reported in press coverage, is $1.25 per million input tokens and $10.00 per million output tokens ^[3]. Output costs more than input, so a task that generates long answers costs more than one that reads long documents and answers briefly.

Second, they are the capacity unit, which brings us to context.

The context window: working memory

The context window is the amount of text a model can look back on and reference when generating, described as a "working memory" distinct from the corpus it was trained on ^[1]. A larger window lets the model handle longer, more complex prompts; a smaller one limits it ^[1]. As one example, OpenAI's open-weight gpt-oss models, released in August 2025, have a 128K-token window ^[4].

A concrete example: you want a model to summarize a 60-page supplier contract and flag unusual termination clauses. The contract, your instructions and the model's answer all have to fit in the window for one request. If they do not, you need to split the document or retrieve only the relevant sections, a technique called retrieval-augmented generation, which passes selected documents into the context window at query time ^[1].

Two phases: training and inference

The glossary notes that a pretrained model alone is "not inherently good at answering questions or following instructions", which is why the refinement step exists ^[1]. Importantly, ordinary use does not teach the model anything. What it knows about your business is only what you place in the context window.

Why answers vary

Temperature is a setting that controls randomness: higher values yield more varied output, lower values more conservative, predictable output ^[1]. Even so, the glossary states that at temperature 0 results are not fully deterministic and identical inputs may produce different outputs across API calls ^[1]. For business processes, this means a model's output should be validated by rules or a person, not assumed repeatable.

What this means for buying decisions

• Ask how a quote is priced: input tokens, output tokens, and any cached-input discount ^[3].

• Ask what has to fit in the window for your use case, and what happens when it does not.

• Ask where company-specific knowledge will come from at run time, since the model will not have it by default.

Practical implication

Before piloting, estimate your documents and prompts in tokens and decide how the model will get the right information into its window. That sizing exercise predicts both cost and accuracy better than any model comparison table.