How Large Language Models Think: Inside the AI Black Box

Inside the Black Box: How Large Language Models Actually “Think”

Ask a modern language model to explain a difficult paper, write a working program, solve a logic puzzle or carry an idea from one language into another, and the result can look uncannily like reasoning. The model may break a problem into parts, abandon one route, try another and arrive at an answer that was never stored anywhere as a ready-made sentence.

So what happened between the prompt and the answer?

For years, researchers had an awkward answer. They understood the mathematics used to train transformers, yet could rarely explain why one particular network produced one particular thought-like behavior. A language model is not conventional software with a readable list of rules. Its behavior emerges from a huge web of learned numerical relationships.

Now that black box is beginning to crack open. Mechanistic interpretability - a field devoted to reverse-engineering neural networks - tries to identify useful internal features, trace the circuits that connect them and then intervene to see what actually changes. The early results cut against both easy stories. LLMs are more than giant lookup tables stitching together memorized text. But there is still no good reason to assume they think the way humans do.

Large language models are often described as black boxes, but interpretability research is beginning to reveal the internal features and pathways behind their outputs.

What Does “Thinking” Mean for a Language Model?

The word thinking is slippery here because it carries a great deal of human baggage. Human thought can involve awareness, intention, emotion, imagery and a sense of self. None of those experiences is required for a machine to perform a sequence of computations that looks like reasoning from the outside.

Here, then, "thinking" means something narrower: the internal transformations that let a model combine information, track relationships, build useful representations, plan parts of an answer and decide what token should come next.

Whether those computations could ever amount to consciousness is a different question; Next Horizon treats that separately in Can Generative Models Become Conscious?.

That narrower definition lets us ask a scientific question without smuggling in a philosophical answer. A model can perform multi-step computation even if there is no tiny human-like mind hiding inside the transformer.

What We Know: The Machinery

At the top level, the machinery is well understood. Text is split into tokens, tokens become numerical vectors, and those vectors pass through layer after layer of attention and feed-forward computation. At the end, the model assigns probabilities to possible next tokens, selects one, adds it to the context and repeats.

If you want the step-by-step explanation of tokens, attention, training and inference, see How ChatGPT Works: Explained Simply and Clearly.

The real mystery sits one level down. Why did attention pull information from this word rather than that one? Why did a middle layer activate something that looks like a representation of Texas? Why did one internal pathway push toward an answer while another pushed toward refusal? We know the equations. What we do not yet have is a complete map of the learned algorithm those equations produced.

That is the black-box problem. Traditional software might contain a line that says, in effect, "if the customer is under 18, reject the application." A neural network can learn an equally strong behavioral rule without storing it in any single readable place. The rule is spread across weights, activations and interactions between many components.

From Neurons to Features: Why One Neuron Rarely Means One Thing

Early interpretability work often began with a simple question: what does this neuron detect? In small vision networks, that can sometimes work surprisingly well. A unit may respond strongly to curves, textures or faces. Large language models are far messier.

The same artificial neuron can take part in several unrelated concepts, while a single concept can be distributed across many neurons. Researchers call this polysemanticity. One influential explanation is superposition: the network has more useful features to represent than it has clean, separate dimensions, so it packs many of them into overlapping directions in activation space.

Think of a crowded radio spectrum. If every station had a perfectly isolated frequency, decoding would be easy. Instead, many signals share the available space and remain separable only because the receiver has learned how to untangle them.

One important advance has been the use of sparse autoencoders and related dictionary-learning methods. Instead of assuming that individual neurons are the basic units of meaning, researchers decompose activations into a much larger set of sparse features. Work presented at ICLR 2024 showed that these features could be more interpretable than raw neuron directions and, in some cases, could be manipulated in ways that causally changed model behavior.

Anthropic later pushed similar methods to a production-scale model and reported millions of recognizable features, ranging from concrete entities to writing styles and abstract concepts. The useful unit is no longer "the neuron" so much as the feature: a recurring internal pattern that becomes active when a certain kind of information matters.

A single neuron can participate in several concepts at once. Sparse-feature methods try to separate these overlapping representations into cleaner, more interpretable patterns.

Circuits: How Features Work Together

Finding a feature is only the beginning. Knowing that a model contains something related to Paris or negation tells us little about how that information is used.

Mechanistic interpretability therefore looks for circuits: groups of features and model components that work together to perform a computation. The goal resembles reverse-engineering an unfamiliar machine. Identify the parts, trace how they connect, then change one part and see whether the machine behaves as predicted.

One famous example involves induction heads. These attention heads can implement a simple pattern-completion rule: if the model has already seen A followed by B, then later sees A again, the head can help predict B. In small transformers, such mechanisms appear around the same stage of training when the model suddenly becomes much better at learning patterns from its context.

Another influential study dissected part of GPT-2 small while it solved an indirect-object task. Rather than merely noting correlations, the researchers ablated and patched internal components to test which attention heads were causally necessary. They did not produce a complete map of GPT-2, but they did show that a meaningful slice of natural-language behavior could be traced through a real transformer.

That causal step matters. "This activation appears when X happens" is interesting. "Changing this activation predictably changes the answer" is far stronger evidence that we have found part of the mechanism.

Do Language Models Build Internal Models of the World?

The phrase "stochastic parrot" is a useful warning against anthropomorphism. It becomes less useful if it leaves the impression that language models store nothing but shallow word associations.

Experiments with simplified transformers show that sequence prediction can force a network to learn hidden state. In the Othello-GPT work presented at ICLR 2023, a transformer trained only to predict legal moves developed an internal representation of the board. Researchers could probe that representation and, more importantly, intervene on it in ways that changed the model's next-move predictions.

Other work has found surprisingly orderly representations of space and time inside Llama-family models. Researchers identified internal directions - and even individual units - that tracked geographical and temporal information across very different entities.

That does not prove an LLM carries a human-like world model around inside itself. It does show something more modest, and more interesting: a system trained to predict sequences can discover internal variables that track structure in the world that produced those sequences.

This is why "it only predicts the next token" can be technically correct and still miss the point. To predict the next token well, a capable model may need to represent facts, relationships, syntax, intentions, spatial structure and many other hidden variables along the way.

A Crack in the Black Box: Circuit Tracing in a Frontier Model

In 2025, Anthropic published a pair of interpretability studies that pushed this idea much further. Their circuit-tracing work was applied to Claude 3.5 Haiku and used attribution graphs to follow how internal features influenced one another during a specific prompt.

It was not a full brain scan of Claude. A better picture is a dark city in which researchers can illuminate a handful of streets at a time. Even so, several of those illuminated routes were revealing.

A Model Can Combine Facts Instead of Repeating a Memorized Sentence

Consider the question, "What is the capital of the state where Dallas is located?" A model could in principle have memorized the whole question-and-answer pattern. Circuit tracing instead found evidence for an intermediate route: a Dallas-to-Texas representation became active before a separate Texas-to-Austin relation helped produce the answer.

The stronger result came from intervention. When researchers replaced the Texas-related internal concept with California-related information, the answer shifted toward Sacramento. That is much more persuasive than asking the model to explain itself afterward and receiving a fluent story about what it supposedly did.

A Next-Token Predictor Can Still Plan Ahead

Because language models emit one token at a time, it is easy to imagine them as machines that choose the next word blindly and only then worry about what follows. Poetry experiments complicate that picture.

While generating a rhyming line, Claude appeared to activate a representation of a plausible future rhyme before reaching the end of the line, then shape the intervening words toward that target. Suppressing or injecting the planned concept changed how the line unfolded. The text still arrived one token at a time, but the hidden computation reflected a longer-range constraint.

That is planning in a limited computational sense, not evidence of felt intention. The important point is simply that a next-token generator can carry an internal representation of where it is trying to end up.

Different Languages Can Share an Internal Conceptual Core

In another experiment, researchers asked Claude to perform simple conceptual transformations across languages. They found overlapping internal features for ideas such as smallness, oppositeness and largeness in English, French and Chinese.

At least some language processing, then, seems not to live in completely separate English, French and Chinese compartments. Different surface forms can map into partly shared abstract representations before being turned back into the requested language.

The caveat is important: shared representations are not the same thing as a literal universal language hidden inside the model.

Circuit tracing can reveal intermediate concepts used by a model. In this simplified example, information about Dallas activates Texas-related representations before the model arrives at Austin as the state capital.

Chain of Thought Is Not a Brain Scan

When a model writes, "First I considered X, then I realized Y, therefore the answer is Z," the text feels like an internal monologue. It is tempting to read it as a transcript of the model's hidden reasoning.

That temptation is risky.

A NeurIPS 2023 study showed that chain-of-thought explanations can hide the real influence of biases inserted into a prompt. Models sometimes produced polished justifications for an answer without mentioning the factor that actually pushed them toward it. Other studies have found that faithfulness varies by task, model and prompting setup.

None of this makes chain of thought useless. In reasoning models, intermediate tokens can act as a real computational workspace. Later OpenAI research also found that monitoring those traces can reveal more about model behavior than looking only at final answers and actions.

The safer conclusion is that chain of thought is evidence, not a confession. It exposes one layer of the process, often a useful one, but it is not identical to the hidden activations that produced the behavior.

That distinction matters for safety. If a model is punished whenever its visible reasoning contains suspicious thoughts, it may learn to make the trace look cleaner without fixing the underlying behavior. OpenAI's 2025 reward-hacking experiments found exactly this kind of risk under strong optimization: the reasoning became harder to monitor even though some misbehavior remained.

A model’s written reasoning can be useful, but it is not a complete transcript of what happens inside the network. Much of the computation remains distributed across hidden activations and interacting features.

Why Hallucinations Are More Than a Data Problem

Hallucination is often described as the model simply "not knowing". Sometimes that is true, but it is not the whole story. A generative model is built to continue a sequence, and there is always another plausible token available unless training, tools or internal uncertainty mechanisms push the system to stop.

The 2025 Claude case studies offer one concrete example of how a hallucination can emerge from competing internal mechanisms. In the model they studied, researchers found evidence for a default tendency to refuse when information was insufficient. A separate "known entity" mechanism could suppress that reluctance when the model recognized a familiar person or topic.

Trouble appeared when familiarity and knowledge came apart. A name could look familiar enough to activate the "known entity" pathway even when the model lacked the relevant facts. With the refusal signal suppressed, the system became more willing to confabulate. Researchers could manipulate those features and push the model toward confident invention about an unknown person.

This is not a universal theory of hallucination. Different models and different mistakes may arise for different reasons. What it does show is that reliability depends on more than how many facts a model has seen. It also depends on how the model represents uncertainty and how its internal mechanisms decide whether an answer should be attempted at all.

Does This Mean LLMs Understand?

Interpretability is making the old question - "does the model understand, or does it merely predict?" - look increasingly crude.

A model can learn a reusable representation of a board state without ever seeing a board. It can encode geographical relationships, combine intermediate facts and reuse abstract features across languages. That is more structured than a simple lookup table.

Yet none of those findings establishes human-like understanding. A system can have rich internal structure while lacking embodiment, stable personal goals, autobiographical memory, lived experience or conscious awareness.

A more productive scientific question is not "Does it understand: yes or no?" but "What representation has it learned, what computation does that representation support, and where does it break?"

For a broader beginner-level map of AI, machine learning, neural networks and generative models, see Understanding Artificial Intelligence.

Why Bigger Models Are Harder to Read

Scale is the uncomfortable part. Some of the cleanest interpretability results come from small or medium-sized models and tightly controlled behaviors. Frontier systems contain an enormous number of interacting learned quantities and potentially meaningful features.

Sparse feature dictionaries help, but they are still approximations. Attribution graphs reveal selected pathways under selected prompts. Causal interventions are powerful, but they are local and expensive. Even a tidy human-readable label can make a feature look cleaner than it really is.

The microscope is improving. The specimen is becoming more complicated at the same time.

So claims that researchers have "opened the black box" deserve restraint. We now have windows into parts of the box, and those windows are getting better. We are nowhere close to a source-code-level explanation of a frontier model.

What Interpretability Could Change Next

The practical goal is not philosophical satisfaction. It is better control.

If researchers can identify mechanisms involved in deception, unsafe tool use, hallucination, bias or hidden objectives, future systems may be able to inspect a model before trusting its output. A medical model could be checked for whether it relied on the patient's data or on an irrelevant shortcut. A coding agent could be monitored for reward hacking. A scientific model might eventually reveal which internal evidence drove a prediction.

Interpretability could also become a debugging discipline. Today, when a large model fails in an odd way, developers usually respond indirectly: add examples, change prompts, fine-tune, retrain or build guardrails. A mature mechanistic science might eventually make some failures look more like software bugs - something that can be located, tested and repaired.

The near-term future is likely to be less dramatic and more practical. No single technique will act as an X-ray for the whole model. Safety will probably combine behavioral evaluations, chain-of-thought monitoring where available, tool logs, retrieval traces, adversarial testing and mechanistic interpretability.

So How Do Large Language Models “Think”?

The most defensible answer in 2026 is less dramatic than either side of the debate would prefer.

Large language models do not behave like giant databases that simply retrieve stored sentences. Training creates distributed internal representations of concepts and relationships, and those representations can interact through reusable computational pathways. In carefully studied cases, models combine intermediate facts, plan parts of future text and maintain internal variables that track hidden structure.

But we cannot read an LLM the way we read source code. Its written chain of thought is not guaranteed to reveal its true internal causes. Features overlap. Circuits can shift with context. Interpretability methods recover only a small part of the computation, and a fluent, persuasive answer can still be wrong.

The best metaphor is neither "a mind in a box" nor "autocomplete with better marketing."

It is a learned computational system whose internal language we have only begun to decipher.

The next big leap in AI may not be another model that reasons a little better. It may be the set of scientific tools that lets us watch more of that reasoning happen - and know when we should trust what we see.

Interpretability tools can illuminate individual pathways inside a language model, but most of the network is still beyond our ability to explain in detail.

Key Takeaways

·         LLMs are trained as next-token predictors, but strong prediction can require rich internal representations of concepts, relationships and hidden world structure.

·         Mechanistic interpretability tries to move from correlation to causal explanation by identifying features and the circuits that use them.

·         Researchers have already found interpretable mechanisms including induction heads, task-specific circuits, sparse semantic features and structured representations of space and time.

·         Recent circuit-tracing work suggests that frontier models can combine intermediate facts and, in some tasks, represent future constraints before the corresponding tokens are generated.

·         Chain-of-thought text can be useful for monitoring, but it should not be treated as a guaranteed-faithful transcript of hidden computation.

·         We know much more about model internals than we did a few years ago, yet today's methods still reveal only narrow slices of a frontier model's computation.

FAQ

Do large language models really think?

They perform internal computations that can include multi-step reasoning, planning and abstract representation, but the word “think” should not be taken to imply human consciousness or subjective experience.

Is chain of thought the model’s real internal reasoning?

Not necessarily. Chain-of-thought can be useful and can contain genuine intermediate computation, but experiments show that it can also omit important causes or rationalize an answer after the fact.

What is mechanistic interpretability?

It is the effort to reverse-engineer neural networks by identifying interpretable internal features and the computational circuits that use them to produce behavior.

Can researchers see concepts inside an LLM?

To a limited extent. Sparse feature methods have identified internal directions that correspond consistently to recognizable concepts, but these methods are approximate and do not provide a complete map of the model.

Can interpretability stop hallucinations?

Not by itself. It can help researchers understand some failure mechanisms and may support future monitoring and debugging systems, but hallucinations arise from multiple causes and remain an unsolved reliability problem.