A few articles ago we met chain-of-thought: asking a model to reason step by step instead of blurting out an answer. It’s a useful technique, and for most problems it’s all you need. But some problems are trickier in a particular way: there isn’t one obvious path to the answer. There are several possible approaches, and the best one isn’t clear until you’ve explored a bit.
For those, there’s a heavier cousin called tree-of-thought. This article is about the difference between the two, and just as importantly, about knowing when the fancier one is worth the extra effort and when plain chain-of-thought is the smarter choice.
Chain-of-thought: one train of thought
Let me recap chain-of-thought in a single image: it’s the model following one line of reasoning, step by step, from start to finish. Step one leads to step two leads to step three leads to the answer. It’s a straight path, a single train of thought carried through to its conclusion.
This works well when there’s a fairly clear route to the answer, even if that route has several steps. Our earlier apple problem was like this: work out Monday, then Tuesday, then you’re done. One sensible path, followed carefully. For most reasoning tasks, that’s exactly what you want.
Tree-of-thought: explore several paths, then choose
But now imagine a problem where the approach itself is uncertain: a planning puzzle, a strategy question, a riddle where the first idea you try might hit a dead end. Following a single line of reasoning is risky here: if you happen to start down the wrong path, you’ll carry that early wrong turn all the way to a wrong answer, with no chance to reconsider.
Tree-of-thought handles this by letting the model explore multiple lines of reasoning, like branches spreading from a trunk. Instead of committing to one path, it considers several possible approaches, develops each a little way, evaluates which ones look promising, and pursues the best, sometimes backing out of a dead end and trying another branch. Figure 1 shows the contrast.
Figure 1: Chain-of-thought follows one line of reasoning straight to an answer. Tree-of-thought branches out, explores several possible paths, evaluates them, and pursues the most promising, able to abandon a dead end and try another.
The everyday analogy is how you’d tackle a hard decision. With something simple, you just think it through once, in a line. With something knotty, you don’t commit to your first idea. You brainstorm a few different approaches, weigh each, maybe start down one and realise it won’t work, and switch. That deliberate exploring-and-comparing is tree-of-thought.
What it looks like in a prompt
You don’t need special software to get a taste of this. You can encourage the branching behaviour right in your prompt. Rather than “think step by step,” you ask the model to consider several approaches and compare them:
Solve this problem. First, brainstorm three different approaches you
could take. Briefly explore where each one leads. Then evaluate which
approach works best, and use it to reach your final answer.
Problem: [your tricky problem here]That prompt nudges the model to generate several branches, assess them, and choose. That’s the essence of tree-of-thought, achievable with careful wording. (Full tree-of-thought systems, where software automatically manages and scores many branches, go further than a single prompt. But for most people, this prompt-level version captures much of the benefit.)
The catch: it costs more
Here’s the trade-off, and it’s the reason you shouldn’t reach for tree-of-thought by default. Exploring several paths means the model does much more work: more reasoning, more text generated, more time, and, if you’re paying per token, more money. Where chain-of-thought walks one path, tree-of-thought walks several and then compares them. That’s inherently slower and more expensive.
So tree-of-thought isn’t simply “better” than chain-of-thought. It’s a heavier tool for harder problems. Using it on a task that a single line of reasoning would have solved is just paying extra for no benefit.
Which to use: a simple rule
Figure 2 gives you the decision at a glance, but the rule is easy to remember.
Figure 2: Default to chain-of-thought. Reach for tree-of-thought only when the problem has several possible approaches, the best one isn’t obvious, and getting it right is worth the extra cost.
- Default to chain-of-thought. For most reasoning tasks, including most maths, logic, and multi-step problems, a single careful line of reasoning is enough, and it’s faster and cheaper.
- Reach for tree-of-thought when the problem has multiple possible approaches, the right one isn’t obvious up front, a single wrong turn early would ruin the whole answer, and the problem is important enough to justify the extra effort. Think complex planning, certain puzzles, open-ended strategy.
This echoes a principle that runs through this track and, really, through all of working with AI: use the simplest approach that solves your problem, and add complexity only when the problem demands it. Tree-of-thought is a useful tool to have in your kit, but a tool you reach for deliberately, for the hard cases, not a default upgrade.
The bigger lesson
Beyond the specific techniques, there’s a mindset here worth carrying forward. As problems get harder, the useful move is often to give the model room to explore rather than forcing it down a single path, but always weighed against the cost of that exploration. Knowing not just how to use a technique but when it’s worth it is what separates thoughtful prompting from just piling on complexity.
We’ve now covered reasoning, from a single chain to a branching tree. Next, we shift from helping the model think to controlling exactly what it outputs, specifically how to get clean, structured results that a program can rely on.
Next in the series: Structured Output: Reliable JSON Every Time: getting output a program can actually use.