From Prompt to Product: When Prompting Becomes Engineering

Part 3 of Demystifying Prompt Engineering

Part 1 gave us the mental model. The LLM finishes documents. Part 2 used it to write a better single prompt. This part is about everything after the single prompt.

Because the demo always works. You craft one clever prompt, it nails the example you tried, and for a second it feels like magic again. Then you wire it into something real, whether a feature, an agent, or a pipeline, and it falls over in ways your demo never showed.

The fix isn’t a better prompt. It’s treating the model for what it is. It’s powerful, fast, and fundamentally undependable, but you don’t argue it into reliability. You build the system around it. Here’s what that takes. 💡

1. Let it think before it answers

The model has no inner monologue. Ask a hard question and it blurts the first plausible continuation, with no scratch work, no “let me reason this through,” just an instant guess. And an instant guess on a multi-step problem is usually wrong. Now, some models do have advanced thought, where it tests the first plausible continuation against other things it knows to be true instead of just taking it and outputting it as the answer outright.

So give it room to think out loud, in the output. This is chain of thought, and it’s almost embarrassingly simple. Ask the model to work through its reasoning before giving the final answer. Even a plain “let’s think step by step” measurably bumps accuracy on anything with moving parts. The reasoning is the model doing the work it can’t do silently.

The grown-up version is ReAct, short for reason + act. The model alternates between thinking and doing. It thinks about what it needs, takes an action (search, look something up, call a tool), observes the result, and thinks again. It’s how you get from “answer from memory” to “go find out, then answer.” For anything that needs facts the model doesn’t already have, that loop is the difference between confident fiction and a real answer.

Here’s the rule of thumb. If a problem has more than one step, make the model show the steps. 🛠️

2. Tools without tears

A bare model is trapped in its own head. It can’t look up today’s data, can’t do reliable math, can’t do anything in the real world. Tools fix that. You tell the model what’s available, it decides when to call one, and your application runs the actual function and feeds the result back.

Here’s the thing nobody tells you. A tool definition is a prompt. The model reads your function names, descriptions, and arguments the same way it reads everything else. So write them for a human to understand:

• Name things clearly. getUserEmail reads; retrieveemail doesn't. The model leans on names that look like code it's seen.
• Keep arguments few and simple. The more arguments, and the fancier their types, the more the model fumbles, including inventing argument values that were never there. (Yes, it hallucinates function arguments. Same root cause as everything else.)
• Don’t paste your whole API in. Too many tools and the model gets confused about which to reach for. Give it a small, clean set that covers the job.

And one hard line. Never trust the model to gate a dangerous action. “Ask the user before deleting anything” will fail some fraction of the time, because the model is undependable by nature. Don’t make it a rule in the prompt, make it a wall in your code. Intercept the destructive tool call and require a real human sign-off before it runs. The model can suggest; your application decides.

3. Agents vs. workflows: know which one you’re building

When one prompt isn’t enough, people reach for an “agent,” handing the model some tools and a goal and letting it figure out the rest. Sometimes that’s right. Often it’s not.

There’s a real fork here:

• A workflow is a fixed path you design, where step one feeds step two feeds step three. Rigid, but strong, because you’ve encoded the structure. Reach for it when the job is specific and repeatable.
• An agent decides its own path at runtime. Flexible, but weaker and less predictable. Reach for it only when the path genuinely can’t be known ahead of time.

Most “agent” projects should’ve been workflows. If you know the steps, encode the steps, and don’t make the model rediscover them on every run and hope it gets there.

Two principles keep these systems sane, both straight from the people who built Copilot:

• Whenever you can avoid using an LLM, do so. A web crawler, a database write, a plain old classifier. If regular code can do a step reliably and cheaply, use regular code. Save the model for the parts that actually need language.
• Confine the LLM to small, well-defined tasks, then wire those tasks together with ordinary, deterministic plumbing. When something breaks, and it will, you want to isolate the failure to one task, not debug a mysterious blob of “the agent did something.”

The pattern that scales is boring, predictable structure on the outside, with the model doing one clear job at each node on the inside.

4. Write the eval first

This is the one everyone skips, and it’s the one that actually separates a toy from a product.

You cannot improve what you can’t measure. Change a prompt and the output looks different. Better? Worse? You genuinely can’t tell from eyeballing one or two runs, because the model’s nondeterministic and your gut is biased toward the version you just wrote. The team behind Copilot put it bluntly. The first thing they built was the evaluation, and it’s the only reason they could move fast without breaking things.

You don’t need a research harness on day one. Start tiny:

• An example suite. Five to twenty representative inputs, a script that runs your prompt against all of them, and a way to diff the results. That’s it. The moment you can see all your outputs side by side, the typical failure modes jump out.
• Grade on the aspect that matters. Exact-match is usually too strict for free-form output, so check the thing you actually care about, like whether the support bot resolved the issue, not whether it phrased things word-for-word like your gold answer.
• You can let a model grade, but carefully. LLM-as-judge works, but never let a model grade its own output, and never tell it it’s grading itself, because that invites every bias you’d expect. Structure the grading. Ask specific questions, on a defined scale (say 1 to 5 with each level described), across the separate aspects you care about. Precise questions force more objective answers.

Offline evals like this let you kill bad ideas cheaply, before they ever touch a user. The good ones graduate to a real test against live traffic.

The whole series in one line

Three parts, one arc. The LLM finishes documents (Part 1). So write it a clean one (Part 2). And when one document isn’t enough, wrap it in a system that reasons, uses tools carefully, stays mostly deterministic, and measures itself (Part 3).

Notice what happened to the magic along the way. It didn’t survive contact with any of this. A document-completion engine, curated context, deterministic plumbing, and an eval suite add up to plain engineering. And that’s the good news, because engineering is learnable. You don’t need the secret prompt. You need the mental model and the habits, and now you’ve got both.

If you want the full, unabridged version of all this, the book this series distills is genuinely worth your time. It’s Prompt Engineering for LLMs by John Berryman and Albert Ziegler. These three articles are the map. The book is the territory. ✨

You’ve now got the whole framework, so which part are you putting to work first? Tell me what you’re building, and where the model’s been fighting you.