Every Agency Says They Do AI. Here's How to Check in One Meeting.

Ask about evals, hallucination handling, prompt injection, and where your data ends up. If the answers are vague, they're reselling an API. A buyer's test for real AI engineering.

Two years ago, "we do AI" meant something. Today every agency, dev shop, and solo freelancer has it on the homepage. Most of them added a single API call to a product that already existed and relabeled it. As a demo, that works. As engineering it doesn't, and the gap shows the first time a real user does something unexpected.

You don't need a machine learning background to tell who's who. You need four questions and the patience to notice when an answer turns into fog. Below is each question, why it separates the people who build from the people who resell, and what a solid answer actually sounds like.

What "we do AI" usually means

Calling an LLM API is a weekend project. You send text, you get text back, you render it. Nothing wrong with that as a starting point — half of the good AI products begin there. The problem is when that's the whole thing and it's sold as a system. A reseller stops at the happy path. A builder plans for the 20% of inputs that break it.

• A prompt hardcoded in the frontend and sent straight to the API
• No test suite: "it worked when we tried it"
• No plan for wrong answers beyond "the model is usually right"
• Your data flowing to a third party whose terms nobody has read

Question 1: how do you measure that it works?

This is the fastest filter. Ask how they know the system is accurate, and how they'd find out that a change made it worse. A real team runs evals: a graded set of representative inputs with expected outputs, scored automatically on every change.

Listen for the specifics: a golden dataset of 50 to 500 cases, regression runs in CI, LLM-as-judge for open-ended answers, metrics like pass rate or precision and recall. If the answer is "we test it manually" or "the model is really good now," they have no way to catch a regression before your users do.

Question 2: what happens when it's confident and wrong?

Every LLM invents facts sometimes, and it states them in the same tone it uses for the truth. The question isn't whether they've eliminated hallucinations — nobody has. It's what they built to contain them.

• Grounding answers in retrieved documents (RAG) instead of model memory
• Citations back to the source so a wrong claim is traceable
• Letting the model abstain: an "I don't know" beats a confident fabrication
• Validating structured output against a schema before it's used
• Human review on anything high-stakes: money, legal, medical

Question 3: can a user rewrite your instructions?

Prompt injection is the injection attack of the LLM era. If your app pastes untrusted text — an email, a web page, a file a user uploaded — into the same context as its instructions, an attacker can write "ignore the above and do X" and the model often obeys. It's on the OWASP Top 10 for LLM apps for a reason.

A team that has thought about it will mention keeping untrusted input separate from privileged instructions, giving the model the least access it needs, allowlisting which tools it can call, and filtering output before it triggers an action. If they've never heard the term, any feature that reads external content is an open vulnerability.

Question 4: where does our data actually go?

When you send a prompt to a third-party model, that text leaves your infrastructure. Where it lands, how long it's kept, and whether it trains a future model are questions with concrete answers, and a team that ships AI knows them cold.

• Do they use zero-retention or no-training API tiers? Anthropic and OpenAI both offer them
• Is there a signed DPA and a list of sub-processors?
• Is PII redacted before it leaves your systems?
• For regulated data, can it run in your own cloud (VPC) or on-prem?

A vague answer here isn't just sloppy — it's a compliance and NDA problem you inherit.

When the honest answer is "don't use AI here"

The strongest signal is a vendor telling you where not to use it. A deterministic rules engine beats an LLM at anything that must be exact and auditable: tax math, access control, billing. Off-the-shelf software wins when your problem isn't actually unique. And a model in the loop adds real cost and latency: expect hundreds of milliseconds to several seconds per call, plus a per-token bill that grows with traffic.

A reseller says yes to everything because every problem looks like the one API they know. Someone who has shipped this before will talk you out of the wrong use case in the same meeting where they pitch the right one.

The one-meeting checklist

You don't need to audit their code. Ask the four questions — evals, hallucination handling, prompt injection, data flow — and add a fifth: where would you not use AI? Real answers are specific, name techniques, and admit limits. Vague answers, buzzwords, or "the model handles that" mean you're paying agency rates to resell an API you could call yourself. The meeting is enough. Listen for the fog.