Why choosing an AI agent partner is hard
Many agencies now describe chatbots as agents, and almost any team can build an impressive demo. The difference shows up on real data: wrong answers, failed API calls, runaway costs and actions nobody approved. This checklist helps you find the teams that plan for those problems.
Proof: have they shipped agents that work?
- Production work, not demos. Ask for a walkthrough of an agent that runs on real data today: what it does, what it connects to and how it is monitored.
- A client you can speak to. A team that has shipped agents can usually arrange a reference call, even if the case study is under NDA.
- Reviews that mention AI work. Check Clutch, GoodFirms or G2 for reviews that describe agent or LLM projects specifically, not just web or mobile work.
Engineering: will the agent hold up on real data?
- Real tool calling. The agent should act through your APIs and databases with validated inputs, not by copying text between screens.
- Evaluations. Ask to see how they build a test set from your cases and what score an agent must reach before launch.
- Failure handling. Ask what happens when the model times out, an API fails or the agent is unsure. Good answers include retries, fallbacks and hand-off to a person.
- Logging and monitoring. Every run should be traced, including tool calls, cost and latency, with alerts when things drift.
- Permissions and human approval. Risky actions such as sending, paying or deleting should require approval, and credentials should be least-privilege.
Security and ownership
- Data handling. Your data should go through business API tiers that do not train on it, with personal data redacted where needed and a clear answer on where it is stored.
- You own the work. Code, prompts, evaluation sets and infrastructure should belong to you, not to the agency's platform.
- No model lock-in. The system should let you switch model providers as prices and quality change.
Commercials
- A paid pilot with success criteria. Before a large build, agree on one workflow, a few weeks of work and the numbers that decide whether to continue.
Also ask for a written estimate of running costs per task and the terms of post-launch support.
Red flags
- Promises of 100% accuracy or 'fully autonomous' agents from day one.
- No evaluation set, or quality judged only by trying a few prompts.
- Demos that only run on sample data the vendor prepared.
- No answer on what an agent costs per task to run.
- A proprietary platform you cannot leave without rebuilding.
Questions to ask on the first call
- Which agent you built is running in production today, and what does it do?
- How will you measure whether the agent is good enough to launch?
- What happens when the agent is wrong or unsure?
- Which actions will need human approval?
- What will it cost per task to run, and how will we see that?
- Who owns the code, prompts and evaluation data at the end?
