AI Agents vs Chatbots vs Automation: What’s right for your business?
July 17, 2026
AI Workflow Automation for SMBs: 9 use cases that pay for themselves in 30 days
July 17, 2026AI Agents vs Chatbots vs Automation: What’s right for your business?
July 17, 2026AI Workflow Automation for SMBs: 9 use cases that pay for themselves in 30 days
July 17, 2026Most agency evaluations are theatre. The agency runs a polished discovery call, shows you three impressive demos that may or may not be theirs, sends a proposal heavy on logos and light on specifics, and gets the contract signed before you have time to think.
Six months later you have a half-finished system, a relationship that has soured, and a budget that has tripled.
This is avoidable. AI agencies are easy to evaluate if you know what to ask. The questions below are the ones we have seen separate the agencies that ship from the agencies that pitch. Use them on every shortlist call.
If you have not read our cornerstone piece on what an AI agency actually does in 2026 and our breakdown of agents vs chatbots vs automation, start there. This piece assumes you know what you want built. Now we are evaluating who builds it.
Twelve questions. Ten minutes each. Worth every minute.
Ask all of these. In this order.
Each question reveals something the agency may not want you to see. The order matters — early questions filter, later questions deepen.
"Walk me through three AI projects you have shipped — with measurable outcomes."
Why you're asking: to separate agencies that have shipped from agencies that have pitched. Anyone can have a portfolio page; not everyone can describe what they actually built.
Good answer: three specific projects, named clients (or named industries if NDA), specific problems, specific solutions, specific numbers. "We built a lead-qualification agent for a UK insurance broker. Cut their response time from 4 hours to 4 minutes. 38% increase in booked meetings in 90 days."
Bad answer: vague case studies. "We helped many clients leverage AI to transform their workflows." Logos with no story.
"What does your discovery process look like before you scope?"
Why you're asking: agencies that scope projects on a single sales call always under-scope. Real discovery takes time. The depth of their discovery tells you the seriousness of the engagement.
Good answer: a structured process with named steps and a defined output. "We run a paid 2-week discovery sprint. Workshop with your team, audit your current systems, define success metrics, document the workflow as-is and to-be. Deliverable is a 20-page scope document with fixed pricing."
Bad answer: "We can start next week." If they will scope your project from one Zoom call, the change orders will follow.
"What happens when the AI gets it wrong?"
Why you're asking: AI gets things wrong. The agency's answer reveals whether they have actually run AI systems in production.
Good answer: a clear-eyed walk-through of failure modes. Hallucinations. Confidence scoring. Human-in-the-loop checkpoints. Escalation paths. Logging. Rollback plans. The agency talks about failure as inevitable and designs around it.
Bad answer: "We use the latest models so accuracy is very high" or "we test it thoroughly before deployment." Both translate to "we have never been on the phone at 2am because a bot did something stupid."
"Who owns the code, prompts, data, and credentials after launch?"
Why you're asking: some agencies build on proprietary platforms that you cannot leave without rebuilding. Some keep the prompts in their own accounts and refuse to hand them over. Both are tactics to lock you in.
Good answer: "You own everything. Code in your GitHub. Prompts in your repo. Credentials in your accounts. Data on your infrastructure. You can fire us tomorrow and another agency can pick it up next week."
Bad answer: any hedging on this. Walk away. There are dozens of other agencies who will commit in writing.
"What is the all-in cost — including model usage — over twelve months?"
Why you're asking: the build cost is often only 40% of the year-one total. The other 60% is model API fees, hosting, monitoring, and retainer. Agencies that hide this are setting up surprise invoices.
Good answer: a 12-month total that includes build, expected model spend at projected volume, hosting, monitoring, and retainer. Itemised. With sensitivity ranges. "If your usage doubles, model spend will go from $400 a month to $800."
Bad answer: a build cost with no mention of run cost. "Model fees are pass-through, we'll let you know." That is a phrase you will see again on invoice #4 and you will not enjoy it.
"Which AI models do you use, and why?"
Why you're asking: agencies that are loyal to one model provider are usually loyal because of partner discounts, not because the model is best for your problem. You want model-agnostic thinking.
Good answer: "We use GPT for tool-heavy agents because OpenAI's function calling is the most reliable. Claude for nuanced writing and reasoning. Gemini when the workflow touches Google Workspace. For your project we would use X because Y." Specific, reasoned, opinionated.
Bad answer: "We use ChatGPT for everything" or "we use the most advanced model available." Both signal the agency has not thought hard about model selection.
Put us on the list. Ask us the twelve questions. See how we answer.
"How do you handle our data, and what compliance frameworks do you work to?"
Why you're asking: for UK clients, GDPR is non-optional. For US clients in healthcare, finance, or legal, frameworks like HIPAA and SOC 2 matter. For Gulf clients, the UAE Personal Data Protection Law and Saudi Arabia's PDPL have data residency rules that catch agencies off-guard.
Good answer: specific. "We deploy to your AWS or Azure tenant in your region. We don't train on your data. We use zero-data-retention API endpoints from OpenAI / Anthropic. We sign a DPA. Here's our standard one."
Bad answer: "We take privacy very seriously." That's a slogan, not an answer.
"Can I speak to two of your past clients this week?"
Why you're asking: testimonials on a website are theatre. A 20-minute call with a real past client is the single most useful evaluation you can do.
Good answer: intros within 48 hours, ideally to two clients in a similar industry or with a similar problem. No script for the reference. You ask whatever you want.
Bad answer: "We need to check with them first" that turns into a week of silence. Or one warm reference instead of two. Or a logo wall with no one you can call.
"What's the realistic timeline, and what would slow it down?"
Why you're asking: the second half of the question is the important one. Agencies that have shipped real projects know exactly what slows them down. Agencies that haven't will give you a confident timeline that they will miss.
Good answer: a realistic timeline plus an honest list of slowdown risks. "Eight weeks if you can give us clean training data in week one. Twelve weeks if you cannot. The biggest risk to your timeline is internal stakeholder review cycles — please tell us now who needs to sign off on what."
Bad answer: a confident timeline with no caveats. They will miss it.
"How will we measure success — and what numbers will we look at in 90 days?"
Why you're asking: if you cannot define success up front, you will never agree on whether the project worked. This is also a useful filter — agencies that think in outcomes will give you specific numbers; agencies that think in deliverables will list features.
Good answer: two or three measurable KPIs tied to your business. "Support tickets resolved without escalation: target 40% by week 8. Average handle time: down 30% by week 12. CSAT for AI-resolved tickets: above 4.2/5."
Bad answer: "We'll deliver a chatbot." That's a deliverable, not a success metric.
"What would make you turn this project down?"
Why you're asking: an agency that has never said no to a brief has no taste. You want the one with opinions about what is a good project and what is not.
Good answer: a specific, honest list. "We turn down projects where the client cannot name an internal champion. Where the underlying process is not documented. Where the budget is below what the scope actually requires. Where the executive sponsor has not used ChatGPT themselves at least a dozen times."
Bad answer: "We can find a way to make almost any AI project work." Translation: they are desperate.
"Who specifically will work on my project — and can I meet them?"
Why you're asking: the bait-and-switch is the oldest move in agency land. The senior pitches; the junior delivers. Sometimes that's fine if the senior actually reviews the work. Often it isn't.
Good answer: named people. "Your lead engineer will be Sara. She has shipped six RAG systems in the last 18 months. She will be on every weekly call. Here is her LinkedIn. Here is a 15-minute video of her explaining one of her projects."
Bad answer: "Our team is highly experienced and we will assign the right people." You will be assigned to whoever is least busy that week.
Six red flags that override any answer
Even if the agency answers all twelve questions perfectly, these six behaviours during evaluation should make you reconsider.
- ×Urgency tactics. "We can only hold this price until Friday." Real agencies don't run flash sales on six-figure engagements.
- ×Buzzword density. If the proposal mentions "transformative", "synergistic", or "AI-powered ecosystem" more than once, the work product will follow the same pattern.
- ×No written scope. "We'll figure out the details once we start." This is how every change-order disaster begins.
- ×Lopsided payment terms. 100% upfront. 80% upfront. No milestone-based payments. You should pay as work is delivered, not before.
- ×Always agreeing with you. If you propose something dumb and they don't push back, they will not push back when it matters either.
- ×No questions back at you. An agency that has not asked you ten hard questions by the second call has not started thinking about your project.
Bring all twelve questions to a 30-minute call. We'll answer every one — on the record.
Five green flags worth weighting heavily
Equally important. These signals do not appear on most agency websites. They show up in how the agency behaves during evaluation.
- ✓They tell you no. They turn down scope items, recommend smaller projects than you asked for, or suggest a phased approach. Confidence to lose the deal in service of a better outcome.
- ✓They send you to a past client unprompted. Before you even ask.
- ✓The engineer joins the sales call. Not the founder. Not the AE. The person who will actually build it.
- ✓They give you something useful for free. A loom video critiquing your current workflow. A scope outline before you sign discovery. An honest opinion you didn't pay for.
- ✓The proposal is shorter than you expected. Specific, honest, well-scoped. Long proposals usually hide weak thinking under volume.
What to send agencies in your initial outreach
The quality of the responses you get is a direct function of the quality of the brief you send. Generic emails get generic replies. Give them enough to think with and you will get serious answers.
A good first email includes:
- 1. The problem in one sentence. Not the solution. The problem.
- 2. Why it matters to the business. Time, money, customer experience.
- 3. What you have tried already. Off-the-shelf tools, internal builds, other agencies.
- 4. Rough budget range. Saying "$10k-$25k" filters out agencies that only work above $100k. Saving everyone time.
- 5. Timeline. When you want it live. Honest, not aspirational.
- 6. Who is the decision-maker. One name. If it is a committee of seven, say so up front.
- 7. Specific ask. "Reply with three things: rough scope, ballpark price, a question you would ask before quoting."
Send this to five agencies. The quality of the replies will tell you who to put on a call.
We reply to every brief within 48 hours.
Use the format above. You'll get rough scope, ballpark price, and one question we'd ask before quoting.
In one paragraph
Ask the twelve questions. Watch for the six red flags. Reward the five green flags. Send every shortlisted agency a real brief and judge their reply, not their website. Hire the agency that pushes back on bad ideas, names the people doing the work, owns the failure modes, and turns down the wrong project. That is the whole job.
Common questions
How many AI agencies should I shortlist?+
Five for the initial brief. Three for the full evaluation call. One to hire. Fewer than three and you have no comparison; more than three and you waste weeks evaluating people you will never pick.
Should I hire a local agency or a remote one?+
For AI work specifically, location matters less than communication discipline. A remote agency with a structured weekly cadence will out-deliver a local agency that "drops by." Time zone overlap of at least four working hours is the real requirement.
Should I pay a flat fee or hourly?+
Flat fee with milestone payments for the build. Monthly retainer for ongoing optimisation. Avoid hourly billing on builds — it punishes the agency for being efficient and you for being decisive.
What if the agency refuses to give me a fixed price?+
Fair, if it is genuinely a research-heavy or exploratory project. In that case insist on a paid discovery phase with a fixed price and a fixed deliverable (usually a scoped project document). Then take that scope and decide if you want to continue or take it to another agency.
Can I hire a freelancer instead of an agency?+
Yes, for narrow well-defined builds. A great freelancer can deliver a single automation or chatbot more cheaply than any agency. Where freelancers fall short is multi-skill projects (design + dev + ops + integrations) and ongoing reliability. If the freelancer goes on holiday, your system is on holiday too.
How do I know if the agency is overcharging me?+
Get three quotes for the same scope. If one is more than 2x the others for similar work, ask them to justify it. The justification might be valid (deeper experience, better engineering, lower failure rate) or it might be a markup. Either way, the question forces the answer.
Ready to put us on your shortlist?
Bring the twelve questions. We'll answer all of them on the record. If we're not the right fit, we'll tell you who is.
101 Digital Frames LLC · Florida, USA · Birmingham, UK · Muscat, Oman
