What "AI-first" actually means when you hire a development agency

A year ago, "we use AI" on an agency website was a differentiator. Now it is wallpaper. Everyone says it, which means it tells you nothing, which means you need better questions.

I run a small agency that works AI-first. Every developer here builds with a model in the loop, all day. So I am not writing this to talk anyone out of it. I am writing it because I have seen what the bad version looks like, and I would rather you did not pay for it.

The bad version

The bad version is a developer pasting a ticket into a chat window, pasting the answer into the codebase, and running it to see if it works. Sometimes it does. The code is a bit longer than it needs to be, the variable names are fine, there is a test that passes because it does not test anything, and nobody on the team could explain why it is structured the way it is.

Do that for six months and you have a codebase nobody understands, built by a model that has no memory of building it. That is the slop everyone is worried about, and the worry is fair.

The good version

The good version looks almost boring from the outside. A senior developer decides what to build and how it should be shaped. They write the test first. They give the model a small, precise brief and a budget. They read what comes back, properly, the way you would read a junior's pull request. They push back, twice if needed. Then they put their name on it and it ships.

The output is code a senior developer would have written, produced in a fraction of the time. That is the whole trick. AI does not lower the bar. It lets a person who already clears the bar do it more often.

Five questions to ask

You do not need to be technical to ask these. You need to listen for whether the answer is specific.

1. Who reviews the code the AI produces, and how?

The answer should name a role and a process. "A senior developer reads every change before it merges" is good. "We have tooling for that" is not, because tooling is exactly what produced the code in the first place.

2. Do you write tests before or after?

Before is the right answer. A model asked to make a real test pass has far less room to be creative with the truth. After means the test was written to match whatever came out, which proves very little.

3. Who pays for the tooling, and how is it priced?

There are two answers you want. The developer tooling should be a flat cost the agency carries, not a metered line that grows the more they use it. And if the product they build for you uses AI, ask how they keep the running cost down: what needs a model call, what gets worked out once and stored, what gets cached. If the answer is a shrug, the bill will be a surprise later, and it will be yours.

4. Where does my code go?

You want named providers, commercial terms that exclude training on your data, and a sub-processor list your security team can read. "It's all encrypted" is a non-answer.

5. Who is accountable when it breaks?

The right answer is a person's name. Every change that ships should have a human approver in the commit history, and someone you can ring. If the answer starts with "the model", walk away.

What you should expect to see

If an agency is genuinely good with AI, you should notice three things. Delivery is faster than you are used to, without the quality dipping. The developers can explain any part of the codebase, because they read all of it. And the bill for the tooling is a known, boring number that arrives with an explanation.

If you see two of those but not the third, ask about the third. That is usually where the surprises live.

A last, honest thing

Being good with AI is mostly knowing when not to use it. Architecture, data models, anything touching money or personal data, final copy, final artwork: those stay with people at our place, and I would be nervous about any agency where they do not.

The tools are extraordinary. They are also confident when they are wrong. Hire the people who can tell the difference.