How to choose a software development agency: the questions your vendor dreads

We sell software development for a living. German-led project leadership, a senior engineering team in Pakistan, clients mostly in Europe. So when we tell you how to vet a software development agency, we're describing a process we sit through from the other side of the table several times a month.

That gives us an unfair vantage point. We know which questions make an agency's sales lead go quiet, and we know what we'd do if the roles flipped and we had to hire someone like us.

This is that playbook. It's opinionated, because vetting is opinionated work. Where we'd disqualify ourselves, we say so.

Start with the portfolio test (read it right)

Every agency has a portfolio page. Almost nobody reads one correctly.

The mistake is treating it as a gallery. Screenshots of polished apps tell you the agency once had a client with a budget and a designer. They say nothing about delivery. A portfolio is evidence, and evidence has to be cross-examined.

Here's how we'd read ours. Our case studies page lists five projects at the moment. If you were vetting us, the right move is not to admire them. Pick one and ask: which parts did you actually build? What was already there when you arrived? Who made the architecture decisions, your team or the client's? What broke after launch, and who fixed it?

Two things to watch for in any portfolio. First, whether the projects are live. An agency whose case studies are all Dribbble-grade mockups and expired domains has a design team, not a delivery team. Second, whether the work is boring anywhere. Polished consumer apps are the easy part of this industry. The agencies that can deliver have unglamorous entries: an internal tool, a migration, an integration with some ERP system from 2009, a rescue of somebody else's abandoned codebase. Boring work is where delivery discipline shows.

A portfolio with nothing boring in it is a marketing artifact. Treat it like one.

The 5 questions that expose delivery maturity

Portfolios get you to a shortlist. Questions get you past the sales layer. These five are the ones we dread from good buyers, and we dread them precisely because they can't be answered with marketing.

1. Who writes the first pull request, and when?

This is the single most revealing question in agency vetting. It asks whether the senior people from the pitch actually touch code, and how soon.

The answer you want names a person and a week: "Your first PR comes from the tech lead you met on the call, inside week one." The answer you don't want is a process description. "Our teams follow an agile onboarding methodology" means the senior people you met are already on the next pitch, and your codebase is about to become a training ground for juniors.

When buyers ask us, we name the engineer. If an agency can't, ask why not.

2. Show me a project that failed. What happened?

An agency that can't show you a failure usually hasn't been around long enough, or isn't being straight with you. Software projects regularly go sideways; vendors without failure stories tend to be vendors without much history.

We've had projects go sideways. On a 2024 build, scope grew 40% because nobody forced the hard prioritization conversation at kickoff. On another, the client's internal stakeholder left mid-build and took all the product knowledge with him. What you want to hear is the postmortem: what the agency misjudged, and what it changed afterward. An agency that describes failure as entirely the client's fault will describe your project the same way when it slips.

3. How do you handle scope change mid-sprint?

Every buyer asks about the price. Almost nobody asks about the moment the price changes.

Scope change is where agency relationships go to die, and the answer reveals the whole commercial character of the vendor. A good answer has mechanics: a written change-order process and an estimate before any extra work starts. A bad answer is "we're flexible." Flexible means the invoice gets flexible later.

Ask to see the last three change orders they issued. Not the policy, the artifacts. An agency with a real process can produce them within a day.

4. Who owns the repo, the CI pipeline, and the credentials on day one?

The only acceptable answer: you do, from the first commit, inside your organization's accounts.

Agencies that host your code in their GitHub org, run builds on their infrastructure, and hold your cloud credentials "for convenience" are building switching costs you'll discover at the worst possible moment. We've taken over projects where the previous vendor held everything; in one handover we ran in 2025, the negotiation cost the client more than a month of development. Ownership belongs in the contract as a non-negotiable, not granted as a favor.

5. What does your estimate assume, and where does it usually break?

Every estimate is a pile of assumptions wearing a number. A mature agency can tell you what the pile is made of: the quality of the third-party APIs involved, and how fast your own team answers questions.

The follow-up is the real test: where do your estimates usually break? Ours break on integrations with undocumented systems and on stakeholder availability, and we say so in our proposals. An agency that claims its estimates are accurate across the board has never measured them.

Proposal red flags

By the time you hold two or three proposals, vetting is mostly pattern recognition. Six patterns we'd treat as disqualifying:

1. No named people. "A senior cross-functional team" with no names or CVs attached means the staffing decision happens after you sign. Ask who, specifically, and get it written into the contract.
2. One number, no breakdown. A total with no phase split and no assumptions section is a number you can't compare against anything. Uncomparable quotes are the point of the exercise.
3. A suspiciously round timeline. "Three months" with no week-level plan means nobody estimated; someone negotiated. Real estimates come out lumpy: nine weeks, eleven weeks.
4. No exclusions section. A proposal that lists everything included and nothing excluded is hiding the boundary. The exclusions are where the change orders live.
5. Silence on warranty and maintenance. If the proposal ends at launch, so does the agency's interest in your code quality. Ask what happens when something breaks in week six.
6. Zero pushback. A proposal that accepts your requirements document point for point, questions nothing, and challenges no assumption was written by a sales team planning to win the deal first and discover the problems later. Good agencies argue with you before you sign. It's the cheapest consulting you'll ever get.

Reference calls: what to actually ask

Most reference calls are useless because buyers ask the question the reference was booked to answer: "Were you happy?" Nobody puts an unhappy client on a reference list.

Ask questions with edges. "What went wrong, and how did they handle it?" "Did the team that pitched you stay on the project?" "How did the final invoice compare to the estimate, and where did it move?" "Would you sign the same contract again, unchanged?" That last one is the closest thing to a lie detector we know. People who'd renew with amendments will tell you the amendments.

Two upgrades most buyers skip. First, ask the agency for a reference from a client who left. How a vendor treats a departing client tells you more than how they treat a happy one. Second, check independent review platforms like Clutch alongside the provided references, not because the reviews are unbiased, but because negative reviews are often the most revealing material you'll get.

The trial-sprint option

Everything above is inference. A trial sprint is evidence.

The structure: a fixed-scope, paid engagement of about four weeks, sized to produce something real but small enough to abandon cheaply. We offer exactly this as our MVP sprint package, and we tell buyers to treat it as an audition rather than a deliverable. The working software is almost secondary. What you're buying is four weeks of observation.

Judge the things a proposal can't show you. Did the first PR arrive when promised? Were standups status theater or actual decisions? What happened when you changed a requirement mid-sprint, which you should do deliberately, once, just to watch the change-order machinery engage? If the estimate slipped, how early did they tell you?

An agency that refuses any form of paid trial is telling you its sales process converts better than its delivery. Believe it.

Price signals: what cheap and expensive each hide

In the proposals founders bring us for a second opinion, quoted rates run from $15 an hour to $250, and the number alone tells you less than buyers think. Both ends of that range are hiding something.

Cheap hides staffing. A $20-an-hour quote is a junior team with no QA function and an estimate built to win rather than to hold. The real cost arrives later, in rework and in the second agency you hire to finish the job. We've been that second agency. The rescue always costs more than the original build would have.

Expensive hides overhead. Past a certain point you're paying for account managers and the brand's last famous client. The code doesn't get proportionally better.

Our own band is $25–49 an hour, and it works because of the structure: German-led project leadership, senior engineers in Pakistan, delivery in English and German. It's the right choice for a lot of builds and the wrong one for some. If you need forty engineers starting next month, you need a bigger firm. If you need people on-site in Munich every day, you need a local one. If you want a partner who takes equity and shares product risk, that's a different contract than we sign. If your industry has a compliance regime you need the team to already know cold, a specialist beats us. A good agency tells you when it's the wrong fit. Ours just did.

If you're mid-selection and want to compare notes, look at what we actually build, or skip the reading and put us through the process above. We'd rather be vetted hard than chosen casually. The vetting is the preview of the relationship.