Quick Answer: What questions should I ask a generative AI development company before hiring?
Ask a generative AI development company about regulatory exposure, data handling, security testing, model provenance, evaluation methods, and contract terms before you sign anything. These six areas decide whether a project stays compliant and affordable or turns into an expensive cleanup job six months later.
The questions split into two groups. Some are about the technology itself, such as which base models the vendor uses and how it measures accuracy and hallucination rates. Others are about the business relationship, such as who owns the outputs, what happens when a model is deprecated, and how incidents get disclosed.
Before hiring, compare regulatory role and classification, data processing location and retention, training-data use, security certifications, red-team results, evaluation methodology, model versioning, indemnities, service levels, and exit terms.
Why These Questions Matter More Than a Portfolio

A slick demo proves very little. Generative AI systems fail in ways that slide decks never show: a prompt injection that leaks customer records, a model quietly retired by its provider, or an accuracy claim that evaporates on real data. The gap between a prototype and a production system is where budgets die.
The regulatory clock is also running. The EU AI Act entered into force on 1 August 2024 and applies in phases: prohibitions from 2 February 2025, general-purpose AI model obligations from 2 August 2025, and most high-risk obligations from 2 August 2026. A Commission proposal floated in November 2025 could delay some high-risk deadlines, so confirm the current status before relying on that date. Either way, a vendor who cannot explain which category your use case falls into is a risk.
If your project touches customer-facing chat, the buying checklist for AI chatbot security in 2027 covers nine related questions that overlap heavily with what follows. This article goes wider, covering the whole engagement rather than just the bot.
Regulatory and Compliance Questions
Which EU AI Act category does our use case fall into?
Ask the vendor to walk through Annex III of the regulation and state plainly whether your system is high-risk. Then ask who holds the provider role and who holds the deployer role. Those roles carry different obligations, and vendors often blur them to sound simpler than the law is.
Can you map your controls to a recognised framework?
The NIST AI Risk Management Framework, released in January 2023, organises risk work into four functions: Govern, Map, Measure, and Manage. A Generative AI Profile followed in July 2024. Separately, ISO/IEC 42001, published in December 2023, sets out requirements for an AI management system. A serious vendor should be able to point to one of these and show how its processes fit, rather than only naming the standard.
Who handles data protection, and where?
Under GDPR, which has applied since May 2018, and UK GDPR, personal data inside prompts, fine-tuning sets, and logs is still personal data. Ask where processing happens, whether a Data Processing Agreement is in place, how long logs are retained, and whether any data crosses borders. Vague answers here are a red flag, not a detail.
Data, Security, and Model Questions
Is our data used to train models?
This is the question vendors dodge most often. Get it in writing: is customer data used for training by default, can that be switched off, and does the answer change for the vendor’s own models versus third-party APIs?
How do you test for the OWASP LLM risks?
The OWASP Top 10 for LLM Applications, in its 2025 edition, lists prompt injection, sensitive information disclosure, supply-chain vulnerabilities, data and model poisoning, and excessive agency, among others. Ask how the vendor tests for each one and whether it will share red-team results. A vendor that has never heard of excessive agency has probably never scoped tool permissions properly.
Which base models do you use, and can you pin versions?
Ask which models and versions sit under the hood, whether versions can be pinned, and what happens when an upstream provider deprecates a model. Providers publish migration notices, but dates shift constantly. The real question is whether the vendor has a versioning and fallback plan already written down.
What are your security certifications?
SOC 2 Type II and ISO/IEC 27001, the information-security management standard whose 2022 revision is current, are the baseline. Ask about penetration testing cadence too. If the vendor handles regulated data, a managed detection and response checklist of twelve questions before buying will help you judge whether monitoring is genuinely covered or just mentioned in a slide.
Evaluation, Accuracy, and Delivery Questions
How do you measure accuracy and hallucination rates?
There is no universal benchmark, so treat every accuracy percentage as unverified until you see the test set and method. Ask about golden datasets, human review, and automated evaluations. Then ask for error rates on tasks that resemble yours.
What does the evaluation plan look like before launch?
A good answer describes a baseline, a test set built from your own data, and a threshold for go-live. A weak answer describes a demo.
How will you handle model drift and cost creep?
Token costs, retries, and human review all add up. Ask for a cost model at realistic volume, not at pilot volume, and ask who monitors quality after launch.
What happens at handover?
Ask for documentation, prompt and configuration files, evaluation scripts, and the right to keep using what you paid for. Exit terms belong in the first conversation, not the last.
Contract and Commercial Questions
Who owns the outputs, and what indemnities apply?
Ask who owns generated outputs, what indemnity is offered for IP infringement, and whether any upstream copyright protection passes through to you. Several foundation-model providers offer indemnities to business customers, but terms vary and change, so verify current wording rather than trusting a summary.
What service levels and incident disclosure terms apply?
Uptime, latency, and support response times should be explicit. So should incident disclosure: how fast you hear about a breach, and through which channel.
How do you price the work?
| Pricing model | What it suits | What to watch |
|---|---|---|
| Fixed scope | Well-defined pilots | Change requests inflate fast |
| Time and materials | Exploratory builds | No ceiling without a cap |
| Retainer plus usage | Ongoing operations | Token costs can drift |
| Outcome-based | Mature, measurable tasks | Hard to define success |
Ask which model the vendor prefers and why. A vendor that only offers one should explain what happens when scope shifts.
Practical Tips and Common Mistakes
The most common mistake is treating the demo as evidence. A second is skipping the data processing conversation because it feels like paperwork. A third is signing without a model deprecation clause.
In practice, three habits separate smooth engagements from painful ones. First, ask for a written answer to every question above before the contract, even if the answer is “we do not do that yet.” Second, run a small paid evaluation on your own data before committing to a build. Third, name one person on each side who owns the compliance questions, so nothing gets lost between legal and engineering. (Managed detection response checklist 12 questions ask before)
Buyers researching adjacent purchases often apply the same discipline elsewhere. The approach in this Rillabox review for buyers before ordering, or in this guide to Fidelity money market funds for savers before committing, is the same: verify claims, read the fine print, and compare terms rather than marketing.
Red Flags Worth Walking Away From
Some answers should end the conversation. A vendor that cannot say whether your data trains models, that dismisses the EU AI Act as irrelevant, or that offers accuracy claims with no test set is telling you something useful. So is one that has no answer on model versioning, or that treats security testing as a phase-two item.
Less obvious red flags include refusing to name subcontractors, quoting a price with no usage assumptions, and promising a launch date before seeing your data. None of these are automatically fatal, but each one deserves a direct follow-up. If the follow-up is vague too, that is your answer.
Conclusion
Hiring a generative AI development company is mostly an exercise in asking better questions. The technology will change, models will be retired, and regulations will keep shifting, but the fundamentals hold: know your regulatory category, control your data, test for known attack patterns, demand evidence for accuracy claims, and settle ownership and exit terms before work starts. Vendors who welcome those questions are usually the ones worth hiring. Vendors who dodge them are saving you time by showing you the door early.
FAQ
How many vendors should I evaluate before choosing one?
Three is a sensible minimum. Fewer than that and you cannot tell whether a quote reflects the market or just one team’s overhead. Compare the same written questions across all three so the answers are actually comparable.
Is AI management system certification necessary?
Certification is not necessary, but it is useful. ISO/IEC 42001 certification signals that a vendor has documented how it governs AI risk. A smaller vendor without certification can still be a good choice if it can show equivalent policies and evidence.
How do I verify an accuracy claim?
Ask for the test set, the scoring method, and the error rate on tasks similar to yours. Then run a small evaluation on your own data. If the vendor will not share the method, treat the number as marketing.
What should a pilot cost?
Costs vary widely by scope, so there is no reliable benchmark. What matters is that the pilot has a defined question, a fixed budget, and an agreement on who keeps the code and data afterwards.
How do I protect against a model being retired?
Put it in the contract. Ask for pinned versions, a documented fallback model, and a notice period before any upstream change affects your system. Test the fallback before you need it.
Does EU AI Act compliance apply if my company is outside Europe?
It can, if your system’s output is used within the EU or affects people there. Ask the vendor to explain the territorial scope for your specific case rather than assuming geography settles it.
Who is liable when something goes wrong?
Ask how liability is allocated when a model produces harmful or infringing output, and whether that allocation is capped. Vendors often limit liability to fees paid, which can be trivial against the harm a bad output causes. Get the cap and the carve-outs in writing.
What does onboarding look like in the first thirty days?
A credible vendor should describe a structured start: access provisioning, a kickoff with named roles, a data-handling checklist, and a first milestone you can inspect. If onboarding is described only as “we’ll figure it out together,” you are the one absorbing the risk.
Can you show a reference for a use case like ours?
Ask for two references in a comparable sector or regulatory position, and ask the vendor to explain what went wrong on those projects. Every real engagement has friction points; a vendor that reports none is either new to the work or not being candid.
Final Word
None of these questions requires specialist knowledge to ask, only the discipline to ask them before signing. Write them down, send them in advance, and compare answers side by side. The vendor that answers plainly, flags what it does not yet do, and puts commitments in the contract is the one to hire. The rest are telling you, politely, that the hard conversations will come later, at your expense.


