Quick Answer: What questions should I ask a machine learning consulting company before hiring?
Ask how the firm measures success and whether it will show a baseline comparison on your own data rather than only a headline accuracy number from a public benchmark. The core answer is simple: treat the hiring process like a technical audit, covering data provenance, privacy compliance, model documentation, ownership, monitoring, and who actually does the work.
Some firms sell strategy decks, while others build and deploy models; managed service providers sit in between. They differ in staffing, subcontracting, and how much they hand over at contract end.
Before signing, compare references, security certification scope, pilot terms, IP and weights ownership, drift monitoring plans, data processing agreements, and total cost including handover.
Why This Hiring Decision Deserves a Real Checklist

Would you hand your customer database to a team you have never audited? Most companies effectively do exactly that when they hire a machine learning consultancy, and often with less scrutiny than they apply to a new office cleaner.
The stakes are concrete. Models trained on personal data fall under privacy law. Trained weights and pipelines can become core business assets. And a vendor’s impressive demo may rest on a benchmark that looks nothing like your actual data. The questions below are designed to surface those risks before money changes hands, not after the kickoff meeting.
One practical tip from procurement teams: send the questions in writing before the first call. How a firm responds, and how fast, tells you almost as much as the answers themselves. Vague replies to specific questions are a signal.
Questions About Data, Licensing, and Provenance
Where did the training data come from, and under what license?
Public datasets frequently carry restrictive terms. ImageNet, for instance, has historically limited commercial use, and that debate has run publicly since at least 2021. Ask which datasets were used, under which license, and request the current terms in writing, since licenses change.
Will our data leave the country or the vendor’s premises?
Offshoring is common and not automatically bad, but you need to know. Ask where data is stored, which subcontractors touch it, and whether any of it feeds the vendor’s own product development.
Who acts as controller and who as processor?
Under the GDPR, in force since 25 May 2018, the distinction determines legal responsibility for personal data, including data used in model training. In the US, sector rules vary, such as HIPAA for health data, enacted in 1996. Ask whether a data processing agreement is in place and get it reviewed.
Questions About Evaluation and Evidence
This is where most buyers get burned, and where a little skepticism pays.
What is the baseline, and can you beat it on our data?
An accuracy figure means little without a comparison to a simple baseline or your current process. Ask for a held-out evaluation on representative client data, plus confidence intervals or an error analysis. A vendor whose headline metric comes from a public benchmark that differs from your distribution has told you very little.
What does “success” mean in the contract?
Push for measurable criteria: error rates on defined segments, latency targets, cost per prediction. If success is defined as “model delivered,” you have bought a deliverable, not an outcome.
Will you document the model?
Model cards, introduced in a 2019 paper by Mitchell and colleagues, and datasheets for datasets, published by Gebru and co-authors (first version 2018, revised 2021), are widely cited templates for documenting intended use, limitations, and training data. Ask whether these are contractual deliverables. If a firm has never heard of them, that is informative.
Questions About Ownership, Handover, and Exit
Who owns the model, the code, and the trained weights?
There is no universal default here; it depends on jurisdiction and contract. Ask explicitly about source code, trained weights, and the data. “We’ll host it for you” is a business model, not an answer.
What happens at contract end?
Ask for terms on data return and deletion, plus a handover plan. A model you cannot run without the vendor gives the vendor the upper hand; it is not an asset.
Can your own team maintain it?
Ask what training and documentation the vendor provides. If the answer is a two-hour walkthrough, plan for dependency.
Questions About Security, Team, and Process
Do you hold a current ISO/IEC 27001 certificate, and what is its scope?
ISO/IEC 27001 is the widely recognized information security management standard, with the current edition dated 2022. Ask for the certificate and the scope statement, because a certificate covering one office says nothing about the team handling your data.
Who actually does the work?
Subcontracting and junior staffing are common. Ask for named personnel, their roles, and whether work is offshored. Then check that those names appear in the contract.
How will you monitor the model after launch?
Models degrade as data changes, a phenomenon called concept drift. Ask what monitoring is proposed, who is responsible, and what triggers a retrain. NIST’s AI Risk Management Framework, version 1.0 released in January 2023, offers a voluntary structure for ongoing risk management across the lifecycle; a serious vendor should be able to map its monitoring plan onto something similar.
Can we start with a paid pilot?
A small, time-boxed pilot with defined success criteria reduces risk more than any reference call. Insist on written criteria before it starts.
If your project also touches conversational interfaces, the same scrutiny applies to chatbot vendors; the questions in this guide to AI chatbot security overlap heavily with the ones above. Buyers comparing security operations providers will find a similar structure in this managed detection response checklist, which is worth reading for how to phrase vendor questions that get real answers.
Red Flags and Common Mistakes
Buyers make the same errors repeatedly. Here are the ones that show up most.
Mistake one: accepting a benchmark number as proof. A model scoring well on a public leaderboard may fail on your distribution. Always ask for a test on your data.
Mistake two: skipping the baseline. Without a comparison to your current process, you cannot tell whether the model adds anything.
Mistake three: leaving ownership vague. “We’ll sort out IP later” almost always favors the vendor.
Mistake four: ignoring the handover. Ask what you receive on day one of the exit, not day one of the project.
Mistake five: treating the pilot as a formality. Define success criteria in writing, including what “no-go” looks like.
A useful habit: score each vendor’s written answers on a simple grid, one row per question, one column per firm. It forces consistency and exposes the firms that answer in paragraphs of nothing. The same discipline applies when you are evaluating hardware purchases; this breakdown of what money counting machine buyers get wrong shows how specification gaps cause expensive surprises in a completely different category.
How the Main Vendor Types Compare
Not every firm does the same job, and the questions shift depending on which type you are talking to.
| Vendor type | What they typically deliver | Question to press hardest |
|---|---|---|
| Strategy consultancy | Roadmaps, feasibility studies, business cases | Who builds it, and what happens after the deck? |
| Build shop | Custom models, pipelines, deployment | Do we own the code and weights? |
| Managed service | Hosted model with ongoing monitoring | What is the exit plan and data deletion process? |
| Staff augmentation | Named engineers working in your team | Who manages them, and what is the knowledge transfer? |
| Product vendor with services | Their platform plus configuration | How much is custom versus configuration? |
The distinctions matter because a strategy firm and a build shop answer the ownership question very differently. Knowing which type you are talking to prevents the classic mismatch where a buyer expects a deployed model and receives a slide deck.
What a Good Answer Looks Like
Vendors that have been through real audits answer differently. They volunteer the baseline. They mention the model card before you ask. They have a standard data processing agreement ready. They name the people who will do the work and put them in the contract. They describe a monitoring plan with thresholds and owners.
Weak answers share a pattern too: confident adjectives, no numbers. “Industry-leading accuracy” without a baseline is marketing. “State-of-the-art” without a held-out test is a claim, not evidence. Ask a follow-up question every time an answer contains an adjective instead of a figure, and watch how the room reacts.
It also helps to ask what went wrong on a previous project. Firms that answer honestly, describing a failed deployment and what changed afterward, tend to be the ones worth hiring. Firms that claim a spotless record are either new or not telling you everything.
Budgeting and Commercial Terms
Ask how the vendor prices: fixed fee, time and materials, or outcome-based. Each shifts risk. Fixed fee protects you on scope but invites corner-cutting on evaluation. Outcome-based contracts sound attractive until the definition of “outcome” is negotiated, so get it in writing with thresholds and measurement methods.
Ask also about the cost of running the model after launch. Cloud inference bills, monitoring tools, and retraining cycles are recurring costs that rarely appear in the proposal. A model that costs more to run than the process it replaced is a loss, however accurate it is.
Finally, ask about insurance and liability. If the model makes a bad decision that costs you money, who pays? The answer belongs in the contract, not in a conversation.
Conclusion
Hiring a machine learning consultancy is a technical decision wrapped in a commercial one, and the questions above cover both halves. The pattern is consistent: ask for evidence rather than adjectives, ask for written terms rather than verbal assurances, and ask who owns what before the first invoice rather than after the last. Buyers who send a written question list, score the answers, and insist on a paid pilot with defined criteria consistently avoid the expensive surprises that plague this market. The firms worth hiring will welcome the scrutiny; the ones that dodge it have told you what you needed to know.
Frequently Asked Questions
How many questions should I actually ask a vendor?
Ten to fifteen written questions is a workable number. Fewer and you miss material risks; more and vendors start sending generic responses. Group them by data, evaluation, ownership, security, and process, then score the answers side by side.
Is a paid pilot always necessary?
Not always, but it is the cheapest way to test a vendor’s claims. If a firm refuses even a small, time-boxed pilot with defined criteria, treat that as a warning. Pilots also reveal how a team communicates when something goes wrong, which no reference call will show you.
What if the vendor will not share its training data?
Many vendors legitimately cannot share proprietary datasets. That does not excuse silence on licensing. Ask for the license terms, the dataset documentation, and confirmation that commercial use is permitted. A datasheet for the dataset is a reasonable request even when the data itself stays private.
How do I check a security certification claim?
Ask for the certificate itself and the scope statement, then confirm the certificate is current. A certificate covering a single office or a narrow service says little about the team handling your data. Cross-check the scope against where your project will actually run.
Should I insist on owning the trained model weights?
It depends on your bargaining position and the vendor’s business model. Owning weights matters most when the model is core to your product. If the vendor hosts the model as a service, focus instead on exit terms, data deletion, and a documented way to migrate. Either way, get the answer in writing.
How do I compare two vendors that both look strong?
Score them on the same grid: baseline evidence, documentation deliverables, ownership terms, security scope, named personnel, monitoring plan, and pilot terms. Weight the criteria by what would hurt most if it went wrong, and let the totals decide rather than the quality of the presentation.


