Quick Answer: What questions should I ask a conversion rate optimization company before hiring?
You should ask a conversion rate optimization (CRO) company how it designs experiments, how it calculates sample size and significance, what it measures, and how it reports failed tests. The core answer is simple: treat the interview like a methods review, not a sales pitch, and press for specifics on statistics, data ownership, tooling, privacy compliance, and contract exit terms.
Some firms sell CRO as a design refresh with a lift percentage attached, while others run disciplined hypothesis testing with documented methodology. The difference matters more than the pitch deck.
Before signing, compare testing methodology, sample-size rules, conversion definitions, reporting cadence, data and account ownership, privacy and consent handling, tooling compatibility, pricing model, minimum term, and exit clauses.
Why the Questions Matter More Than the Portfolio

CRO looks deceptively simple from the outside. You change a button color, the number goes up, everyone celebrates. In practice, most of the difficulty hides in the statistics and the process. An agency that skips the hard parts can still produce beautiful slide decks full of green arrows.
The academic literature on online controlled experiments is blunt about the traps. Peeking at results before a test reaches its planned sample size, stopping early when a result looks good, and running underpowered tests all inflate false positives. A vendor that does any of these will eventually report a “win” that evaporates the moment you ship it.
So the interview is really a filter. Good questions separate teams that run experiments from teams that run opinions with a dashboard attached.
Questions About Testing Methodology and Statistics
How do you decide sample size before a test starts?
This is the single most revealing question. A rigorous team calculates the required sample size in advance based on baseline conversion rate, the minimum detectable effect it cares about, and statistical power. A weaker team runs traffic for two weeks and calls whatever happens a result.
Listen for terms like statistical power, minimum detectable effect, and confidence intervals. If the answer is vague, ask them to walk through a recent test calculation step by step.
What is your rule on stopping a test early?
Early stopping is where good intentions meet bad incentives. Ask directly: if a variant is winning by 30% on day three, do you stop? The right answer involves a pre-committed duration or a formal sequential testing method, not a judgment call made in a Slack channel.
How do you handle failed tests and null results?
An agency that only shows winners is showing you a highlight reel. Ask what percentage of their tests produce flat or negative results, and how those get reported. Willingness to publish null results signals scientific honesty. Reluctance usually means the reporting is curated for renewal meetings.
Do you correct for multiple comparisons?
Run enough variants and something will look significant by chance. Ask whether they adjust for this, and how they limit the number of simultaneous tests per page. You do not need a statistics lecture in response, but you do need evidence the question has been considered. (Managed detection response checklist 12 questions ask before)
Can you show a case study with the baseline, sample size, and duration?
Lift percentages without context are marketing. A credible case study states the starting conversion rate, the traffic volume, the test duration, and whether the result replicated in a follow-up. Ask for one and read it closely.
Questions About Measurement and Definitions
How do you define conversion for our business?
Definitions differ. A micro conversion (newsletter signup) and a macro conversion (completed purchase) measure different outcomes. Ask the agency to tie metrics to revenue or another business outcome rather than click-through rate alone.
What does your measurement plan look like before any test runs?
A serious engagement starts with analytics auditing, funnel mapping, and a prioritized hypothesis backlog. If the first deliverable is a redesign mockup rather than a measurement plan, the process is backwards.
How do you separate real lift from seasonality and external noise?
Ask how they use holdout groups, control segments, or long-term holdback experiments. This is especially important for businesses with strong seasonal swings, where a November test can look brilliant purely because of holiday traffic.
Questions About Data, Tooling, and Ownership
Who owns the test accounts, data, and code?
This question prevents the most common post-breakup headache. Ask explicitly whether accounts sit under your billing and admin control, whether custom code is handed over, and what happens to historical test data if you leave. Get it in the contract, not in an email.
Which testing and analytics platforms do you work with?
Tooling compatibility shapes cost and speed. Ask which platforms they use, what access levels they need, and whether their setup requires engineering time from your team for each test. If every experiment needs a developer sprint, velocity will suffer.
What access do you need, and how is it secured?
Vendors often request admin rights across analytics, tag managers, and CMS. Ask for the minimum viable access, how credentials are stored, and whether activity is logged. If security review matters in your organization, the same discipline you would apply when vetting any software vendor applies here: least privilege, clear ownership, and documented offboarding.
How do you handle consent and privacy in tests?
Any test involving tracking, cookies, or personalization touches privacy law. In the EU, the GDPR has applied since May 25, 2018. In California, the CCPA took effect on January 1, 2020. Ask how the agency handles consent signals, data minimization, and whether test variants ever change what a user consented to. A vendor that shrugs at this question is a compliance risk.
Questions About Velocity, Reporting, and Communication
How many tests per month, and what is realistic for our traffic?
Test velocity depends on traffic volume. A site with 5,000 monthly visitors cannot support the same cadence as one with 500,000. Ask for a realistic estimate and a plan for what happens when traffic is too low for reliable A/B tests, such as qualitative research or sequential rollouts.
What does a monthly report contain?
Ask to see a redacted sample report. Look for hypothesis statements, sample sizes, confidence levels, decisions made, and learnings captured. A report full of screenshots and arrows but no methodology is a warning sign.
Who actually does the work?
Sales calls often feature senior strategists; delivery may fall to junior staff. Ask who will run your account day to day, what their experience is, and how much senior review happens. This is the same due diligence discipline buyers apply when they read a vendor checklist of questions to ask before signing: the person in the pitch is not always the person doing the work.
How do you prioritize which tests to run?
Listen for a scoring model, such as ICE or PXL, combined with qualitative research like session recordings, surveys, and user testing. Prioritization based purely on gut feel or “best practices” lists tends to produce low-value tests.
Questions About Contracts, Pricing, and Exit
What is the pricing model, and what incentives does it create?
Retainers, project fees, and performance-based pricing each shape behavior. Performance-only deals sound attractive but can encourage gaming metrics, such as optimizing a micro conversion that never turns into revenue. Ask how the agency guards against that.
What is the minimum term and the exit clause?
Ask about notice periods, minimum commitments, and what happens to in-flight tests if you leave mid-cycle. A fair contract lets you exit with your data and a documented handover.
What happens during onboarding?
A good onboarding includes analytics validation, historical data review, stakeholder interviews, and a hypothesis backlog. Ask how long it takes before the first test goes live. Long ramp-ups are fine if the work is real; they are a problem if the delay is unexplained. (Rillabox review buyers should know before ordering)
Comparing Agency Types at a Glance
| Agency type | Typical strength | Typical weakness | Question that exposes it |
|---|---|---|---|
| Full-service digital agency | Broad channel integration | CRO as a side offering | Who leads experiment design? |
| Specialist CRO shop | Deep testing methodology | Narrower scope | Show a null-result report |
| Freelance consultant | Cost and flexibility | Limited capacity | What is your backup plan? |
| In-house team | Context and speed | Bandwidth constraints | How do you handle peak volume? |
Use the table as a starting frame, then let the answers to the questions above do the real sorting.
Practical Tips and Common Mistakes
The most common mistake is treating the sales deck’s lift numbers as evidence. Ask for the underlying methodology instead, and be suspicious of any figure presented without a baseline.
A second mistake is skipping the analytics audit. If your tracking is broken, every test result is suspect, and no agency can fix that with clever design. Insist on a validation step before testing begins.
A third mistake is ignoring the handover clause. Teams change, budgets shift, and agencies part ways. The buyer who owns the accounts and the code walks away cleanly; the one who does not spends months reconstructing history.
Finally, watch for the “best practices” pitch. Button colors and hero images are fine starting points, but real gains usually come from offer clarity, pricing presentation, form friction, and page speed. Buyers who vet vendors carefully in other categories already know the pattern: specifics beat slogans.
Conclusion
Hiring a CRO company is a methods decision disguised as a vendor decision. The questions above cover the ground that matters: how experiments are designed, how results are judged, who owns the data, how privacy is handled, and what happens when the relationship ends. Ask them in the first call, ask for evidence in the second, and put the answers in the contract. A team that welcomes the scrutiny is the team worth hiring.
Frequently Asked Questions
How many questions should I ask in the first call?
Aim for five to seven core questions in the first conversation, focused on methodology, measurement, and ownership. Save contract and pricing details for a second call once you know the approach fits. Too many questions at once turns the call into an interrogation and hides the important answers.
Is a performance-based pricing model a good idea?
It can align incentives, but it carries risk. When pay depends on a metric, vendors may optimize the easiest metric rather than the most valuable one. If you choose performance pricing, define the metric precisely, tie it to revenue, and include a baseline and audit clause.
How much traffic do I need for reliable A/B testing?
It depends on your baseline conversion rate and the size of the effect you want to detect. Low-traffic sites often need several weeks per test, which makes velocity claims worth scrutinizing. Ask the agency to calculate a realistic sample size for your actual traffic before promising a test count.
Should the agency have access to our analytics admin account?
Grant the minimum access needed to do the work, and prefer accounts you own with roles assigned to the vendor. Admin access across every tool is rarely necessary. Review permissions quarterly and revoke them at offboarding.
What signals a weak CRO proposal?
Vague answers about sample size, case studies with lift percentages but no baseline, no mention of null results, and a first deliverable that is a redesign rather than a measurement plan. Any one of these is worth a follow-up question; several together suggest a different service is being sold.
Can we run CRO in-house instead?
Yes, and many teams do. In-house CRO offers context and speed but requires statistical literacy, tooling, and engineering support. A common middle path is hiring an agency to build the process and train the team, then bringing testing in-house once the workflow is proven.


