How to Run a Geo Holdout Test on Meta

A geo holdout test isolates the incremental impact of Meta ad spend by pausing campaigns in selected geographic regions while measuring sales lift against control regions.

Why Geo Holdout Tests Matter

Meta's attribution window and iOS privacy changes have made incrementality measurement harder. Geo holdout tests bypass those constraints by using geography as a natural experiment boundary. You pause spend in a test region, measure the sales drop, and compare it to a control region where spend continues. The difference is your incremental revenue.

This matters because last-click attribution inflates ROAS. A customer might see your ad on Tuesday, browse on Wednesday, and buy on Thursday via direct traffic. Last-click gives credit to direct. A geo test credits the Meta impression. For DTC brands running $50k+ monthly Meta spend, the gap between attributed and incremental ROAS often ranges 20 - 40 percent.

Geo tests also reveal channel cannibalization. If pausing Meta in Ohio drops total online revenue by 3 percent instead of the attributed 8 percent, you know 5 percent of Meta sales would have happened anyway via organic search or email. That's the true incremental lift.

Sample Size and Region Selection

The first mistake is picking regions too small. A test region needs enough baseline weekly revenue to detect a meaningful lift. If your test region does $2,000 weekly revenue and you expect a 5 percent lift, you're looking for a $100 signal in noise. That requires 8 - 12 weeks of data to reach statistical significance at 90 percent confidence.

A practical rule: test regions should represent at least 5 - 10 percent of your total online revenue. If you do $100k monthly online revenue, pick regions that do $5k - $10k monthly. For a $500k monthly brand, $25k - $50k test regions are viable. Smaller regions need longer test windows or higher expected lift to be valid.

Match test and control regions on customer density, income, and existing Meta penetration. If your test region is rural and your control is urban, regional differences in online shopping behavior will confound results. Use Nielsen DMA (Designated Market Area) or state-level splits if you have enough volume. Avoid cherry-picking regions with unusual seasonality or competitive pressure.

Run at least two test regions and two control regions. Single regions are vulnerable to local events, supply chain issues, or competitor activity. If one test region has a data anomaly, the second validates findings. This also lets you estimate variance and improve sample size calculations for future tests.

Test Duration and Statistical Power

Minimum test duration is 4 weeks. This covers two full weeks of pre-test baseline and two weeks of holdout measurement. Four weeks is the floor; it assumes high baseline revenue in test regions and a large expected lift (8 percent or higher). Most tests should run 6 - 8 weeks for 90 percent statistical confidence.

The math: statistical power depends on baseline variance, sample size (weekly revenue in test region), and effect size (expected lift). If weekly revenue in a test region is $5,000 and you expect a 5 percent lift ($250), you need roughly 8 weeks to reach 80 percent power. If you expect 10 percent lift, 4 - 5 weeks is sufficient. Use an online calculator (search 'two-sample t-test power calculator') and plug in your baseline weekly revenue standard deviation and expected lift.

Account for seasonality. Never run a test that spans a major holiday or promotional window in only the test region. If Black Friday is week 5 of your test, the control region will see a revenue spike that makes the test uninterpretable. Run tests during flat seasonal periods, or ensure both regions experience the same promotional calendar.

Avoid peeking. Checking results weekly and stopping early when you hit significance is a form of p-hacking. It inflates false positive rates. Commit to a duration upfront, analyze once, and report.

Common Design Mistakes

Mistake 1: Pausing all paid channels in the test region. If you pause Meta but keep Google Ads, email, and TikTok running, you're not measuring Meta's incremental value - you're measuring the combined effect of all channels. Pause only Meta. Let other channels run normally in both test and control. This isolates Meta's contribution.

Mistake 2: Changing creative, targeting, or budgets during the test. Consistency is critical. If you launch a new creative in week 3 of an 8-week test, you've introduced a confounding variable. Lock all campaign settings before the test starts. The only change is the pause in the test region.

Mistake 3: Using revenue from the test region's paid traffic to measure lift. If you pause Meta in Ohio but still see Ohio revenue from organic, direct, and email, that's your control metric. Don't subtract paid revenue from the holdout region and compare it to the control region's total revenue. Compare total revenue (all channels) in test vs. control. The difference is incremental.

Mistake 4: Ignoring cross-region customer movement. Some customers in your control region may have shipped to addresses in the test region, or vice versa. This is usually small (2 - 5 percent) but can bias results in small tests. If possible, attribute revenue by customer shipping address, not IP address.

Mistake 5: Running tests during new customer acquisition campaigns. If you're testing a new audience segment or lookalike in the test region, the holdout becomes uninterpretable. Run tests on stable, mature campaigns. New campaign tests require different methodology (incrementality studies via randomized holdout groups, not geo tests).

Measurement and Analysis

Establish a pre-test baseline. Collect 2 - 4 weeks of data before pausing spend. Calculate average weekly revenue in test and control regions. This baseline is your null hypothesis: revenue should be equal in both regions if Meta has no effect.

During the holdout, track weekly revenue in both regions. Use a two-sample t-test to compare mean weekly revenue in the test period (weeks 5 - 8) against the baseline period (weeks 1 - 4). The t-statistic tells you if the difference is statistically significant. A p-value below 0.10 (90 percent confidence) is acceptable for business decisions; 0.05 is stronger.

Calculate incremental revenue as: (Control region weekly revenue in test period - Test region weekly revenue in test period) × number of weeks. If the control region averaged $10,000 weekly and the test region dropped to $9,200 weekly over 4 weeks, incremental revenue is ($10,000 - $9,200) × 4 = $3,200.

Divide incremental revenue by total Meta spend in the control region during the test period to estimate incremental ROAS. If you spent $8,000 on Meta in the control region and generated $3,200 incremental revenue, incremental ROAS is 0.4x (or 40 cents per dollar spent). This is your true return, not attributed ROAS.

Scaling and Iteration

One test is a data point, not a strategy. Run at least two geo holdout tests per year to account for seasonality and campaign changes. Results from a summer test may not hold in winter if your customer base is seasonal.

Use test results to inform budget allocation. If incremental ROAS is 0.5x but attributed ROAS is 1.2x, you know Meta is profitable but less efficient than attribution suggests. This might justify shifting budget to email or organic, or it might justify continued Meta spend if other channels are lower-margin.

Document assumptions. Record expected lift, baseline revenue, test duration, and regions. When you run the next test, compare results. If incremental ROAS is consistently 0.4x - 0.6x across multiple tests, that's your true Meta efficiency. Use it for forecasting and budget planning.

Share results with finance and leadership. Incremental ROAS is the metric that matters for profitability, not attributed ROAS. A brand running $500k monthly Meta spend at 0.5x incremental ROAS is generating $250k incremental revenue. That's the number that drives decisions.

Tools and Logistics

You need three things: a way to pause Meta spend by geography, clean revenue data by region, and a statistical calculator. Meta Ads Manager lets you pause campaigns by country and state. If you need city-level precision, you'll need custom audiences or pixel-based exclusions, which are harder to manage.

Revenue data must be clean and attributed by customer location (shipping address, not IP). Most ecommerce platforms (Shopify, BigCommerce) can filter orders by state or region. If you use a data warehouse, write a query that segments revenue by region for the test period and baseline period.

Use a simple spreadsheet or statistical tool to run the t-test. Google Sheets has TTEST function. Python scipy.stats has ttest_ind. Online calculators work too. The key is documenting your inputs and outputs so results are reproducible.

FAQ

How much revenue drop should I expect in the test region?

It depends on Meta's share of your traffic and the strength of your brand. If Meta drives 20 percent of your online revenue and has a 5 percent incremental lift, you'd expect a 1 percent revenue drop in the test region (20% × 5%). For a region doing $10k weekly, that's a $100 drop. If you expect 10 percent incremental lift, expect a 2 percent drop ($200). Larger drops are possible if Meta is a primary driver of traffic.

Can I run a geo test on TikTok or Google Ads instead of Meta?

Yes. The methodology is identical. Pause TikTok or Google in test regions, measure revenue lift in control regions, and calculate incremental ROAS. TikTok geo targeting is less granular than Meta (country and region level), so you may need larger test regions. Google Ads geo targeting is strong at state and city level. The statistical requirements (sample size, duration) remain the same.

What if my test shows negative incremental ROAS?

Negative incremental ROAS means revenue in the test region stayed flat or increased when you paused Meta. This suggests Meta spend is not driving incremental sales, or that other channels (organic, email, direct) are substituting for the paused spend. This is valuable data. It means you should reduce Meta spend or reallocate budget. Verify the result with a second test before making major changes.

How do I account for customer overlap between regions?

Some customers in the control region will ship to addresses in the test region, and vice versa. If this is significant (more than 5 percent of revenue), attribute revenue by shipping address, not by the region where the customer saw the ad. This requires more detailed data but gives a cleaner signal. For most brands, this overlap is small enough to ignore, but document it in your test report.

FAQ

How much revenue drop should I expect in the test region?

It depends on Meta's share of your traffic and the strength of your brand. If Meta drives 20 percent of your online revenue and has a 5 percent incremental lift, you'd expect a 1 percent revenue drop in the test region (20% × 5%). For a region doing $10k weekly, that's a $100 drop. If you expect 10 percent incremental lift, expect a 2 percent drop ($200). Larger drops are possible if Meta is a primary driver of traffic.

Can I run a geo test on TikTok or Google Ads instead of Meta?

Yes. The methodology is identical. Pause TikTok or Google in test regions, measure revenue lift in control regions, and calculate incremental ROAS. TikTok geo targeting is less granular than Meta (country and region level), so you may need larger test regions. Google Ads geo targeting is strong at state and city level. The statistical requirements (sample size, duration) remain the same.

What if my test shows negative incremental ROAS?

Negative incremental ROAS means revenue in the test region stayed flat or increased when you paused Meta. This suggests Meta spend is not driving incremental sales, or that other channels (organic, email, direct) are substituting for the paused spend. This is valuable data. It means you should reduce Meta spend or reallocate budget. Verify the result with a second test before making major changes.

How do I account for customer overlap between regions?

Some customers in the control region will ship to addresses in the test region, and vice versa. If this is significant (more than 5 percent of revenue), attribute revenue by shipping address, not by the region where the customer saw the ad. This requires more detailed data but gives a cleaner signal. For most brands, this overlap is small enough to ignore, but document it in your test report.