Design an A/B test results workbook

Author: AILesson9 min setupTested with:ChatGPTReviewed: 2026-08-28

Quick answer

Structure assignment, exposure, outcomes, exclusions, uncertainty, and decision rules without overstating a test. Provide: Experiment design, Data schema and counts, Metrics and analysis rules. Expected result: An auditable experiment workbook with metric definitions, formulas, balance checks, and decision notes.

Add your context

Your text stays in this browser. AILesson Prompts does not send it to a model or server.

Experiment designData schema and countsMetrics and analysis rules

Your prompt

Unfilled fields remain visible as placeholders, so you can still copy and edit the prompt

Design an Excel workbook for the supplied A/B test without turning association into a stronger causal claim than the design supports.

Hypothesis, variants, randomization, eligibility, dates, stopping, exclusions, and owner:
[design]

Fields, grain, IDs, assignment, exposure, outcomes, timestamps, missingness, duplicates, counts, and samples:
[data]

Metrics, denominators, population, confidence method, practical threshold, segments, multiplicity, and review limits:
[analysis]

Separate experiment plan from observed results. Define one row per randomization unit or a reproducible aggregation from event data. Preserve assigned group even when exposure fails for the primary intention-to-treat view; label any per-protocol analysis as secondary. Specify formulas for denominator, rate or continuous outcome, absolute and relative difference, uncertainty, and guardrails. Add sample-ratio, eligibility, duplicate-ID, missing-outcome, timing, contamination, and pre-experiment balance checks. Do not choose a winner from p-value alone, peek without the planned stopping rule, infer user-level effects from event rows, or claim causality when assignment was not randomized. Return: sheet architecture; data dictionary; metric registry; formulas using supplied cells/tables; quality checks; results table; segment policy; decision template; limitations; and tests that recalculate a small known example.

Private by defaultPrompt assembly happens locally in your browser. Avoid placing confidential information into any AI service unless your organization allows it.

From input to outcome

A worked example

See how concrete context turns this recipe into a usable result

Actual input

Experiment design: Randomized email subject test for eligible newsletter subscribers. Control A is current subject, B is shorter subject. Randomization unit subscriber ID, 1:1 assignment on 3 August 2026 before send. One email per eligible subscriber. Primary hypothesis: B increases unique click-through within 72 hours. Run once to the full eligible list; no early stopping. Exclude only prelisted internal/test IDs and hard bounces under the approved metric rule. Marketing lead owns decision.
Data schema and counts: Raw event export can contain many rows per subscriber. Fields SubscriberID, AssignedVariant, Eligible, Sent, HardBounce, UniqueClick72h, Unsubscribe72h, Revenue72h, AssignmentTime, SendTime. Aggregate to one row per SubscriberID. Expected assigned counts A=5,012, B=4,988. Hard bounces A=102, B=98. Unique clicks A=421, B=455. Unsubscribes A=23, B=31. Revenue is skewed and missing for 14 rows because tracking failed. Duplicate SubscriberID in assignment is invalid. Exposure crossover can be detected if sent subject differs from assignment.
Metrics and analysis rules: Primary unique CTR denominator is sent minus hard bounce, by assigned variant; record that this differs from pure all-assigned ITT and provide all-assigned sensitivity. Primary test two-sided 95% confidence interval for difference in proportions. Practical threshold +0.5 percentage points absolute. Guardrail unsubscribe rate denominator sent, non-inferiority margin +0.2 percentage points. Revenue secondary descriptive only. Segments exploratory with n and no winner claims; no multiplicity adjustment planned.

Example output

Workbook: Plan (locked hypothesis, population, stopping and thresholds), UnitData (one row per SubscriberID), MetricRegistry, QualityChecks, Results, and DecisionLog. UnitData must be aggregated from events with explicit MAX rules for binary outcomes and a duplicate-assignment stop; never count clicks as independent rows.

Primary denominators are A 5,012−102=4,910 and B 4,988−98=4,890. CTRs are A 421/4,910=8.574% and B 455/4,890=9.305%; absolute difference B−A=0.731 percentage points and relative lift about 8.53%. Excel rate formulas can use =SUMIFS(UnitData[UniqueClick72h],UnitData[AssignedVariant],A2)/SUMIFS(UnitData[PrimaryDenominatorFlag],UnitData[AssignedVariant],A2). Use a preapproved two-proportion confidence calculation or reviewed analysis tool; do not improvise the statistical method after seeing results. The point estimate exceeds the +0.5 pp practical threshold, but the decision also requires the interval and guardrail.

Guardrail: A unsubscribe 23/5,012=0.459%; B 31/4,988=0.622%; difference +0.163 pp, below but close to the +0.2 pp non-inferiority margin. Report its interval before calling the guardrail met. QualityChecks: assigned split 50.12%/49.88%; compare with planned allocation using the approved sample-ratio check; zero duplicate assignments; all assignments predate send; list crossover, ineligible, unsent, missing tracking, and exclusions by group. All-assigned sensitivity CTRs are 8.400% versus 9.122%. Revenue remains descriptive with n/missingness and median or quantiles, not a causal success claim. Decision template: proceed, hold, or reject, citing primary interval, practical threshold, guardrail interval, quality issues, and any divergence between denominator definitions.

Why this works

1
Keeping the randomization unit and denominator explicit prevents event volume from masquerading as independent sample size.
2
Decision thresholds and guardrails keep statistical uncertainty connected to practical consequences.

Check the result

Does each analysis row represent the randomization unit, with assignment retained regardless of exposure?
Are primary metric, denominator, uncertainty method, stopping rule, and practical threshold defined before reading results?
Are imbalance, missingness, contamination, multiplicity, and guardrails visible beside the result?

Use it with confidence

Frequently asked questions

Practical answers about when to use this recipe, what to provide, and where human review still matters

What should I prepare before using “Design an A/B test results workbook”?

For “Design an A/B test results workbook,” prepare Experiment design, Data schema and counts, and Metrics and analysis rules. Replace placeholders only with information you can verify. If a detail is unknown, preserve that uncertainty explicitly instead of asking the model to infer it.

When is the “Design an A/B test results workbook” result not ready to use?

The result is not ready if it does not yet deliver the stated outcome—An auditable experiment workbook with metric definitions, formulas, balance checks, and decision notes—from the supplied evidence, or if it relies on unresolved assumptions, missing approvals, or invented details. Use the checks as release gates: revise the source inputs or assign a named, authorized reviewer instead of polishing an unsupported output.

Which AI tools have recorded tests for “Design an A/B test results workbook”?

The published test record for “Design an A/B test results workbook” lists ChatGPT as of 2026-08-28. This confirms recorded runs, not guaranteed compatibility or identical results in later product versions. For another tool or version, keep every constraint visible and repeat the result checks before use.

More ways to explore