Design an A/B test results workbook

작성자: AILesson9 분 소요테스트::ChatGPT검토일: 2026-08-28

빠른 답변

Structure assignment, exposure, outcomes, exclusions, uncertainty, and decision rules without overstating a test. 제공할 내용: Experiment design, Data schema and counts, Metrics and analysis rules. 예상 결과: An auditable experiment workbook with metric definitions, formulas, balance checks, and decision notes.

맥락 추가

텍스트는 이 브라우저에 유지됩니다. AILesson Prompts는 이를 모델이나 서버로 보내지 않습니다.

Experiment designData schema and countsMetrics and analysis rules

프롬프트

채워지지 않은 필드는 플레이스홀더로 표시되므로 프롬프트를 복사하고 편집할 수 있습니다

Design an Excel workbook for the supplied A/B test without turning association into a stronger causal claim than the design supports.

Hypothesis, variants, randomization, eligibility, dates, stopping, exclusions, and owner:
[design]

Fields, grain, IDs, assignment, exposure, outcomes, timestamps, missingness, duplicates, counts, and samples:
[data]

Metrics, denominators, population, confidence method, practical threshold, segments, multiplicity, and review limits:
[analysis]

Separate experiment plan from observed results. Define one row per randomization unit or a reproducible aggregation from event data. Preserve assigned group even when exposure fails for the primary intention-to-treat view; label any per-protocol analysis as secondary. Specify formulas for denominator, rate or continuous outcome, absolute and relative difference, uncertainty, and guardrails. Add sample-ratio, eligibility, duplicate-ID, missing-outcome, timing, contamination, and pre-experiment balance checks. Do not choose a winner from p-value alone, peek without the planned stopping rule, infer user-level effects from event rows, or claim causality when assignment was not randomized. Return: sheet architecture; data dictionary; metric registry; formulas using supplied cells/tables; quality checks; results table; segment policy; decision template; limitations; and tests that recalculate a small known example.

Playground에서 사용해 보기

기본적으로 비공개프롬프트 구성은 브라우저에서 로컬로 이루어집니다. 조직에서 허용하지 않는 한 기밀 정보를 AI 서비스에 입력하지 마세요.

입력에서 결과까지

적용 예시

구체적인 맥락이 이 레시피를 바로 사용할 수 있는 결과로 바꾸는 방법을 확인하세요

실제 입력

Experiment design: Randomized email subject test for eligible newsletter subscribers. Control A is current subject, B is shorter subject. Randomization unit subscriber ID, 1:1 assignment on 3 August 2026 before send. One email per eligible subscriber. Primary hypothesis: B increases unique click-through within 72 hours. Run once to the full eligible list; no early stopping. Exclude only prelisted internal/test IDs and hard bounces under the approved metric rule. Marketing lead owns decision.
Data schema and counts: Raw event export can contain many rows per subscriber. Fields SubscriberID, AssignedVariant, Eligible, Sent, HardBounce, UniqueClick72h, Unsubscribe72h, Revenue72h, AssignmentTime, SendTime. Aggregate to one row per SubscriberID. Expected assigned counts A=5,012, B=4,988. Hard bounces A=102, B=98. Unique clicks A=421, B=455. Unsubscribes A=23, B=31. Revenue is skewed and missing for 14 rows because tracking failed. Duplicate SubscriberID in assignment is invalid. Exposure crossover can be detected if sent subject differs from assignment.
Metrics and analysis rules: Primary unique CTR denominator is sent minus hard bounce, by assigned variant; record that this differs from pure all-assigned ITT and provide all-assigned sensitivity. Primary test two-sided 95% confidence interval for difference in proportions. Practical threshold +0.5 percentage points absolute. Guardrail unsubscribe rate denominator sent, non-inferiority margin +0.2 percentage points. Revenue secondary descriptive only. Segments exploratory with n and no winner claims; no multiplicity adjustment planned.

예시 출력

Workbook: Plan (locked hypothesis, population, stopping and thresholds), UnitData (one row per SubscriberID), MetricRegistry, QualityChecks, Results, and DecisionLog. UnitData must be aggregated from events with explicit MAX rules for binary outcomes and a duplicate-assignment stop; never count clicks as independent rows.

Primary denominators are A 5,012−102=4,910 and B 4,988−98=4,890. CTRs are A 421/4,910=8.574% and B 455/4,890=9.305%; absolute difference B−A=0.731 percentage points and relative lift about 8.53%. Excel rate formulas can use =SUMIFS(UnitData[UniqueClick72h],UnitData[AssignedVariant],A2)/SUMIFS(UnitData[PrimaryDenominatorFlag],UnitData[AssignedVariant],A2). Use a preapproved two-proportion confidence calculation or reviewed analysis tool; do not improvise the statistical method after seeing results. The point estimate exceeds the +0.5 pp practical threshold, but the decision also requires the interval and guardrail.

Guardrail: A unsubscribe 23/5,012=0.459%; B 31/4,988=0.622%; difference +0.163 pp, below but close to the +0.2 pp non-inferiority margin. Report its interval before calling the guardrail met. QualityChecks: assigned split 50.12%/49.88%; compare with planned allocation using the approved sample-ratio check; zero duplicate assignments; all assignments predate send; list crossover, ineligible, unsent, missing tracking, and exclusions by group. All-assigned sensitivity CTRs are 8.400% versus 9.122%. Revenue remains descriptive with n/missingness and median or quantiles, not a causal success claim. Decision template: proceed, hold, or reject, citing primary interval, practical threshold, guardrail interval, quality issues, and any divergence between denominator definitions.

효과가 있는 이유

1
Keeping the randomization unit and denominator explicit prevents event volume from masquerading as independent sample size.
2
Decision thresholds and guardrails keep statistical uncertainty connected to practical consequences.

결과 확인

Does each analysis row represent the randomization unit, with assignment retained regardless of exposure?
Are primary metric, denominator, uncertainty method, stopping rule, and practical threshold defined before reading results?
Are imbalance, missingness, contamination, multiplicity, and guardrails visible beside the result?

안심하고 사용하세요

자주 묻는 질문

이 레시피를 언제 사용해야 하는지, 무엇을 제공해야 하는지, 그리고 어떤 부분에서 사람의 검토가 여전히 중요한지에 대한 실용적인 답변

What should I prepare before using “Design an A/B test results workbook”?

For “Design an A/B test results workbook,” prepare Experiment design, Data schema and counts, and Metrics and analysis rules. Replace placeholders only with information you can verify. If a detail is unknown, preserve that uncertainty explicitly instead of asking the model to infer it.

When is the “Design an A/B test results workbook” result not ready to use?

The result is not ready if it does not yet deliver the stated outcome—An auditable experiment workbook with metric definitions, formulas, balance checks, and decision notes—from the supplied evidence, or if it relies on unresolved assumptions, missing approvals, or invented details. Use the checks as release gates: revise the source inputs or assign a named, authorized reviewer instead of polishing an unsupported output.

Which AI tools have recorded tests for “Design an A/B test results workbook”?

The published test record for “Design an A/B test results workbook” lists ChatGPT as of 2026-08-28. This confirms recorded runs, not guaranteed compatibility or identical results in later product versions. For another tool or version, keep every constraint visible and repeat the result checks before use.

더 많은 탐색 방법