variant exposure
A user should consistently see one variant during the session.
Do not switch variant on reload.
clean experiment
A/B tests
OmenHall A/B testing rules: one element, one segment, one window, one conclusion.
An A/B test must not break the user path or hide important warnings.
conversion delta
Measure change against baseline, not one day's excitement.
Do not call tiny noise a win without guardrail.
device split
Read mobile and desktop separately.
A desktop win can be a mobile failure.
Write the hypothesis
Formula: if we change X for audience Y, metric Z will move because a specific friction disappears.
- There is one main changed element.
- There is one primary metric and one guardrail metric.
- There is a pre-written decision after the test.
Launch quietly
Start with a small traffic share or one segment, especially when testing payment, email, trust, or beginners.
- The test can be turned off without deploy chaos.
- Events are logged without private questions or scroll text.
- Support knows what changed.
Read the result
Do not declare a winner from one day. Check device split, traffic source, complaints, refunds, and behavior quality after payment.
- The result is compared with baseline.
- The guardrail did not worsen.
- The outcome is written into the experiment log.
experiment cards
- Hero CTA: 'Get an answer' versus 'Start with a question'.
- Pricing cards: three tiers versus guided recommendation.
- Search hub: popular doors above categories versus categories above popular doors.
rollback rules
- Rollback if checkout errors or rage clicks rise.
- Rollback if support gets 'where do I click' questions.
output
- Decision: ship A, ship B, iterate, or rollback.
- Screenshots of variants and re-check date.
red flags
- Variants differ across several major elements.
- Variant B hides disclaimer or refunds.
related doors