primary metric
One main metric, such as start_question_rate or checkout_started.
Do not choose the metric after seeing the result.
where a test begins
Hypotheses
OmenHall hypothesis template: how to separate a testable idea from a pretty wish that cannot be measured.
A hypothesis must be short, testable, and safe for trust.
guardrail metric
A metric that protects trust: refunds, support complaints, unsubscribes, rage clicks.
If the guardrail worsens, a conversion win may be poison in a golden cup.
minimum runtime
Minimum time before reading results, often at least 7 days for low traffic.
Do not close a test after a random spike.
Write the hypothesis
Formula: if we change X for audience Y, metric Z will move because a specific friction disappears.
- There is one main changed element.
- There is one primary metric and one guardrail metric.
- There is a pre-written decision after the test.
Launch quietly
Start with a small traffic share or one segment, especially when testing payment, email, trust, or beginners.
- The test can be turned off without deploy chaos.
- Events are logged without private questions or scroll text.
- Support knows what changed.
Read the result
Do not declare a winner from one day. Check device split, traffic source, complaints, refunds, and behavior quality after payment.
- The result is compared with baseline.
- The guardrail did not worsen.
- The outcome is written into the experiment log.
experiment cards
- If we shorten the pricing page hero, more users will start checkout because the offer becomes easier to scan.
- If we add a privacy note before gift reading purchase, fewer users will abandon because the recipient boundary becomes clear.
- If My Day shows one real-world action, more users will save the reading because the result feels practical.
rollback rules
- Stop the test if refunds or complaints rise more than the primary metric.
- Stop the test if copy becomes anxious or pushy.
output
- One hypothesis card with owner, dates, segment, metric, guardrail, and expected decision.
- A link to analytics dashboard and experiment log.
red flags
- The hypothesis sounds like 'make it prettier' without a metric.
- The test promises to predict relationships, health, legal, or money outcomes.
- There is no decision owner after the test.
related doors