refund spike
Unusual refund-rate rise or repeated refund reason.
Do not wait for statistical beauty if customers are clearly unhappy.
when to close a test
Rollback rules
Rollback rules: how to stop a test fast when it harms trust, payment, email, or support.
Rollback is not failure. It prevents buying growth with reputation.
support spike
Rise in repeated questions, complaints, or lost paths.
Support often sees the fire before the dashboard.
trust breach
Violation of forbidden claims, privacy, disclaimer, or consent.
Trust breach rolls back immediately.
Write the hypothesis
Formula: if we change X for audience Y, metric Z will move because a specific friction disappears.
- There is one main changed element.
- There is one primary metric and one guardrail metric.
- There is a pre-written decision after the test.
Launch quietly
Start with a small traffic share or one segment, especially when testing payment, email, trust, or beginners.
- The test can be turned off without deploy chaos.
- Events are logged without private questions or scroll text.
- Support knows what changed.
Read the result
Do not declare a winner from one day. Check device split, traffic source, complaints, refunds, and behavior quality after payment.
- The result is compared with baseline.
- The guardrail did not worsen.
- The outcome is written into the experiment log.
experiment cards
- Rollback if checkout success rises but refund reasons mention wrong expectations.
- Rollback if email opens rise but unsubscribe complaints rise faster.
- Rollback if copy increases clicks by using fear or urgency.
rollback rules
- Critical: privacy, legal, payment, forbidden claim. Stop immediately.
- Major: refunds, support confusion, mobile failure. Pause and review.
- Minor: weak metric, no harm. Let run or iterate after scheduled read.
output
- Rollback note with time, reason, affected routes, and owner.
- Support macro if users already saw the bad variant.
red flags
- The team argues with the user instead of reading the signal.
- The test cannot be disabled without deploy.
related doors