decision recorded
Every test ends with a decision record.
No decision, no learning.
lab memory
Experiment log
Experiment log: what was tested, why, when, what happened, and when to recheck.
Without a log the team will open the same door again and again, surprised by the same creak.
recheck date
Recheck date after implementation.
A winning test can age within a month.
linked artifact
Screenshots, copy diff, dashboard, support notes.
Memory without artifacts quickly becomes myth.
Write the hypothesis
Formula: if we change X for audience Y, metric Z will move because a specific friction disappears.
- There is one main changed element.
- There is one primary metric and one guardrail metric.
- There is a pre-written decision after the test.
Launch quietly
Start with a small traffic share or one segment, especially when testing payment, email, trust, or beginners.
- The test can be turned off without deploy chaos.
- Events are logged without private questions or scroll text.
- Support knows what changed.
Read the result
Do not declare a winner from one day. Check device split, traffic source, complaints, refunds, and behavior quality after payment.
- The result is compared with baseline.
- The guardrail did not worsen.
- The outcome is written into the experiment log.
experiment cards
- Log fields: id, owner, route, hypothesis, variant, metric, guardrail, result, decision.
- Decision values: ship, rollback, iterate, inconclusive, never again.
- Recheck: 7 days, 30 days, after traffic source changes.
rollback rules
- If a test is rolled back, log the exact trigger and timestamp.
- If inconclusive, do not quietly ship because it 'felt better'.
output
- Clean experiment record ready for future team review.
- Update calendar entry if winning change needs recheck.
red flags
- The log contains only wins.
- There is no link to analytics or support notes.
related doors