Experiment documentation fails in two opposite ways. Some teams keep only a result screenshot; others create a report so heavy that people stop testing. The useful middle is a short decision record with links to deeper evidence.
The record should answer what was tested, why, what success meant, what actually happened and what the team will do next.
Document the decision before launch, link rather than duplicate raw analysis, and close the record with an evidence-labelled conclusion and next action.
Before launch: make the decision inspectable
Write the hypothesis before seeing the outcome. Name one primary measure, the intended audience, the exposure window and the conditions that could invalidate the result.
- State the problemDescribe the customer or business behaviour that needs to change.
- Write the hypothesisUse a because statement that connects the intervention to the expected behaviour.
- Set the judgement ruleDefine the primary measure, guardrails and earliest sensible review point.
- Capture the changeStore the exact control and treatment, not only their internal names.
During the run: preserve events that affect interpretation
Avoid narrating every day. Add only material events: tracking breaks, budget changes, stock constraints, overlapping launches, audience shifts or implementation deviations. These notes often explain a surprising result later.
After the run: distinguish result, evidence and decision
The result is what the data showed. The evidence label describes how that result was obtained. The decision says what the team will do. Keeping them separate prevents a positive observation from being retold as proven causality.
Include the effect in practical terms. A statistically detectable movement may be too small to justify operational cost; a directional signal may still justify another test.
Link to analysis; do not copy the warehouse
Keep the durable summary in the log and link to dashboards, notebooks, recordings or tasks. The summary must remain meaningful if those links later lose context, but it should not reproduce every chart.
Close with a reusable sentence
Write one sentence that another team can safely apply: audience, intervention, outcome, confidence and boundary. Then name the next action and owner. This is the part most likely to change a future decision.
Clear answers
Frequently asked questions
How long should an experiment document be?
The core record can usually fit on one screen. Add depth through linked evidence when the risk, complexity or analysis requires it.
When should documentation happen?
Create the hypothesis and judgement rule before launch, note material events during the run, and add the result and conclusion at the agreed review point.
What if multiple changes shipped together?
Record the bundle and avoid claiming which element caused the result. If the components need separate conclusions, follow with controlled tests or smaller changes.
Make this part of the next decision.
Diffined keeps the change, reason, evidence and conclusion together in one private history.
Try the working prototype ↗