imbahub
Menu +
US
Talk to us
IMBAHUB / Knowledge & guides

Check that your experiment measures the version actually shown.

A variant label in an event is insufficient when the screen shows something different. Test continuity from assignment to content and defined outcome. This guide concerns technical evidence quality rather than choosing a statistical winner.

Make three observations comparable
  1. Chosen variant

    Which version was assigned in this experiment?

  2. Shown variant

    Which content was actually delivered?

  3. Observed outcome

    Which defined action connects to that exposure?

A correct event label can still describe the wrong content.

The preview link shows B and sends ordinary conversions. Is internal traffic excluded?

Not without receiving-side evidence. Test the whole collection route and separate test observations under the plan.

Save the experiment version and expected assignment

Record experiment identity, variant versions, assignment rule and exclusions. Use an isolated test mode with fictional events. Editorial preview routes should be clearly distinguished from a real participant's route.

A fictional example compares A and B explanations of a working process. B receives new copy after the initial checks. Old and new B must not be mixed as identical content without an explicit version rule.

Compare actual content with its event

Reach each variant through the agreed test mechanism and inspect content and recording together. Define when exposure can count. A script starting or an identifier being assigned can precede the relevant content appearing.

Try reloads, direct links and back navigation where the implementation uses them. Check caches reuse the right response and page sections do not mix variants. Keep evidence of the displayed version rather than just an event name.

TestRiskExpected evidence
B assigned with cache activeA delivered but B recordedContent and exposure signal agree
Return visitUnintended variant switchingApproved continuity rule holds
PreviewInternal review enters the experimentTest route identified and handled
Version changeDifferent B versions combinedVersions separable under analysis decision
Stopped experimentOld assignment continuesAgreed final state without test triggering

Exercise missing storage and internal routes

Test a fresh state and one where agreed storage is unavailable. Expectations follow the approved data requirements and experiment model. Do not promise recognition of the same person across devices or blocked clients.

Open the editorial review link and check whether its events are separated under the plan. A parameter named test does not help if another tag still collects the same outcome. Inspect real receiving routes and avoid actual leads in test cases.

Document drift before interpreting results

When recorded and displayed variants differ, identify affected version, route and period. Correct one established cause and rerun positive controls and exceptions. Do not relabel the report just to make a result look cleaner.

Record variant, actual display, exposure condition, outcome and failure boundary. Decide separately how affected observations are treated. A corrected implementation does not establish whether earlier data is sufficient or whether the change has a reliable business effect.

Sources and further reading

Questions, answered

Can internal tests affect the experiment?

Yes, when collected as normal participant activity. Define and test the actual collection route.

Can older data stay after a variant change?

That needs a documented analysis decision and identifiable versions. Different content versions are not automatically the same treatment.

IMBAHUB

Let’s plan your website

Choose a case with assignment, screen content and outcome recording, then compare all three.

Talk to us
Existing English product interface with fictional demonstration content; reused for the US website.

Existing English product interface with fictional demonstration content; reused for the US website.