Skip to content
HotLayout Docs
npmpub.devGitHub
Menu

A/B testing with data

HotLayout has no built-in experiment feature. It does not split users, it does not count results, and it has no reports. You can still run an A/B test today, with the pieces that exist: the variant arrives as a value in the screen data, the document shows the part that matches it, and your app reports what happened to your own analytics.

Related pages: data dictionary · widgets · placements · React quickstart

The recipe#

1. Your side decides who sees which variant#

The split belongs to you: your backend, or whatever tool the app already uses for flags and experiments (a remote-config service, an in-house assignment service). It must give the same answer for the same user every time, so a person does not see variant A in the morning and B in the evening. HotLayout takes no part in this step.

2. The variant travels as data#

Add a path to the project's data dictionary, for example experiment.bannerVariant of type string, and send it with the other values of the screen in the flat map of single values:

{
  "user.firstName": "Alex",
  "experiment.bannerVariant": "b"
}

Use short, fixed strings ("a", "b"), because the document compares the value as written. Send a value for every user in the experiment, the control group included. Path names must match your response exactly, as for any other path (see Name fields exactly as your API does).

3. The document holds both variants#

In the editor, put each variant in its own node and give each one a visibility condition on that path. In the JSON of the document the condition is a visibleIf with op eq and a value (see VisibilityCondition):

{ "path": "experiment.bannerVariant", "op": "eq", "value": "a" }
{ "path": "experiment.bannerVariant", "op": "eq", "value": "b" }

A node whose condition does not hold is not drawn, and the layout closes up. If the whole difference between variants is a colour, a size or another style field, one node is enough: give it a styleWhen case with the same kind of condition, and the first case that holds is laid over the base style (see widgets). Text and structure differ per node, so use two nodes for those.

A missing value counts as empty: isEmpty holds, and every other operator, neq included, does not. A user for whom your backend sent nothing therefore sees neither a nor b. Add a third node with { "path": "experiment.bannerVariant", "op": "isEmpty" } if you want a default, or make sure the backend always sends a value.

Both variants live in the same document, so one publication changes the whole experiment.

4. Your app measures#

HotLayout sends nothing anywhere by itself. What it hands to your code:

  • a press on a node whose action has track ({ "event": "banner_cta" }) reaches onTrack once per press, with the id of the pressed node and the screen. React, Flutter, iOS and Android all have it;
  • a screen shown reaches onScreenChange in React and onScreenShown in Flutter, once per screen.

Put a track on the button of each variant, with the same event name in both, and add the variant in your handler, since your app already knows it from step 1:

onTrack={(event, source) => {
  analytics.track(event.event, { ...event.params, node: source.nodeId, screen: source.screenId, variant: experimentVariant })
}}

Here analytics and experimentVariant are your own code. The handler is the one from Analytics and screen views with one added field.

A view of a variant has no event of its own: the document does not report that a node became visible. Count a view as the screen being shown for a user who was assigned that variant. That is exact when the variant sits at the top of the first screen, and approximate when it sits below the fold.

5. Placements: one document, the variant inside it#

The placement map is the same for every user. A placement names one document, and there is no way to give half of the users one document and half another. So do not run the experiment as two documents in one place. Put both variants into the one document of the place, as above, and pass the variant in the data of the placement: <HotlayoutPlacement> takes the same data prop as <HotlayoutScreen>.

What HotLayout does not do#

It does not divide traffic, weight variants, stop an experiment, collect results or compute significance. Assignment and statistics are done by your tooling, and the result is read in your analytics. HotLayout only makes one document look different depending on a value you supply, and tells your code when something was pressed or shown.