A/B Test

An A/B test is an experimentation method that randomly shows different versions of the same page or content item to comparable visitors, then measures which version best reaches a goal set in advance. It replaces a debate of opinions (which headline, which image, which button) with evidence from real behavior. In science, the same logic is called a randomized controlled trial.

Definition and how it works

The starting version is the baseline, known as the control variant. Every other variant is compared with it on a single success criterion, the conversion rate of a goal. Random assignment is what keeps the comparison fair, because the groups differ only by the version they saw.

In Jahia jExperience, an A/B test applies to a whole page or to a content item placed on a page. The contributor creates the test from the page in Page Builder, adds variants, then sets a start date, an end date, the control variant and an end strategy: automatically serve the variant with the highest conversion rate, or keep serving the control. A goal must be defined. It cannot be changed once the test has started so that results stay consistent.

Results are shown per variant: views, visitors who reached the goal, conversion rate and the percentage difference from the control (the lift). A segment filter shows whether another variant works better for part of the audience.

Academy documentation: creating A/B tests in Page Composer

A/B test vs personalization: the differences

A/B tests and personalization both work with content variants, but for different purposes. A test is for learning. Personalization applies what you already know about each visitor.

How an A/B test differs from related terms
Term	Difference from an A/B test
Personalization	Personalization picks a variant based on conditions about the visitor (segment, behavior, context). An A/B test assigns variants at random to compare their results. A test can lead to a lasting personalization, segment by segment.
Goal	The goal is the action that defines success. An A/B test cannot name a winner without one: the goal provides the comparison criterion.
Conversion rate	The conversion rate is the measure a goal produces. The test compares this rate across variants and also expresses it as a difference from the control.

A/B test in practice

A digital team cannot decide between two hero messages for an offer page. It creates an A/B test on the hero block, keeps the current version as the control and sets the goal as a visit to the meeting request page. It sets an end date and decides in advance to keep the control if the difference does not hold.

Three decisions are made before launch, not after. They cover the goal (it is locked once the test starts), the duration and what happens at the end. Changing them midway would mean changing the rules during the game.

Frequently asked questions

Do we need a separate A/B testing tool if we already use Jahia jExperience?

For pages and content managed in Jahia, jExperience lets you create the test, its variants and its goal, then follow its results, without a third-party tool. A separate tool can still make sense for journeys that do not run through pages rendered by Jahia. The real question is which pages you need to test, not whether the feature exists.

Can we change the goal of an A/B test that is already running?

Once the test has started, the goal is locked in jExperience. This is deliberate. Switching criteria midway would mix measurements of different actions and make the results inconsistent. If the goal was poorly chosen, the sound approach is to stop the test and start a new one with the right goal from the outset.

Does the variant with the most clicks always win?

Clicks do not pick the winner. The variant that best reaches the predefined goal does. A more visible button can generate more clicks without producing more completed requests. You also need enough volume over a representative period. A gap observed over a few dozen visits is not enough to draw a conclusion.