A/B Test
An A/B test is an experimentation method that randomly shows different versions of the same page or content item to comparable visitors, then measures which version best reaches a goal set in advance. It replaces a debate of opinions (which headline, which image, which button) with evidence from real behavior. In science, the same logic is called a randomized controlled trial.
Definition and how it works
The starting version is the baseline, known as the control variant. Every other variant is compared with it on a single success criterion, the conversion rate of a goal. Random assignment is what keeps the comparison fair, because the groups differ only by the version they saw.
In Jahia jExperience, an A/B test applies to a whole page or to a content item placed on a page. The jExperience 3.6.2 release notes introduce Page Builder compatibility for A/B tests. The Academy procedure, written for Page Composer, has the contributor add variants, then choose the control variant and an end strategy: automatically serve the variant that converts best, or keep serving the control. The start and end dates are optional. When the start date is left empty, the test begins immediately, as soon as the page is published. When the end date is left empty, it runs until someone stops it or until the Maximum hits setting defined for it is reached. A goal must be defined, and it is locked once the test has started so that results stay consistent.
Results are shown per variant: views, visitors who reached the goal, conversion rate and the percentage difference from the control (the lift). A segment filter shows whether a different variant performs better with part of the audience.
Academy documentation: creating A/B tests in Page Composer
Sources: jExperience 3.6.2, Jahia Academy, release notes (page dated 29 September 2025).
Related terms: Conversion Rate · Goal · Content item · Conversion
Resources that cover this term
- How to build a website personalization strategy?
- Customer Data & Personalization
- Enterprise CMS: Definition, Key Features and Benefits
- Headless CMS personalization: How to deliver truly adaptive experiences
- How to connect Salesforce to your CMS: challenges & issues
- How to Justify and Measure Investment in a DXP?
A/B test vs personalization: the differences
A/B tests and personalization both work with content variants, but for different purposes. A test is for learning. Personalization applies what you already know about each visitor.
| Term | Difference from an A/B test |
|---|---|
| Personalization | Personalization picks a variant based on conditions about the visitor (segment, behavior, context). An A/B test assigns variants at random to compare their results. A test can lead to a lasting personalization, segment by segment. |
| Goal | The goal is the action that defines success. An A/B test cannot name a winner without one: the goal provides the comparison criterion. |
| Conversion rate | The conversion rate is the measure a goal produces. The test compares this rate across variants and also expresses it as a difference from the control. |
A/B test in practice
A digital team cannot decide between two hero messages for an offer page. It creates an A/B test on the hero block, keeps the current version as the control and sets the goal as a visit to the meeting request page. It sets an end date and decides in advance to keep the control if the difference does not hold.
Three decisions are made before launch, not after. They cover the goal (it is locked once the test starts), the duration and what happens at the end. Changing them midway would mean changing the rules during the game.
Frequently asked questions
Do we need a separate A/B testing tool if we already use Jahia jExperience?
For pages and content managed in Jahia, jExperience lets you create the test, its variants and its goal, then follow its results. The Academy procedure starts from a page powered by Jahia, so the first question is which pages and content you need to test. A team that also tests elsewhere can keep an additional tool alongside it.
Can we change the goal of an A/B test that is already running?
Once the test has started, the goal is locked in jExperience. This is deliberate. Switching criteria midway would mix measurements of different actions and make the results inconsistent. If the goal was poorly chosen, the sound approach is to stop the test and start a new one with the right goal from the outset.
Does the variant with the most clicks always win?
Clicks do not pick the winner. The variant that best reaches the predefined goal does. A more visible button can generate more clicks without producing more completed requests. You also need enough volume over a representative period. A gap observed over a few dozen visits is not enough to draw a conclusion.