One of the most important parts of creating an A/B test is proving or disproving a hypothesis. In order to do that, you need to measure a change using one or more metrics.
When it comes to assigning metrics to experiments, Optimizely Web gives you a number of different assignment options to pick from. Assigning a metric correctly is key to getting the most out of your experimentation program. If you want to learn how to make the most from your metrics in Optimizely Web, this is the article for you 🔥🔥🔥
Setting up a metric using Optimizely Web is simple enough. On any given experiment, from the Metric section you can add and remove metrics. One of the unique things about Optimizely is the stats engine it uses.
Optimizely Web uses something called sequential testing, mixed with a low fault discovery rate technique. This technique was pioneered in a partnership with Optimizely and Stanford University. The reason why you should care about the underlining algorithm is that it means that you can attach multiple metrics to an experiment and not be penalised for the time it takes to get statistical significance. Ths focus of this article is metrics, however,r if you want to learn the ins and the outs of the stats engine you can here.
Adding a metric to an experiment is simple enough:
Within Optimizely experimentation, metrics are sent to the platform via events. Optimizely provides four different types of events that you can use to measure success within an experiment. You can measure success using an in-build metric, a click, a page view, or a custom event. Click and page view events can be created within the Optimizely editor by non-technical people. Custom events need to be created at the code level by a developer. The last event which is an in-built event is called Overall Revenue, it is installed out of the box, however, it does require some code to be added to your website before it will work. The process to create one of these events within the UI is shown below:
To recap, after you send an event to Optimziley it can be used to measure the results of an experiment. In order to measure the data captured from an event onto an experiment, you create a metric. Regardless of the metric that you create, you can also configure what success means for that metric in a few different ways. For each metric, Optimizely allows you to configure the aggregator, the scope, the winning direction and the field type.
The winning direction is simple enough to understand. When set as an increasing event, when an event is triggered if it goes up then it is measured as a success. When set as a decreasing event, when an event is triggered if it goes down it is measured as a success.
To make things a little more user-friendly within the UI the editor is not asked which aggregator or type they would like to use. Instead, metrics are configured in a more user-friendly manner using two dropdowns within a sentence. The only reason why you might need to learn about these behind-the-scenes terms is if you are more technical and you want to work with metrics using the Optimizely REST API. If you fall into that camp, you can learn a little more about this from the API here. It is worth noting that you will have a little more power when using the API compared to the UI. For example, the denominator is not configurable, however, it is via the API!
The winning direction is simple enough to understand. When set as an increasing event, when an event is triggered if it goes up then it is measured as a success. When set as a decreasing event, when an event is triggered if it goes down it is measured as a success.
The aggregator value will be automatically set for you depending on the type of metric you create. The aggregator can be set as either unique,count, or sum
uniquemeasures the unique number of visitors/sessionscountmeasures the total number of occurrences of the eventsumis the sum of the 'field' value. You would usesumto calculate the average order value!
Custom Events Metrics
If you want to determine statistical significance against a metric that is powered from a custom event then you can measure it using either:
Unique conversions: The unique number of events added for that metric. In this mode, the aggregator will be set as unique
Total conversions: The total number of events added for that metric. In this mode, the aggregator will be set as count
Total revenue: Measures the total number of events added for that metric. The data is measured in a way that allows it to accurately works with financial data, e.g. no rounding issue. This will set the aggregator as count
Total value: Measures the total number of events added for that metric. This mode will encounter rounding issues and should not be used to measure money. Examples of when you might use this are to measure the duration someone watched a video, a page load time, time spent listening to a webinar, the number of articles read the number of steps within a funnel they completed. This model set the aggregator as count
Click or a page view event metrics
If you try to track an experiment using a click or a page view event, you get two additional options:
Bounce rate: A bounce rate is determined when someone views your website and leaves straight away without looking at any other page.
Exit rate: The exit rate measure when they leave your website.
Overall Revenue
Optimizely Web also provides a special event called the Overall Revenue metric. You can use this metric to calculate an order value. In terms of configuration, the Overall Revenue metric will show slightly different options compared to any of the above metrics. Overall Revenue allows you to track a metric slightly differently. Within the UI you can see the denominator by visitor or by conversion. It is only through the Overall Revenue metric that you can set the denominator. Setting the denominator will set the scope value in the backend. Using 'session' divides the P-Value by the total number of sessions, using 'visitor' divides by the total number of visitors!
That is pretty much everything you need to know in order to be a metric guru using Optimizely 🧘🧘🧘. After this, you should be able to track experiments like a BOSS, Happy Coding 🤘
