Experimentation Lab/Measuring clickthrough rates

This guide explains and shows how to use Experimentation Lab to conduct experiments on user engagement with a feature, as measured by user clicking on a link or button. The guide covers:

Selection of clickthrough rate (CTR) metrics by helping you think about your problem like a product analyst and by helping you either use one of the generic predefined CTRs or define a specific CTR
Implementing an experiment (A/B test) using the standard CTR instrument (built to CTR specification)

Metrics

This portion of the guide is intended for product managers, analysts, and engineers tasked with measuring CTR for a feature.

You must first and foremost consider what is the phenomenon that you are interested in and are trying to model, and why it is important to you. Imagine yourself as a user and the page loading. Which elements should produce an impression event upon becoming visible to you? When you click on a link or a button that has been instrumented, what is the relationship of that click to other clicks you might have made while using the feature? How about while being on the page? How about across multiple pages you have visited or across multiple visits to the same page? What about across multiple sessions within a week-long experiment?

Illustration of different scenarios where impression and click event data would be collected from a user visiting a web page and the different ways to calculate a clickthrough rate.

Consider this scenario: You visit a Special page with 10 instrumented links. There is an impression associated with each link shown. You click on one of those links. You come back to the page and there are 10 same links shown. You don't click any of them. You come back to the page a little bit later and now there are 15 links shown. You right-click (or tap-and-hold if on mobile) and open two of them as new tabs, each producing a click analytics event.

Your session, thus far, can be described as follows:

On the 1^st visit, there were 10 impressions and 1 click.
On the 2^nd visit, there were 10 impressions and 0 clicks.
On the 3^rd visit, there were 15 impressions and 2 clicks.
Across the 3 visits, 2 visits had click-throughs.
In total there were 35 impressions and 3 clicks.

(And this is just for that Special page, we are not even considering what other things you might have clicked as part of your session while visiting other pages and interacting with other features.)

So what is your CTR? Is it…

8.6%? (3 clicks / 35 impressions)
66.7% (2 visits with click-throughs / 3 visits with opportunities to click-through)
7.8%? (average of 10% CTR on 1^st visit, 0% CTR on 2^nd visit, and 13.3% CTR on 3^rd visit?

Furthermore, in aggregate, your session as a whole might be considered successful because you clicked at least once. Perhaps not everybody clicks at all when they visit that Special page. Another type of clickthrough rate we might consider is:

What proportion of users who visit that Special page click on one of the links?

There are many ways to think about CTR and they all represent different user behaviors, answer different questions, and tell different stories. Ultimately it comes down to: what is it that you are trying to change?

Generic clickthrough rates

To make it easier for product teams to get started with experimentation, we have defined a few generic CTRs that can be reused across a variety of contexts:

Clickthrough per user: Generic metric and an instance of a unique clickthrough rate, with the unit being subject of an experiment. At the group level this is the proportion of subjects who clicked at all, regardless of how many opportunities (impressions) they had.; This would answer: What proportion of users who visit that Special page click on one of the links?; It is defined here in the metrics catalog MVP for automated analysis of experiments conducted with Experimentation Lab.
Clickthrough per page visit: Note: Requires performer_pageview_id contextual attribute in collected data (T395692).; Generic metric and an instance of a unique clickthrough rate, with the unit being a page visit by subject of an experiment. The metric is measured per subject and represents "Of all the opportunities we gave users to engage (page visits with clickable links), what percentage resulted in engagement (clicking)?" This is then averaged across all subjects in the group to yield a group-level measurement.; A user may have a CTPPV of 1.0 (every page visit resulted in a click), 0.2 (every fifth page visit resulted in a click-through, on average), or 0.0 (no page visits resulted in a click), for example.; It is defined here in the metrics catalog MVP for automated analysis of experiments conducted with Experimentation Lab.; This would be the 66.7% in the scenario above, at least for you (with 2 visits that had click-throughs out of a total of 3 visits to that page). CTRs for individual users can vary from 0% (never clicked) to 100% (clicked on every visit).; In the analysis of the experiment, we analyze the average of the groups. For example: if a group has 5 subjects with CTRs of 0%, 100%, 10%, 50%, 25%, the group's clickthrough per page visit is 37%.

These metrics are generic because they do not differentiate between impressions/clicks from multiple sources (instruments) of impression/click events that might be present in the data collection for an experiment. Metric calculation is unscoped, so it relies on the data collection to be scoped. An experiment must only produce impressions/clicks that are relevant, and no irrelevant impressions/clicks can be collected as part of the experiment.

If your data collection is not scoped (multiple UI elements produce impressions/clicks during an experiment) and you need to measure CTR for each of those UI elements separately, you would your metric calculation to be scoped.You do that by defining a specific CTR.

Defining specific clickthrough rate

When would you define a specific CTR? When you need to differentiate between impressions and clicks from different elements. How to go about doing that? Currently, defining metrics for use with automated analysis of experiments conducted with Experimentation Lab is done manually. Refer to the guide on metric definition for more information. This portion of the guide assumes you have read that section.

Consider this scenario: When user clicks Publish changes… button they are shown a dialog with 2 buttons: Publish changes and Review your changes. We have an idea to show a little message to the user next to the Review your changes button, prompting them to review their changes before they publish the changes. We think that it will lead to more users reviewing their changes, without hurting the overall rate of publishing changes. These are actually two CTRs:

Proportion of times the Review your changes button is clicked
Proportion of times the Publish changes button is clicked

For both we will consider success/failure at page visit level (similar to the "Clickthrough per page visit" generic CTR metric above) such that the number of times the user is shown the dialog on a single pageview does not matter. (Although we can certainly factor that into our metric definition if we wanted to.) We assume that after the changes are published, the page is reloaded and a new pageview ID (random token) is generated to differentiate it from the previous pageview.

Generic CTRs will not be useful here because they do not distinguish between which of the two buttons was clicked. To keep the example minimal, we will not include governance information (data stewardship) and just focus on the query templates. For our example, we assume that the two buttons were instrumented with the standard CTR instrument (explained in next section) and they have element_friendly_name set to "Publish changes" and "Review your changes", respectively.

- name: Review your changes CTR
  type: mean
  description: Proportion of times the 'Review your changes' button is clicked
  query_template: >
    WITH per_page_interactions AS (
      SELECT
        IF(experiment.assigned = 'control', 'control', 'treatment') AS variation,
        experiment.subject_id,
        performer.pageview_id,
        CAST(SUM(IF(action = 'click', 1, 0)) > 0 AS INT) AS clicked_review
      FROM {table}
      WHERE action IN('impression', 'click')
        AND element_friendly_name = 'Review your changes'
        AND {where_boilerplate}
      GROUP BY 1, 2, 3
    )
    SELECT
      variation,
      subject_id,
      AVG(clicked_review) AS outcome
    FROM per_page_interactions
    GROUP BY 1, 2
- name: Publish changes CTR
  type: mean
  description: Proportion of times the 'Publish changes' button is clicked
  query_template: >
    WITH per_page_interactions AS (
      SELECT
        IF(experiment.assigned = 'control', 'control', 'treatment') AS variation,
        experiment.subject_id,
        performer.pageview_id,
        CAST(SUM(IF(action = 'click', 1, 0)) > 0 AS INT) AS clicked_publish
      FROM {table}
      WHERE action IN('impression', 'click')
        AND element_friendly_name = 'Publish changes'
        AND {where_boilerplate}
      GROUP BY 1, 2, 3
    )
    SELECT
      variation,
      subject_id,
      AVG(clicked_publish) AS outcome
    FROM per_page_interactions
    GROUP BY 1, 2

Instrumentation

This portion of the guide is intended for engineers tasked with collecting data for measuring or experimenting with CTR.

The Experiment Platform team developed a standard CTR instrument according to CTR specification. Version 1 of the CTR instrument is attachable to a DOM element and – crucially – waits until the target element is visible to the user before it fires an action: "impression" event. This makes it possible to attach to a link in the footer (which is usually below the fold) without inflating impression count. It requires two arguments: a selector and an element friendly name (which will show up as element_friendly_name in event data):

const { ClickThroughRateInstrument } = require( 'ext.wikimediaEvents.metricsPlatform' );

const result = ClickThroughRateInstrument.start(
    '{DOM selector}',
    '{element friendly name}'
);

It also accepts an optional Instrument argument – including an Experiment – which must have a submitInteraction method.

const someInstrument = mw.eventLog.newInstrument( STREAM_NAME, SCHEMA_ID );

const ctrInstrument = ClickThroughRateInstrument.start(
    '{DOM selector}',
    '{element friendly name}'
    someInstrument
);

Example

This portion of the guide is intended primarily for engineers tasked with analytics instrumentation and experiment orchestration.

Prerequisites

This portion of the guide assumes you have read our guide to conducting experiments.

Background

The Collaboration list shows a list of ongoing events (may be collapsed by default) and a list upcoming events (expanded by default). It can be used by editors to discover edit-a-thons and other collaboration opportunities they may be interested in participating in. It can also be used by readers who would be prompted to create an account if they click on one of the listed events and decide to register for it.

Scenario

We would like to:

collect data to measure clickthrough rate on events on an ongoing basis, and
conduct an experiment where we include number of currently registered participants in each event as one of the event's details – alongside who organized the events and what type of event it is (online or in-person).

Hypothesis: Showing the number of registered participants will increase the clickthrough per page visit on this page.

Note: In practice we probably would not want want to run an A/B test because Collaboration list gets about 35 pageviews/day,^[1] and with a maximum traffic allocation of 10%, we would have to keep the A/B test running for a long time to obtain a large enough sample to make the results trustworthy.

Analytics instrumentation

Events will flow into the product_metrics.web_base stream and will be queryable from the product_metrics_web_base table in the data lake.

Our instrumentation can go in collaborationList.js in WikimediaEvents and we would document it in OWNERS.md.

Each link to an event has the class ext-campaignevents-events-list-link, so we will use that to select the DOM elements.

For analytics we can attach the CTR instrument to each link:

const { ClickThroughRateInstrument } = require( 'ext.wikimediaEvents.metricsPlatform' );

const instruments = $( 'a.ext-campaignevents-events-list-link' ).each( function( index, element ) {
    ClickThroughRateInstrument.start(
        'a.ext-campaignevents-events-list-link:nth-of-type(' + (index + 1) + ')',
        'Collaboration list event link'
    );
} );

If you need to collect specific contextual attributes, you would use the following code:

const { ClickThroughRateInstrument } = require( 'ext.wikimediaEvents.metricsPlatform' );

const INSTRUMENT_NAME = 'collaboration-list-event-links-ctr';
const instrument = mw.eventLog.newInstrument( INSTRUMENT_NAME );

const instruments = $( 'a.ext-campaignevents-events-list-link' ).each( function( index, element ) {
    ClickThroughRateInstrument.start(
        'a.ext-campaignevents-events-list-link:nth-of-type(' + (index + 1) + ')',
        'Collaboration list event link',
        instrument
    );
} );

And you would then configure the instrument in xLab via https://mpic.wikimedia.org/create-instrument, making sure that the machine-readable name is consistent with the name used in the instrumentation code (e.g. "collaboration-list-event-links-ctr").

Experiment instrumentation: feature toggling

Let us call the experiment "event-participant-count" and call the treatment group "shown". (A control group is always reserved.)

For this specific feature there are two ways we can toggle it:

Option 1

For this particular scenario the simplest implementation is to render the participant count on every page visit, but then use our CSS classes to hide the participant count from everyone but subjects in the treatment group:

.ext-campaignevents-events-list-link-participant-count {
    visibility: hidden;
}

.xlab-experiment-event-participant-count-show .ext-campaignevents-events-list-link-participant-count {
    visibility: visible;
}

/* If we wanted to be more explicit:
.xlab-experiment-event-participant-count-control .ext-campaignevents-events-list-link-participant-count {
    visibility: hidden;
}
*/

Provided our hypothesis is supported by the data and showing participant counts does lead to higher clickthrough rate, this will make it easier for us to adopt the treatment as the new status quo and show the event participant counts to everyone going forward – just by removing those visibility rules from the CSS, without needing to modify the feature code.

Option 2

A more involved implementation is to render the participant count conditionally by using the PHP SDK to only render the participant count when the page is visited by a client that has been enrolled in the experiment and has been assigned to the treatment group. Refer to the guide on server-side instrumentation and feature toggling.

If our hypothesis is supported by the data, we would need to remove the toggling logic from the feature code and go through the deployment process.

With either option, if the hypothesis is not supported by the data and we decide to drop the feature, we would need to remove it from the codebase.

Experiment instrumentation: analytics

We can have two CTR instruments attached to each link – one for ongoing data collection (if clickthrough to events is one of our product health metrics) and one for experiment data collection.

Our experiment's analytics instrumentation can go in a separate file: eventParticipantCountExperiment.js

To collect data for the experiment, we pass the Experiment object to the CTR instrument:

const { ClickThroughRateInstrument } = require( 'ext.wikimediaEvents.metricsPlatform' );

mw.loader.using( 'ext.xLab' ).then( () => {
	const experiment = mw.xLab.getExperiment( 'event-participant-count' );

    if ( experiment.isAssignedGroup( 'control', 'show' ) ) {
        const instruments = $( 'a.ext-campaignevents-events-list-link' ).each( function( index, element ) {
            ClickThroughRateInstrument.start(
                'a.ext-campaignevents-events-list-link:nth-of-type(' + (index + 1) + ')',
                'Collaboration list event link',
                experiment
            );
        } );
    }
} );

What this does: ClickThroughRateInstrument can accept an Instrument that has a method submitInteraction. Experiment uses send for producing analytics events but includes a proxy method submitInteraction for compatibility with this version of the CTR instrument. When the CTR instrument produces an impression or click event, it sends it through Experiment which handles the rest.

FIXME: Update example code once T395144: ClickThroughRateInstrument v1: Handle multiple elements gracefully is resolved

Experiment configuration

When configuring/scheduling the experiment in xLab, we would select edge_unique as the identifier type and set default user traffic per project to 0% and then to 10% (the maximum allowed) on meta.wikimedia.org specifically, which would make the experiment active on Meta wiki (for 10% of traffic) and nowhere else.

Experiment evaluation

See guide on automated analysis of experiments for more details.

After we register our experiment for automated analysis by adding this entry to the registry:

- enrolled: event-participant-count
  table: event.product_metrics_web_base
  metrics:
    - Clickthrough per user
    - Clickthrough per page visit

The results of our experiment – once it is active and collecting data – would be viewable at https://superset.wikimedia.org/superset/dashboard/experiment-analytics/.

Experiment aftermath

If you decide to ship the change, remember to remove the experiment code (whether that's server-side feature toggling or CSS rules) and remove the experiment-related analytics instrumentation.

References

↑ https://pageviews.wmcloud.org/?project=meta.wikimedia.org&platform=all-access&agent=user&redirects=0&range=latest-30&pages=Special:AllEvents

[1] ttps://pageviews.wmcloud.org/?project=meta.wikimedia.org&platform=all-access&agent=user&redirects=0&range=latest-30&pages=Special:AllEvents

[1]