Skip to content

A/B Testing Mediators

After you successfully set up the SDK, you’ll want to compare it against your current mediation solution to make sure that X3M yields better results. An A/B test is an objective way to achieve this task.

There are plenty of tools available to do A/B tests in simple steps (See A/B Testing with Firebase) in case you haven't run one before.

Whether you use your own A/B test system or not, there are a number of good practices you should follow to ensure accurate and statistically reliable results.

A/B testing best practices

  • Due to some placements warm-up time, we recommend running the test for a minimum of 14 days. Many ad networks behave unpredictably the first few days.

  • Include both mediators in the same app build.

  • It’s a good idea to be able to configure the test traffic distribution remotely (for instance using Firebase Remote Config).

  • We recommend distributing traffic from each country 50/50, otherwise it might skew your results.

  • Make sure that each user is always assigned to the same AB group during the duration of the A/B test.

  • Avoid splitting your users in code using a random number generator. Instead use a remote configuration service with values that can be changed dinamically.

  • Make sure that only one mediator SDK gets initialized and used during the whole user session.

  • Use the same partners or ad networks in both A/B test groups.

  • Avoid using the same ad unit for both mediators to isolate X3M traffic metrics for easier performance comparison.

A/B testing with Firebase

If you haven't done an A/B test before, Firebase is a good starting point. It provides a broad set of tools that are simple to use.

This sections covers the usage of only one of them, Firebase Remote Config, for choosing a mediation service, assuming that you already have an analytics service.

Adding Firebase configuration retrieval to your app

The first step is to add a small piece of code to retrieve Firebase Remote Configuration values and initialize the mediation according to this configuration.

The following steps summarize this process:

  1. Make sure Firebase SDK is added to your project. See the official docs for more information.

  2. Add Remote Config to your app. See the Get Started section for more information.

    Default values

    We recommend setting default values in your app in case Firebase fails to fetch the configuration.

  3. Add the required code to retrieve the configuration. See this section in the docs to learn how to get the properties.

    For example, we can expect a boolean property init_xmediator to be true indicating that we must intialize X3M's mediation and false indicating to initialize your current mediation.

Your app is now ready to fetch configurations from Firebase and decide which mediation service should be initialized.

Considerations

  • Firebase already manages the persistence of the configuration, so no additional caching code is required.

Creating the configuration in Firebase console

Now that the app is ready to read configurations remotely, we'll create a condition for the A/B test.

  1. Go to Firebase console, select your project and then go to the Remote Config section.

    Firebase Remote Config

  2. Create a parameter indicating whether your current user should use X3M or not. This parameter must have the same name as the one used in code and don't forget to also add a default value.

    Firebase Remote Config

  3. Create a new condition that will split your users and decide which of the AB test groups they will fall into, or in other words, which users will use X3M and which your current mediation. As stated in our best practices section, we recommend a 50/50 distribution, as seen in the image below.

    Firebase Remote Config

    Firebase Remote Config

    Distribution value

    If you don't feel confident enough with going 50/50, you can start with a lower number to do a technical validation first.

  4. Then assign a value depending on the recently created condition.

    Firebase Remote Config

  5. Save your changes and a button will be enabled in the upper-right corner. This button will publish the changes.

    Firebase Remote Config

    Firebase Remote Config

    Configuration TTL

    Firebase may delay delivering configuration updates to your users. This is because the mobile SDK caches the configuration in the device for 12 hours by default. Refer to Firebase docs for more information on how to change this behaviour for testing purposes.