Today we’re happy to announce the beginnings of support for A/B Testing, possibly the most frequent new-feature request we get at PlayFab. The feature we’re releasing today is fairly limited, but we will be rolling out broader support for A/B testing throughout our entire platform in coming weeks and months.
What is A/B Testing?
A/B testing is a technique for running experiments to determine the optimal setting for a particular variable. For example, let’s say you want to test out different sale prices for an item in your in-game store.
With A/B testing, you create a series of player “buckets” into which you partition your players randomly, with weights assigned to each bucket. Each bucket is assigned a different price. Players see the price assigned to their bucket. You then compare the results for each bucket, to determine the most effective price.
Consider this test, for example:
|Price per item||$1.99||$2.99||$4.99|
|% paying users||2.1%||2.0%||1.8%|
Here, bucket “C”, despite having the highest price, ended up generating the higher average revenue per user, and would appear to the be “winner” of the test.
How PlayFab Supports A/B Testing
We are building support for A/B testing on top of our player segmentation feature of PlayStream. Each test bucket is implemented as a player segment. So if you are running an A/B test called “Price Test” with two buckets, “A” and “B”, then we will create two new segments, “Price Test - A” and “Price Test - B”. All players will be automatically assigned to one of these two segments, based on the bucket weights you selected.
Today, the only feature in PlayFab that can directly show different players different results based on segment is the targeted store override feature of the in-game economy. Effectively this means the only thing you can A/B test today are stores (including both prices, and collection of items). However we will soon be adding the ability for other PlayFab features to target different data to different players, such as title data.
Once a test is turned on, we will start generating a daily report for basic KPIs based on bucket. This will allow you to determine the effectiveness of your tests. Later, we will make it possible to do more detailed analysis using an external analytics provider, like Appuri.
Creating your first A/B test
After signing into the PlayFab Game Manager, click on the PlayStream tab, and then on the A/B testing sub-tab.
From here, you can create your first A/B test. At test creation time, you’ll need to provide a name for the A/B test, the number of buckets, and a name and weight for each bucket. The weights must add to 100, and the names must be unique. The corresponding segments are then automatically named ‘<test name> - <bucket name>’, I.E. ‘Gem Price Test - Control.’
Note that bucket assignment is probabilistic, so the populations of each bucket may vary by up to a few percent of the specified weight. This evens out as your player base grows.
Hooking your test up to stores
Once your test is set up, head to the Store section in your Economy tab. From here, you’ll be able to set overrides for a store.
See our store segment override blog for more details on this.
Once you’ve created a test, we will begin to generate a report called “Daily A/B Test KPI Report” each day. This will give you key KPIs by player bucket for each test you’re currently running.
We recommend letting this report run for at least a few days before configuring store overrides, to get a baseline. We partition your users based on their PlayFab IDs, which are randomly assigned at creation time. Due to the probabilistic assignment of players, the it is unlikely that the KPI will be uniform at the beginning of the test.
Once you're confident about the results of your test, you can then reconfigure your primary store and safely delete the test.
Future Work on A/B Testing
Today, Stores are the only PlayFab feature area that have segment overrides. We have plans for overriding Title Data by segments, as well as other areas. This will look much like store overrides, where title-wide variables will be configurable based on the segment of the user requesting title data. This will allow for tuning a much wider variety of features for your title.
We also have plans to allow you to run A/B tests within an existing Segment, instead of for all players. So, for example, you could try running an A/B test on pricing that applies only to members of the VIP segment.