This one has been hard to keep under wraps, but after months of hard work, we’re excited to announce the public preview of PlayFab Experiments. In 2016, we launched A/B tests, a service that helped hundreds of PlayFab powered games discover how different store configurations impact player engagement. Now, with the launch of Experiments, we bring a new approach that ensures more robust statistical validity with boundless possibilities to experiment.
Identify the best strategy with Experiments
PlayFab’s Experiments service helps you identify the best strategies for your game. Now you can run concurrent trustworthy experiments in a managed and controlled randomization thereafter providing meaningful statistical analysis.
Elevate your player experience by comparing different versions of game configurations, pricing models, outreach mechanisms, and more. Determine what works without resorting to guesswork.
This new service is a tailored solution for experiments in gaming. It’s powered by a subset of the Microsoft internal platform used by Bing, Office, Xbox, Minecraft (and others) for best-in-class experimentation.
Simply experiment by mapping configurations to a single control and multiple treatment variants, monitor their performance based on player behavior, and then select a show-stopping variant as a winner based on your goal conversion metric(s), and you are all set to confidently roll-out the fitting configuration in the game!
This whole experiment experience is achievable with PlayFab’s Experiments service. Thus, allowing experimenters to:
- Create and manage multiple concurrent experiments with the (interactive and self-directing) user interface on PlayFab Game Manager and via scalable APIs.
- Experiment to compare configurations. Each configuration is easily mappable to variants where each variant is supported by variables. These variables allow you to bundle a different set of game experiences via parameterization.
- Target your desired audience at random but in a controlled manner by making use of existing traffic. You can experiment over a segment, and define the percentage of the target audience in each variant.
- Schedule an experiment or run immediately.
- Analyze experiments’ scorecard results. The scorecards are computed reliably and possess enriched statistical details. This gives you an indication of statistical significance and alerts if a Sample Ratio Mismatch is observed
Let's take a deeper look at the Experiments service.
If you haven’t used A/B testing before, you might want to read our guide which details everything from why you should experiment, to the best practices and getting comfortable with the process.
Ready to get started? Sign on to Game Manager and follow the steps below (or refer to the documentation for more details).
Experiments support both A/B testing (also known as split testing) and multivariate testing.
A/B testing is a method of comparing two or more configurations of game experience (variants), whereas multivariate testing allows you to configure several game elements and determine which combinations (variants) perform best at achieving your conversion goals.
Experiments service is available on the left navigation, under Analyze. To create an experiment, click "New Experiment".
Create an experiment
Enter an Experiment name and Description, then pick a Segment and experiment schedule using Start Date & Time, and Duration.
Under Settings, define Control variant and Treatment variant(s) for comparison between configuration(s). Each variant is supported by Variables, you can bundle different set of game experiences via parameterization using variables. Use % of target audience to select desired audience traffic in each of the variants.
You can "Schedule” the experiment to run based on the configured schedule, or use "Run now" to run the experiment immediately.
On the Experiments page, you can find all available experiments based on the status (Running, Scheduled, Stopped/Completed), and can Run, Stop, Clone, or Delete.
To analyze the effect of an experiment, select "View Scorecard".
View and analyze Scorecards
On the Scorecard page, you’ll find the reliably computed performance results for each variant of your experiment. The pre-defined and actionable metrics provided are based on PlayStream events (player_logged_in and player_realmoney_purchase). They are:
· Average logins per user
· Payer conversion rate
· Average revenue per unique user
· Day N retention
These metrics measure the impact on the acquisition, activation, retention, and revenue of your game.
This scorecard is generated on a regular cadence and is designed to be both comprehensive and easy to understand. The trustworthiness of results is ensured by providing metric computation with statistical significance. If a Sample Ratio Mismatch (SRM) is identified, it is flagged to the user for investigation. (Refer to the documentation for more on SRM).
Integrating Experiments with other PlayFab services
Experiments is compatible with Player Profile, PlayStream events, CloudScript and Insights Explorer. For example, you can:
- Make configuration changes using CloudScript: Virtually any configuration-related game code variation can be enabled using a combination of CloudScript and Experiment’s getTreatmentAssignment API.
- Do a drill-down analysis using Insights Explorer: A unique identifier of each variant (VariantID) is stamped on all PlayStream v2 events and two specific PlayStream v1 events (player_logged_in and player_realmoney_purchase). This can be used to do further analysis on specific metrics. (The metrics can be based on the PlayStream events or your own game telemetry.) To query, use the Insights Explorer service or connect your own analytics platform.
Support for the existing A/B Tests service
The existing A/B Tests service is still available and will be fully supported for at least three months. Eventually, we will deprecate this legacy service, though existing tests will continue to function for titles using it after that point.
The next iteration involves the integration of Experiments with other PlayFab services such as Title data, Commerce, and others, such that there is no need for change in server-client side code for configuration of game experiences. Also, we plan to bring the ability to define custom goal metrics for an experiment and other functionality.
What this service means to you
We’re excited that PlayFab Experiments help you figure out what works best for your players with empirical data insights. Visit the documentation site to learn more and get started!
If you have questions or feedback, we would love to hear from you. Please leave a comment in our Forum.