This week we began rolling out a new automated build, test, and deployment tool for our SDKs, called the Jenkernaught. We needed the Jenkernaught, because our SDK complexity has been rising, and with complexity comes a higher risk for bugs. We’re now up to 13 SDKs, written in 7 languages, running on 8 platforms -- and these platforms themselves keep changing (Unity 5.3.5! Unreal 4.12! Lumberyard 1.2!). Hence the need for greater automation.
Because we think this code may be useful to other game developers, we are releasing the code to the Jenkernaught as part of our previously released SDK generator -- you can find it in the “Jenkins Console Utility” folder: https://github.com/PlayFab/SDKGenerator/tree/master/JenkinsConsoleUtility. You are welcome to use the Jenkernaught for your own purposes, though it’s still a work-in-progress, and not yet documented.
What exactly does the Jenkernaught do? It’s not like we weren’t automated before. We already relied on Jenkins to manage our suite of automated SDK tests. Jenkins is really good at running sequences of console commands, and programs to build and deploy software. It has plugins that allow it to interact with Visual Studio, or display pretty test report graphs.
However, we had a hard time building, running, and returning test results from "difficult" targets, such as browsers or phones, back to Jenkins. These targets don’t have the capability to directly communicate their test results back to Jenkins, which means it was hard to automate tests on those targets. And we needed a system that could deploy builds and get test results from Windows, Linux, Mac, Playstation, XBox, Android, iOS, Windows-Phone, and more.
This is why we built the Jenkernaught. Jenkernaught is designed to receive test results from devices, and relay them back to our build system so the test results can be combined and evaluated in one place. Ultimately, this allows us to run more automated tests on a wider variety of targets, and get test results about those targets without requiring a human tapping on a physical phone.
The Jenkernaught combines code from Jenkins itself (https://jenkins.io/), previously shared PlayFab code (e.g., https://api.playfab.com/sdks/sdk-generator), and new code (e.g., the Jenkins Console Utility). The Jenkernaught has also required us to make a few changes to our SDKs, particularly in our example projects included in each SDK. These example projects double as our automated testing projects, used by the Jenkernaught.
We will be making frequent changes to these frameworks as our testing system evolves, but will not call out these changes as "breaking changes" in the release notes since they're not part of our primary product.
You’re welcome to use these frameworks, of course, perhaps even adding automated tests of your own. Just know that if you do rely on these frameworks, you may need to make changes from time-to-time. If you have any issues with any test framework after upgrading an SDK, please let us know in the PlayFab community forums (https://community.playfab.com), and we'll help to sort you out.
Now that Jenkernaught is released, we hope you will see improved stability in our SDKs, resulting from greater cross-platform testing for the PlayFab SDKs for all of our various devices and platforms.