Internet Scale Experimental Design and Deployment for Educational Games using BrainPOP®

Large-scale online design experiments (A/B testing, multivariate testing, etc.) are increasingly important for online gaming sites that seek to optimize game mechanics, but for most developers the ability to collect data and run experiments requires custom experimentation and data logging systems. We present an approach that combines a novel digital number line game and a free game portal popular among teachers and students to allow multiple experiments to be executed simultaneously. A pilot study conducted over six months resulted in the collection of millions of data points representing hundreds of thousands of participants in dozens of experimental conditions. These results indicate that this approach can be successfully implemented at a large-scale. Furthermore, analysis of player logfile data revealed insights into learning and game play that can inform the next round of experiments creating a continuous feedback loop. Finally, limitations of the study as well as future work are discussed.