What are Metrics in Game Data Science?

The terminology around game data science is a bit confusing. The field still doesn’t have an agreed-upon standard definition for many terms and concepts. Hence, in this lesson, a brief glossary is introduced to denote some of the concepts utilized in this course. We must emphasize that these are our definitions. While they are based on the terminology used in the industry and the field, there’s substantial variance in how they’re used, so be prepared for these differences as we engage more broadly in game data science. Furthermore, ours is not an exhaustive list of terms, and we anticipate that these terms and metrics will evolve as the field evolves.

Telemetry, metrics, and KPIs

Telemetry data refers to data collected by a system that collects, transmits, and stores it over a distance. Telemetry data is usually collected in a raw form before it has been manually or computationally processed, and thus called raw data.

Game metrics are interpretable quantitative measures of in-game attributes or objects. If telemetry data is the information in its most raw form, metrics imply some interpretation. An example is a player’s total playtime as 15 hours and 30 minutes.

Key Performance Indicators (KPIs) are strategically selected metrics that demonstrate quantitatively how an objective has been achieved. KPIs usually require contextual information to draw conclusions from them and also entail some sort of ground truth to compare with. For example, knowing that a game has 100 Daily Active Users (DAU) is fine, but is it more or fewer than yesterday? Last week? Last month? And what is the reason for any drop/increase? While KPIs are adequate for forming a quick impression, analysts should be cautious about their use and resist drawing conclusions from KPIs that they cannot support.

Player, performance, and process metrics

There are three classes of metrics and KPIs: player, performance, and process metrics (see the figure below). Player metrics are related to the people who play games. It’s possible to view players as part of a community (relation to other players), as customers (sources of revenue), or as gameplay generators (agents of behavior). Examples of player metrics are total playtime per player, the average number of in-game friends per player, or average damage dealt per player. Common analyses include time-spent analysis, trajectory analysis, or social networks analysis.

Performance metrics are related to the performance of the technical infrastructure behind a game. Examples include client frame rate, server stability, client crashes, number of bugs, and concurrent users (CCUs). Performance metrics are heavily used in Quality Assurance (QA) to monitor the health of a game. It’s also one of the most mature areas of game data science because the methods employed are derived from traditional software performance and QA techniques and strategies.

Process metrics are related to the actual process of developing games and managing the creative process through development methods. In order to monitor and assess the development process, project managers utilize a combination of task-size estimation and burn-down charts or measure the average turnaround time of new content being delivered to the development pipeline.