System Metrics

Katana’s System Metrics are tools for developers to answer these questions:

What is being used?
What is working?
What's not working?

Each page of System Metrics focuses on a particular part of your application. These are pre-set categories populated when you integrate the calls. For example, one page is devoted to Levels, so if your app has the concept of levels, this page will make sense. There is also a custom-slot page that lets you report on any aspect of your application.

Each System Metrics page has a Global tab, which reports the basics of the category. For example, this page shows Levels, and reports on how many users are in each one, etc. We've also supplied a number of other metrics that we think make sense.

One of the simplest yet most powerful tools on the page is the ability to sort any of the columns. This will let you see the highest and lowest values.

Each page will also let you toggle the columns on or off. There may be some columns you aren't interested in or haven't instrumented. The "Export" button allows you to send the data from this page to another program. For cases where you have very long tables, you may want to choose which values to export. The export wizard will let you choose a column and a <,>,= value for it. You can do multiple ranges as well, e.g. more than this value on this column, AND less than this value on another column.

These same base metrics are then repeated in the other tabs (note their slightly darker shading), and put alongside additional metrics that fit those tabs. For example, the Churn tab shows which Levels are impacting churn.

Here there are three new columns at the far side, which follow the same pattern for most pages: How many people did the action, what % of users was that, and how long did it take?

This last column deserves some extra explanation as it's one of the more powerful metrics on the page. The "Time to <Action>" shows how long it took from when the event in the row (in this case, began level 14) occurred and when the user action (in this case, quitting) occurred. The final column is sorting the table in this example, and it shows that when a user reaches level 14 they take 27 days on average to then quit. Because this is a long time relative to the other levels, this suggests that there is probably something good about level 14. The other levels here lead to much faster quitting.

If the action were Conversion rather than Churn, this same long time span would be a bad thing, i.e. when the user has the event it takes them a long time to ever spend money. Perhaps that event is preventing the conversion. Or, if the conversion happens quickly, the event may be driving it and is a very good thing.

This same approach can be applied to all of the other actions: conversion, time spent, viral invitations, etc. And it can be applied to any of the System Metrics pages, including the custom events you implement. So, you can see how much the asset is being used, and whether it's associated with good or bad outcomes.

This is a simple, yet powerful way of seeing how any event in your system is being used and whether it appears to be leading to good or bad business, experience, or social actions.