One of the primary concerns of any project manager throughout the testing period is whether there’s enough tester engagement. Good tester engagement is instrumental to a successful delta test. But, how do you measure that engagement? Some programs are focused purely on tester feedback submissions — the more feedback, regardless of quality, the more successful they see the test. Other testing programs may focus on tester logins and user testing survey completions while some are laser focused on product usage as the end-all metric for delta project success. Wouldn’t it be nice to review a simple grade or score and know at a glance how your project is doing?
In Centercode, we measure engagement via Delta Health. Health directly reflects how much information you’re getting from testers, how well your testing has covered a given feature, and how confident you can be in the data received. Our platform has been built with all of this in mind and provides a quick and easy way to not only calculate your project health, but also observe and measure project success real-time via several dashboards.
How Centercode Helps Measure Project Health
To begin calculating Centercode’s Delta Health, we need to know your tester target, or number of testers you will target for this test. This is defined during project creation and accessible via your basic project settings. If you’re not sure about your tester target, Centercode provides a helpful recommendation based on project information you provide to the system. This target value is foundational to your health score because it sets the expectation for how much engagement we should expect from our tester team.
Testers primarily provide information to you via activity completions and feedback submissions, which are the other two components of the Delta Health score. When testers complete an activity, they indicate their satisfaction on a scale of 1 - 5 stars and based on that rating, the platform prompts them to submit either an Issue, Idea, or Praise about the feature. Half of our health calculation is based on the number of activities completed and the other half on feedback submitted. This means that regardless of the amount of feedback submitted, you’ll still need your testers to complete activities to reach a high health score. Utilizing both of these indicators ensures the Delta Health score is resistant to undue influence from a number of situations that can occur in testing (e.g. only a small group of testers submitting large amounts of feedback, unengaged testers only giving star ratings without corresponding feedback, etc.)
Health scores start at N/A and range from D to A+ following a standard letter scale. Don’t be alarmed if all you see are N/A’s at the start of your test, or when viewing a testing phase that just kicked off. It’s expected that Delta Health will start out low and build throughout the testing period. Remember that Delta Health is about the project’s health, it isn’t an assessment of your product. A project with a low health score doesn’t indicate that your test product is poor, it just means you can’t be confident yet because you don’t have enough data. Similarly, an A+ Health rating doesn’t mean your product will get five stars when it’s publicly released.
Reviewing Project Health Dashboards
The project health KPI is observable at three levels in Centercode:
First, the highest level, Delta Health, is viewable on the project overview and relates to the total combined health of all features in the project that are assigned to a particular testing phase.
Second, if you take a step lower into a phase overview dashboard you’ll now see Phase Health. The underlying calculations are similar but we’re only observing the activities and feedback related to the features assigned to that phase of testing. In the later phases of a test you may notice that you begin the phase with a relatively high health score. This is likely because the feedback portion of the Delta Health calculation is already high from existing user testing feedback in the form of issues, ideas, and praise.
Lastly, you can view an individual feature via its dashboard to see how it’s doing. Remember, only features assigned to a phase will have activity completion so don’t be surprised by low health scores for any phaseless features you have.
How to Use Delta Health to Manage Product Testing
Health isn’t just some passive feature that affects how success is calculated. It can, and should be part of what helps you plan and react during your tests, particularly if they’ll run for several weeks.
For example, you may have included a specific feature during your first phase of testing, and now three weeks later you still only have a D+ rating for health. This can be a strong indicator that users are having some sort of issue testing, likely blocked from completing the activity, or unenthused about submitting feedback. Even negative feedback builds your Delta Health score, so a low score means they’re not telling you anything, good or bad. Low health should indicate to you to consider moving that feature to a later week of testing to give users another chance to experience the feature and collect additional feedback.
Health touches a number of places on the Centercode platform and serves as a quick measure for how well your testers are engaging. It directly influences the overall success score, and can be viewed at the project, phase, and feature levels. Health should give you confidence that your project is running smoothly or flag potential problems that are preventing users from testing.
Check out Rob's write-ups on the remaining Delta KPIs:
