Nightingale game developers did something many other applications do – they gathered early feedback and used it to help improve the product. What’s particularly interesting is all the different methods and combinations Nightingale developers used.

As you might expect, some of the feedback was simply changes to the balance of certain mechanics if someone felt something was too hard or a mechanic simply wasn’t being understood. But the team went beyond just that by pairing direct feedback with telemetry they were gathering.

Instead of a standard Discord channel that required managers to sift through buckets of messages to collect the few jewels, they let users make suggestions and let people crowdsource and advocate issues to see if others felt the same way. This alone helped get clearer signal over the noise. The devs went further and compared that to the metrics they gathered to see if it was true. They would track things like:

• Retention rate.
• Did people who played on the first day play on the second day?
• How many players logged in?
• How long did they play, and what was the average time played?
• How far did they progress?

When the metrics were compared and combined with the active vote topics in Discord, they were able to combine quantitative analytics to the qualitative Discord feedback. If there seemed to be a point large numbers of people stopped playing, the devs could go to the forums to ask why and get qualitative feedback. This helped not only identify problems – but also get information, or even suggestions, as to how to correct them.

The idea is not new. Others have tried using metrics – and nobody probably more than the gamification methods in Duolingo. I think there is a point at which you go too far; so it’s definitely worth watching and drawing some conclusions.