4 Software Quality Lessons from Pokemon Go’s Wild First Week

Pokemon is near and dear, not just to the hearts of 90s kids, but pretty much everyone who ever played video games. Pokemon Go triggered the nostalgia and took smartphone gaming to the next level.

In fact, Pokemon Go is already the biggest US mobile game of all time, with 21 million active users.

Pokemon Gif

Pretty much everything that could be said about the game has already been said — from how it’s just the beginning of augmented reality gaming on mobile, to how its putting lives in danger and need to be stopped. Some even went ahead and debated the ethics of Pokemon Go, and whether or not we want games to have access to user data or own the virtual space around your home.

But what made the most news about Pokemon Go is the issues people faced with it. I myself couldn’t login using my Google credentials because it ‘Failed to connect to the server.’ When I could eventually login, I couldn’t catch a Bulbasaur sitting two feet away from me because it ‘lost GPS connection’. There are a number of posts already available listing down all Pokemon Go problems and suggesting possible solutions. Google Play and iTunes are flooding with apps that try to ‘fix’ Pokemon Go problem and keep your hunt ongoing.

But this doesn’t stop us from asking a simple question: why so many problems?

Were the game creators not ready for such overwhelming response? Or did they not test all possible use cases — including geolocations, devices, operating systems, hardware, diverse transaction etc. — before launching?

Almost every day of its first week, the Pokemon Go servers have crashed and people have reported numerous other problems with user experience. Thankfully nothing has affected the virality of the game as of now. Niantic, the company behind the game is already learning a lot from every hurdle and has fixed multiple issues already. However, considering that it just launched in a few countries for now, and is set to go worldwide soon, the game makers are in for even bigger and unforeseen challenges.

We are all about software quality at SmartBear. So we decided to list down our suggestions to the game creators and someone who’s trying to learn from Pokemon Go’s obstacles and develop the next viral game!

Lesson #1: You are as strong as your weakest link(s) — testing and monitoring the APIs

Pokemon Go is all about APIs. Right from logging into the app to identifying locations, seeing augmented reality, collecting pokemon, battling in PokeGym or just walking; everything is fueled by APIs that transfer data and requests. Some of those APIs are internally developed by Niantic and some of them are popular public APIs such as Google geolocation APIs.

The CEO and founder of Niantic, John Hanke, is an ex-Googler who worked on Google Earth and Google Maps initial design and development. So let’s say he knows a thing or two about geolocation mapping and it’s potential.

Many players faced problems where the game couldn’t detect GPS location, or in some cases detected wrong locations.

There are multiple problems here:

  • the availability of the API endpoint
  • the speed at which it responds to the request
  • the functional correctness of the payload

Since the whole game is based on geolocation correctness and preciseness, even a slightest error in the API payload can dilute the user experience. Hence, the game makers not only need to test the API functionality from all different angles before deployment, but also need to monitor the API for functional correctness from all diverse user scenarios – geographies, devices, operating systems and service providers.

Lesson #2: Validate user-generated data

Pokemon Go isn’t Niantic’s first rodeo in the augmented reality arena. In early 2011, they had launched a beta version of Ingress, another multiplayer augmented reality game, which let users pick landmarks and historical locations as ‘portal locations’. Ingress gave Niantic a database of over 15 million portal locations which were used in Pokemon Go for gyms and Pokestops. Thousands of Ingress users made sure that you can play Pokemon Go at almost any remote place. In Pokemon Go, users can submit Pokestops.

Crowd-sourced data is phenomenal when it comes to building an augmented reality game, however the data is still user generated and is naturally susceptible to human errors. This post about Pokestops in the strangest places will give you a mere hint of the problem. Unnecessary traffic in some of these locations can raise security and public health and safety concerns. Who should have control over this? Of course the game makers!

This is just one example of user-generated data and its implications, but we all agree that there should be some kind of validation and testing done on user-generated data.

Lesson #3: Test and monitor the front end

Pokémon shows the increasing dependency of mobile apps on back-end APIs. In this case, the APIs and GUI are interdependent on each other to collect information. Hence, testing apps both at the API and GUI layer becomes important.

The front-end graphical user interface (GUI) of Pokémon depends not only on the API responses, but also on mobile device hardware/sensor data. For instance, in this case, the location of the player is determined by location-based sensors GPS, which in turn determines the GUI behavior of the game. An integrated GUI and API testing tool can help resolve the first issue. An ability to pinpoint issues at the GUI or the API layer allows  GUI testers to cut down on intermediary steps. For instance, on realizing the issue is at the API layer, the GUI tester can reach out to the API developer without going to the GUI developer first. This saves testing time, which is critical in the case of mobile.

At the same time, to create a solid test automation strategy that scales, the focus should be on ensuring optimal behavior at GUI and API layer. This can be done by focusing on a test automation pyramid strategy, which is based on creating more unit tests, followed by API tests, and then fewer UI tests.

The game makers have big plans for this game post the worldwide launch, such as advertising and boosting retail traffic. The game is already creating news around how it is boosting foot traffic for local businesses and can even help sell houses. As this game becomes more important piece in the advertising industry, front end errors or usability issues wouldn’t be taken lightly.

Lesson #4: Prepare your app for all possible user scenarios

Just like every other app, gaming or not, no two users are alike. They access the app from different channels, different locations, and different devices but yet expect seamless performance. And they get easily frustrated when the app fails to do what they want it to do.

Irrespective of the problem, 9 out of 10 times they blame the app for their bad user experience. When I first got Pokemon Go, it drastically drained the battery on my Android. Even after trying the battery save mode for a few days, I eventually had to delete the game to make my phone usable again. Per Think with Google research, 72% users delete the app permanently after initial bad experience. While only 23% of the consumers give an app a second chance.

While negative out of the box experience slowing down the adoption may seem like a faraway problem for Pokemon Go right now, as the virality of the game calms down, it could be a deciding factor for Pokemon Go’s future.

I strongly think that the game makers are just getting started and there are some amazing things in store for the game makers and us as augmented reality games gain popularity. There are multiple possibilities for implementation and growth across all verticals using augmented reality, and Pokemon Go is doing a great job of giving us a taste of it. While we don’t know what the future of this game is, we hope that Pokemon Go makers are learning from each and every experience and those wishing to follow their success consider our tips for software quality.

Happy catching ‘em all!

Comments

  1. Carl Shaulis says:

    Perhaps there is a different set of lessons.

    1. Follow Lean Startup principles
    2. Get MVP out as quickly as possible
    3. AB Test your way to continuous improvement
    4. Gather vast amounts of data and learn even more

  2. In the short duration that Pokemon Go has been out, its already been attacked by hackers calling themselves OurMine. Its worth noting that their servers had crashed even before the attack.

    People have been reporting issues with battery life after playing the game. Sometimes the game is not able to find the GPS signal. Sometimes the game freezes.

    In a large scale release of this kind that goes viral, Nintendo should have planned better for scaling its platform. Most users will simply uninstall the game if it doesn’t work the first couple of times.

  3. Bart Wijnen says:

    Turn it around. The launch of the Pokemon GO app also indicates that it’s not neccessary to wait untill everything is tested thouroughly. Release it when it’s good enough and let the crowd test your application.
    This game is strong enough to release with the issues mentioned. Now all the players help to detect these issues and the worse issues are turned in to free publicity.
    Off course you still have to act but think how many effort it would have been to detect these issues in front with only a small number of users testing the app.

    I think it’s a well thought strategy and in this case well placed to launch the app with these issues. The game was simply ´Good enough’

    The 4 lesson are valuable for every development, yet you have to know your impact and choose wisely when to stop testing and start exploring in the crowd.

Speak Your Mind

*