Testing the Internet of Things

Testing_Internet_of_ThingsThe mobile revolution flipped testing on its side. Suddenly teams and testers had to think about connectivity in new ways: power sources, the impact of touch technology, system interruptions, bandwidth, and dropped connections. Then came the collaborative revolution and Testing adapted again. But this time, Testing faces a new challenge: the internet of things, or IoT.

What is the IoT, and Why Does It Matter

While phones and tablets are mobile, they are essentially still “just” personal computers that happen to fit in your pocket. The Internet of Things covers all of those other things connected to the internet, like your thermostat, automobile, or washing machine. All of these tools existed before the internet; so we are simply adding the internet to old things. The smart watch is a great example of an IoT device; people are using them to track not only the length of the run, but also their heartbeat and vitals during the run. The watch uses GPS and a wireless connection to upload the data to the internet, providing graphs and raw data for later analysis.

Two years ago, the smart watch was something that appeared in Dick Tracy comics. A year ago, it was an expensive luxury that conveyed social status. Today you can buy a smart watch atSmartWatch WalMart for about a hundred dollars, or a FitBit for a little less.

The IoT is quickly becoming something the world relies on to drive, report, and optimize performance from the timing of your run, to the gas mileage of your car. And it still needs to be tested.

Start with Usability

Done well, the IoT is deceptively simple; your wrist band might have a single start/stop button to record a run. Things have to be this way because while the devices vary in size, the displays and buttons are generally very small – somewhere between four square inches and a quarter of a square inch. That makes usability supremely important. While software that runs on a browser can give hints and subtle clues about how to use it, IoT devices have no room for subtlety.

When I first started looking at usability, the topic seemed like a mystery, more art than science. Explanations of usability seem easy to understand, even intuitive, but that does not provide a method to actually do usability testing on a new product. The ability to almost instantly recognize functionality and how to use it is crucial for consumer products. There is limited display and functional space, that translates to a need to make things obvious.

Minimalism is one thing that you can look for, more specifically, you can look for the things that are missing. Minimal user interfaces are free of extra information that might be in a normal web page — tool tips, formatting examples, general help.

A lot of these principles are described in depth in the book The Design of Everyday Things by Donald Norman.


I have a Nest Thermostat in my house. The thermostat is incredibly simple on the outside. There are two main ways a person can use the Nest; you can turn the knob, and you can click it. With these two functions, you can do everything you need to do. And you don’t need guidance to figure it out.

On the inside, the Nest is collecting data about local weather conditions, controlling the temperature in my house, and sending and receiving data about energy usage. It is all neatly hidden away behind a slick, usable interface.

Devices, Devices Everywhere

There are thousands, maybe hundreds of thousands of different internet connected products now. All of them are collecting and sending and displaying data all the time. Some companies create and sell their own devices, like the Nest. For these companies, it’s not a big deal to have all or most of the important physical devices on hand. Others create applications or serve data to many different devices, and even if they could identify every hardware / operating system possibility, they couldn’t possibly have them all available for testing.

Companies in this second category have the challenge of finding a minimal reference set to test on, or rotate what devices are used for testing. One way to figure out what to test on is to get a prototype or early version out fast, even if it is just to employees. Once the device is out, get usage information (Google Analytics is free) to figure out what devices are most popular; then select the top devices to test on.

With real information from meaningful customers, you can make educated choices about which devices are important to have, and which might be OK to leave out of the testing rotation for now. Companies that have usage information (or can make informed guesses), can make a list of supported devices, just like software in the 1980’s used to have a list of supported computers.

Sometimes, like with Netflix, you have no control over the devices a service is used on. Communication with the people consuming your service is important in these cases so they can stay in sync with the API and data you provide.


Internet connectivity is what makes this network of things possible, but in many cases that network is intermittent or unreliable. It is important to know that your data will be saved and stored correctly if you unexpectedly lose a connection, and is delivered correctly when your connection is restored. You can simulate this by using airplane mode or temporarily disabling the internet connection on the device, sure.

Even better than simulation is taking a walk in and out of wifi connections and seeing what really happens when you really lose a connection.

Growing Security Concerns In A Connected World

According to a HP study, 70 percent of devices in the Internet of Things are vulnerable to security problems — missing data encryption, minimal password requirements, and access to the user interface without passwords. Security through obscurity, the idea that being secretive about how a product is implemented, may have been good enough when product lines first emerged. The IoT isn’t exactly obscure anymore, and many devices have large customer bases.

Don’t wait till there is a headline news story to understand your products’ security short comings.

Tailor Your Approach for IoT Testing!

There is incredible variety in devices, software, the people that use them, and the needs these products are trying to meet. ‘It depends…’ isn’t a great answer to the question of what you should focus on when testing the the things of the Internet of Things, but it won’t lead you astray. Starting your test strategy with an understanding of what your customers value will be a powerful first step.

Some devices like internet connected storage will have you focusing on performance and security, for medical devices precision of data collection and delivery is important, and gaming consoles should be entertaining. Never forget who your customers are.

The IoT is still a developing market; things are going to change a lot in the next few years, especially in the area of security.


  1. Subhasri Duttagupta says:

    Enjoyed reading the article. In the section ‘Devices, Devices everywhere’ you rightly said that an IoT application cannot be tested on all the different devices in different contexts. That’s precisely the reason why building performance model would be a great step towards assuring performance of these applications. In TCS innovation labs, Mumbai we are taking this approach to come up with performance model that can guarantee QoS. Input to these models would come from device model, communication network model used by the devices, other systems interfacing with the devices. It may be difficult to have all these models in place to start with but the research effort can identify the dependency of application performance based on the performance of underlying systems. In order words, we can identify the sub-components of the performance model for IoT application.

  2. jon hagar says:

    These are all good points, esp security (privacy), but we seem to have more “points” and few solutions. We are building and deploying these things like it is the wild wild west of the old .com days. We need to be smarter skilled testers this time.

  3. The Internet of Things is a computing concept that describes a future where everyday physical objects will be connected to the Internet and be able to identify themselves to other devices.

Speak Your Mind