Measuring the Usability of Programmable Thermostats

March 29, 2011

That fancy programmable thermostat on your living room wall, the one with a backlit digital screen, several tiny buttons, and unintelligible instruction manual, may be making it more difficult to save energy.

A research team at the Lawrence Berkeley National Laboratory (Berkeley Lab) is beginning to figure out why. They are determining how to measure the usability of a thermostat—that is, the ease of accomplishing the things people want to do with them, like switching them on, setting back the temperature while away, etc. This will lead ultimately to heating controls that do a better job of regulating your home's temperature as energy-efficiently as possible.

Programmable thermostats were originally promoted as an energy-saving technology that would help homes use heating energy more efficiently by regulating temperature automatically. People frequently forget to adjust their older-technology manual thermostats whenever they leave the house.

But in 2008, the ENERGY STAR program concluded that homes with programmable thermostats were not saving as much energy as expected and might even be using more compared to homes without them. The suspected cause was that people were programming their thermostats incorrectly, or not bothering at all. They terminated the existing thermostat program and began developing a new specification.

Alan Meier, of Berkeley Lab's Environmental Energy Technologies Division, began a project to understand how people actually use thermostats in their homes, and develop a way of quantifying thermostat usability.

Meier's team first sought to document the extent of current usability problems. One survey revealed two striking facts: half of the homes had their thermostats set to long-term hold, and 20 percent had the wrong time of day set on their thermostats' internal clocks. Long-term hold erases the energy efficiency benefits of programming a temperature setpoint by leaving it at the same temperature, no matter the day or time. Leaving a thermostat set to the wrong clock time has a similar same effect. It negates the programming that's intended to turn the heat down at times of the day when not needed.

The team gathered this data through an online survey they designed using the Amazon Mechanical Turk program. This tool allows those who respond to receive a small payment after carrying out a set of tasks. The responders were asked, in this survey, to take and upload a picture of the home's thermostat, (which supplied the actual clock time), and answer some questions about the device's settings. In 24 hours, Meier's team had 63 responses from around the U.S.

By using this program, the team demonstrated a totally new approach to gathering information, one that other research groups might replicate.

In a separate survey, a crew weatherizing homes in Minnesota recorded information about the thermostat in each home they visited and supplied data. Out of 20 homes visited, the crew found that about half had the thermostat set to indefinite hold, and one home actually had the thermostat switched off.

Although these samples are small, the results are striking. "If these numbers are anywhere near the same on a national scale, it's clear that millions of homes are not getting the benefits of energy savings that are possible from a fully programmed, functioning thermostat," says Meier.

Designing a Usability Test

Now the team turned to the problem of figuring out how to measure usability in a thermostat. "We asked, is it possible to quantify the confusion, annoyance and lack of clarity in some designs of thermostats?" says Meier.

They set up a test facility with five models of thermostats—four of them wall boxes and one, a new type of thermostat with a control interface displayed on a computer screen. The wall boxes were mounted at heights that are typical of actual home installation, forcing users to stand while performing the test actions.

They made videos of test subjects trying to adjust and program a thermostat. These revealed that the time required to accomplish a task, such as setting the clock time or programming a setpoint for a certain hour, varied wildly. Some people accomplished the task in twenty seconds, others took minutes, and some never figured out how. And the time it took to perform a particular task varied from one make and model of thermostat to another.

Meier's research team eventually devised a usability metric based on measuring the time it takes to accomplish a set of five basic tasks on a thermostat model, including noting if the task was never completed.

The 32 participants in the study were asked to perform six tasks on each thermostat, including setting the correct time and setting the thermostat from 'off' to 'heat'. [see box]. The test subjects were given no training in how to use the thermostats, as would be the case in real life. Videos were made of their attempts to accomplish each task—372 in all.

The research team compared the time it took to complete each of the six tasks, and the average time to complete each task by model. There were distinct differences in how fast the same task took test subjects among various models of thermostats tested. In fact, the research team was able to rank-order the five models in terms of their usability. Using the raw data, they also developed a quantitative metric of usability.

"Our study has demonstrated that it is possible to develop a metric and measure the usability of thermostats," says Meier, "and we think that the same approach can be used for the usability of other controls such as TV sets, heat pump water heaters, and lighting controls."

"We've also learned that many homes fail to use the features of programmable thermostats, and so are probably not getting the benefits of automatic energy management, and energy savings that they are capable of providing."

Meier added that "we weren't aiming to change anyone's behavior, but we can facilitate energy-saving behavior by minimizing barriers."

The team is now working to expand their user surveys of thermostat operation to more homes, and developing a usability metric for other appliance controls where settings can strongly influence energy consumption. One intention is to develop a computed "score" based on the usability metrics they developed.

Researchers who participated in the project are: with Cecilia Aragon of its Computational Research Division, Therese Peffer and Daniel Perry of UC Berkeley, and Marco Pritoni of UC Davis.

This research is funded by the U.S. Department of Energy, the U.S. Environmental Protection Agency, and the California Energy Commission.