November 25, 2024

Smart electrical meters and their smart peripherals

When I was a college undergraduate, I lived in a 1920’s duplex and I recall my roommate and I trying to figure out where our electrical bill was going. He was standing outside by the electrical meter, I was turning things on and off, and we were yelling back and forth so we could sort out which gadgets were causing the wheel to spin faster. (The big power sinks? Our ancient 1950’s refrigerator and my massive-for-the-day 20-inch computer monitor.) Needless to say, this was more difficult than it should have been.

More recently, I got myself a Kill-a-Watt inline power meter which you can use at any power outlet, but it’s a pain. You have to unplug something to measure its usage. People with the big bucks will spring for a Ted 5000 system, which an electrician installs in your breaker box. That’s fantastic, but it’s not cheap or easy.

Today, I’m now the proud new owner of an LS Research “RateSaver”, which speaks ZigBee wireless to the “smart meter” that CenterPoint Energy installed on all the houses in our area. How did I get this thing? I went to a League of Women Voters “meet the candidates” event back in October and CenterPoint Energy had a display there. I asked the guy if I could get one of these things and he said he’s look into it for me. Fast forward two months later, and a box arrived in the mail. New toy!

So what exactly is it? It’s a battery-powered light-weight box with a tolerably readable two-inch monochrome LCD display. As I’m sitting here typing, it’s updating my “current usage” every few seconds and is giving me a number that’s ostensibly accurate to the watt. In the last minute, after I pressed the proper button, it’s been alternating between reading 650-750 watts, and 1400-1500 watts. (Hmm… maybe my fridge consumes 700 watts.) If you leave it alone, the refresh rate slows down to maybe once a minute. Also, it’s sometimes reading “0.000 kW” which is clearly incorrect but it returns to the proper number when I press the button. Wireless range is quite good. I’m on the opposite side of the house as our electrical meter and it’s working fine.

The user interface is all kinds of terrible. In addition to slow button response, the button labels are incorrect. LS Research is apparently just rebranding a Honeywell Home Energy Display (for which the Honeywell manual was included). LS Research apparently rearranged the button labels without changing the corrresponding software. Bravo! Thankfully, the Honeywell manuals have the proper labeling. Also amusing: there’s a message in the system saying that “non-peak price starts at 7:00 PM. Save Money by waiting” when in fact my electrical pricing deal is for a flat rate (which floats with market conditions and is presently $0.0631 per kWh).

Update: I’ve since learned that Honeywell acquired LS Research, so this is something of a transitional screw-up. Welcome to the world of beta products.

Since I’m a security guy, I assumed I’d have to go through some kind of protocol to get the thing activated, and the manual from inside the box describes an activation procedure where you make a phone call to your energy company, giving them the hardware ID numbers of the outdoor smart meter and the indoor display box. Conflicting instructions were also included with my display, describing setup which was as simple as “turn it on and hit the connect button” so I went with the easy instructions. Time passed and the box started working without requiring any additional input from me. I hope that my display box was pre-configured to work exclusively with my house, but this does lead me to wonder about whether they got the security right. (I experimentally turned lights on and off while watching the meter updates and validated that I am, in fact, looking at the usage of my own house.)

At the end of the day, I and everybody else here is now required to pay a $3.24 “advanced meter charge” in order to have all this functionality (which, incidentally, saves the electric company money since it no longer needs human meter readers). Is it worth it? Presumably, at some point I’ll have some kind of variable-priced electricity and I could then hack my refrigerator and air conditioning system to pay attention to the spot price of electricity. If electricity got extra cheap during a five minute window of the hot summer, the controller could then crank the A/C and drop the house an extra few degrees. Of course, if everybody was following this same algorithm, you’d either have insane demand swings, when everybody jumps on to consume cheaper electricity when it’s available, or you’d have to carefully engineer the pricing system such that you had stable demand. Presumably, if you got somebody who understood control theory to design this properly, you could end up incentivizing both demand and pricing to be fairly stable across the space of any given hour of the day.

Probably the biggest benefit of these smart meters will come the next time we have a major hurricane that comes through and knocks out power. Hurricane Ike left my house without power for ten days. At the time, CenterPoint Energy had a vague and useless web site that would give you an idea what neighborhoods were being repaired. Since it was too hot to stay in our house, we stayed instead with a friend who had power and drove by our place every day to see if it had power. This was very frustrating. (I unplugged all my computer equipment, since I didn’t want flakey power to nuke my equipment. Consequently, I couldn’t just do something simple like ping my home computer.) Today, I can log into CenterPoint Energy’s web site and see the power consumption of my house, in 15-minute intervals, and so can the people coordinating the repairs. If they integrated that with a mapping system, they’d have real-time blackout maps, which have obvious value to emergency planners and repair operations coordination.

I just hope they have somebody with a clue looking over the security of their system. (If somebody from CenterPoint reads this: people like me are more than happy to do private security evaluations, red-team exercises, and so forth.)

Future work: there’s a mini USB port on the side of the box. Now I just have to find some documentation. It’s probably bad form for me to go reverse-engineer it myself.

Unpeeling the mystique of tamper-indicating seals

As computer scientists have studied the trustworthiness of different voting technologies over the past decade, we notice that “security seals” are often used by election officials. It’s natural to wonder whether they really provide any real security, or whether they are just for show. When Professor Avi Rubin volunteered as an election judge (Marylandese for pollworker) in 2006, one of his observations that I found most striking was this:


Avi Rubin


“For example, I carefully studied the tamper tape that is used to guard the memory cards. In light of Hursti’s report, the security of the memory cards is critical. Well, I am 100% convinced that if the tamper tape had been peeled off and put back on, nobody except a very well trained professional would notice it. The tamper tape has a tiny version of the word “void” appear inside it after it has been removed and replaced, but it is very subtle. In fact, a couple of times, due to issues we had with the machines, the chief judge removed the tamper tape and then put it back. One time, it was to reboot a machine that was hanging when a voter was trying to vote. I looked at the tamper tape that was replaced and couldn’t tell the difference, and then it occurred to me that instead of rebooting, someone could mess with the memory card and replace the tape, and we wouldn’t have noticed. I asked if I could play with the tamper tape a bit, and they let me handle it. I believe I can now, with great effort and concentration, tell the difference between one that has been peeled off and one that has not. But, I did not see the judges using that kind of care every time they opened and closed them. As far as I’m concerned, the tamper tape does very little in the way of actual security, and that will be the case as long as it is used by lay poll workers, as opposed to CIA
agents.”

Avi is a first-rate expert in the field of computer security, in part because he’s a good experimentalist—as in, “I asked if I could play with the tamper tape.” To the nonexpert,
security seals have a mystique: there’s this device there, perhaps a special tape or perhaps a thing that looks like a little blue plastic padlock. Most of us encounter these devices in a context where we can’t “play with” them, because that would be breaking the rules: on voting machines, on our electric meter, or whatever. Since we don’t play with them, we can’t tell whether they are secure, and the mystique endures. As soon
as Avi played with one, he discovered that it’s not all that secure.

In fact, we have a word for a piece of tape that only gives the appearance of working:

band-aid: (2) a temporary way of dealing with a problem that will not really solve it (Macmillan Dictionary)

In the last couple of years I’ve been studying security seals on voting machines in New Jersey. For many decades New Jersey law has required that each voting machine be “sealed with a numbered seal”, just after it is prepared for each election (NJSA 19:48-6). Unfortunately it’s hard for legislators to write into the statutes exactly how well these seals must work. Are tamper-indicating seals used in elections really secure? I’ll address that question in my next few articles.

Join CITP in DC this Friday for "Emerging Threats to Online Trust"

Update – you can watch the video here.

Please join CITP this Friday from 9AM to 11AM for an event entitled “Emerging Threats to Online Trust: The Role of Public Policy and Browser Certificates.” The event will focus on the trustworthiness of the technical and policy structures that govern certificate-based browser security. It will include representatives from government, browser vendors, certificate authorities, academics, and hackers. For more information see:

http://citp.princeton.edu/events/emerging-threats-to-online-trust/

Several Freedom-to-Tinker posts have explored this set of issues:

Hacking the D.C. Internet Voting Pilot

The District of Columbia is conducting a pilot project to allow overseas and military voters to download and return absentee ballots over the Internet. Before opening the system to real voters, D.C. has been holding a test period in which they've invited the public to evaluate the system's security and usability.

This is exactly the kind of open, public testing that many of us in the e-voting security community — including me — have been encouraging vendors and municipalities to conduct. So I was glad to participate, even though the test was launched with only three days' notice. I assembled a team from the University of Michigan, including my PhD students, Eric Wustrow and Scott Wolchok, and Dawn Isabel, a member of the University of Michigan technical staff.

Within 36 hours of the system going live, our team had found and exploited a vulnerability that gave us almost total control of the server software, including the ability to change votes and reveal voters’ secret ballots. In this post, I’ll describe what we did, how we did it, and what it means for Internet voting.

D.C.'s pilot system

The D.C. system is built around an open source server-side application developed in partnership with the TrustTheVote project. Under the hood, it looks like a typical web application. It's written using the popular Ruby on Rails framework and runs on top of the Apache web server and MySQL database.

Absentee overseas voters receive a physical letter in the mail instructing them to visit a D.C. web site, http://www.dcboee.us/DVM/, and log in with a unique 16-character PIN. The system gives voters two options: they can download a PDF ballot and return it by mail, or they can download a PDF ballot, fill it out electronically, and then upload the completed ballot as a PDF file to the server. The server encrypts uploaded ballots and saves them in encrypted form, and, after the election, officials transfer them to a non-networked PC, where they decrypt and print them. The printed ballots are counted using the same procedures used for mail-in paper ballots.

A small vulnerability, big consequences

We found a vulnerability in the way the system processes uploaded ballots. We confirmed the problem using our own test installation of the web application, and found that we could gain the same access privileges as the server application program itself, including read and write access to the encrypted ballots and database.

The problem, which geeks classify as a “shell-injection vulnerability,” has to do with the ballot upload procedure. When a voter follows the instructions and uploads a completed ballot as a PDF file, the server saves it as a temporary file and encrypts it using a command-line tool called GnuPG. Internally, the server executes the command gpg with the name of this temporary file as a parameter: gpg […] /tmp/stream,28957,0.pdf.

We realized that although the server replaces the filename with an automatically generated name (“stream,28957,0” in this example), it keeps whatever file extension the voter provided. Instead of a file ending in “.pdf,” we could upload a file with a name that ended in almost any string we wanted, and this string would become part of the command the server executed. By formatting the string in a particular way, we could cause the server to execute commands on our behalf. For example, the filename “ballot.$(sleep 10)pdf” would cause the server to pause for ten seconds (executing the “sleep 10” command) before responding. In effect, this vulnerability allowed us to remotely log in to the server as a privileged user.

Our demonstration attacks

D.C. launched the public testbed server on Tuesday, September 28. On Wednesday afternoon, we began to exploit the problem we found to demonstrate a number of attacks:

  • We collected crucial secret data stored on the server, including the database username and password as well as the public key used to encrypt the ballots.
  • We modified all the ballots that had already been cast to contain write-in votes for candidates we selected. (Although the system encrypts voted ballots, we simply discarded the encrypted files and replaced them with different ones that we encrypted using the same key.) We also rigged the system to replace future votes in the same way.
  • We installed a back door that let us view any ballots that voters cast after our attack. This modification recorded the votes, in unencrypted form, together with the names of the voters who cast them, violating ballot secrecy.
  • To show that we had control of the server, we left a “calling card” on the system's confirmation screen, which voters see after voting. After 15 seconds, the page plays the University of Michigan fight song. Here's a demonstration.

Stealthiness wasn't our main objective, and our demonstration had a much greater footprint inside the system than a real attack would need. Nevertheless, we did not immediately announce what we had done, because we wanted to give the administrators an opportunity to exercise their intrusion detection and recovery processes — an essential part of any online voting system. Our attack remained active for two business days, until Friday afternoon, when D.C. officials took down the testbed server after several testers pointed out the fight song.

Based on this experience and other results from the public tests, the D.C. Board of Elections and Ethics has announced that they will not proceed with a live deployment of electronic ballot return at this time, though they plan to continue to develop the system. Voters will still be able to download and print ballots to return by mail, which seems a lot less risky.

D.C. officials brought the testbed server back up today (Tuesday) with the electronic ballot return mechanism disabled. The public test period will continue until Friday, October 8.

What this means for Internet voting

The specific vulnerability that we exploited is simple to fix, but it will be vastly more difficult to make the system secure. We've found a number of other problems in the system, and everything we've seen suggests that the design is brittle: one small mistake can completely compromise its security. I described above how a small error in file-extension handling left the system open to exploitation. If this particular problem had not existed, I'm confident that we would have found another way to attack the system.

None of this will come as a surprise to Internet security experts, who are familiar with the many kinds of attacks that major web sites suffer from on a daily basis. It may someday be possible to build a secure method for submitting ballots over the Internet, but in the meantime, such systems should be presumed to be vulnerable based on the limitations of today's security technology.

We plan to write more about the problems we found and their implications for Internet voting in a forthcoming paper.


Professor J. Alex Halderman is a computer scientist at the University of Michigan.

Why did anybody believe Haystack?

Haystack, a hyped technology that claimed to help political dissidents hide their Internet traffic from their governments, has been pulled by its promoters after independent researchers got a chance to study it and found severe problems.

This should come as a surprise to nobody. Haystack exhibited the warning signs of security snake oil: the flamboyant, self-promoting front man; the extravagant security claims; the super-sophisticated secret formula that cannot be disclosed; the avoidance of independent evaluation. What’s most interesting to me is that many in the media, and some in Washington, believed the Haystack hype, despite the apparent lack of evidence that Haystack would actually protect dissidents.

Now come the recriminations.

Jillian York summarizes the depressing line of adulatory press stories about Haystack and its front man, Austin Heap.

Evgeny Morozov at Foreign Affairs, who has been skeptical of Haystack from the beginning, calls several Internet commentators (Zittrain, Palfrey, and Zuckerman) “irresponsible” for failing to criticize Haystack earlier. Certainly, Z, P, and Z could have raised questions about the rush to hype Haystack. But the tech policy world is brimming with overhyped claims, and it’s too much to expect pundits to denounce them all. Furthermore, although Z, P, and Z know a lot about the Internet, they don’t have the expertise to evaluate the technical question of whether Haystack users can be tracked — even assuming the evidence had been available.

Nancy Scola, at TechPresident, offers a more depressing take, implying that it’s virtually impossible for reporters to cover technology responsibly.

It takes real work for reporters and editors to vet tech stories; it’s not enough to fact check quotes, figures, and events. Even “seeing a copy [of the product],” as York puts it, isn’t enough. Projects like Haystack need to be checked-out by technologists in the know, and I’d argue the before the recent rise of techno-advocates like, say, Clay Johnson or Tom Lee, there weren’t obvious knowledgeable sources for even dedicated reporters to call to help them make sense of something like Haystack, on deadline and in English.

Note the weasel-word “obvious” in the last sentence — it’s not that qualified experts don’t exist, it’s just that, in Scola’s take, reporters can’t be bothered to find out who they are.

I don’t think things are as bad as Scola implies. We need to remember that the majority of tech reporters didn’t hype Haystack. Non-expert reporters should have known to be wary about Haystack, just based on healthy journalistic skepticism about bold claims made without evidence. I’ll bet that many of the more savvy reporters shied away from Haystack stories for just this reason. The problem is that the few who did not got undeserved attention.

[Update (Tue 14 Sept 2010): Nancy Scola responds, saying that her point was that reporters’ incentives are to avoid checking up too much on enticing-if-true stories such as Haystack. Fair enough. I didn’t mean to imply that she condoned this state of affairs, just that she was pointing out its existence.]