February 25, 2018

Counterfeits, Trojan Horses, and shady distributors

Last Friday, the New York Times published an article about counterfeit Cisco products that have been sold as if they were genuine and are widely used throughout the U.S. government.  The article also raised the concern that these counterfeits could well be engineered with malicious intent, but that this appears not to have been the case. There was an immediate Slashdot thread as well, but a number of issues are still worth commenting on.

First things first: the facts, as best we understand them.  The New York Times reports that approximately 3500 counterfeit Cisco components (worth $3.5M) have been discovered as a result of a two-year FBI investigation.  A Cisco spokesman is quoted saying that they found “no evidence of re-engineering.”  In other words, we’re talking about faithful knock-offs of legitimate products.

If you go to the FBI’s unclassified PowerPoint presentation (dated January 11, 2008), you’ll see all the actual information.  This is a fascinating read.  For starters, let’s talk about the cost.  The slides claim you can get a counterfeit router for approximately 1/6 the cost of a genuine router.  (You can do similarly well buying used gear on eBay.)  The counterfeit gear looks an awful lot like the genuine article.  Detecting differences here is as difficult as detecting counterfeit money, counterfeit Rolex watches, or counterfeit signatures from sports stars.  Given the apparent discrepancy between component cost and street value, we should be no more surprised to find knock-off Cisco gear than we are to find knock-off everything else.

Counterfeit vs. Original Cisco line card

It’s claimed that these counterfeits are built to lower manufacturing standards than the original equipment, causing higher failure rates. One even caught fire due to a faulty power supply.  Likewise, the fakers are making stupid errors, like building multiple components with the same MAC address.  (MAC addresses, by design, are meant to be unique – no two ever the same.)

The really interesting story is all about the supply chain. Consider how you might buy yourself a new Mac.  You could go to your local Apple store.  Or you could get it from any of a variety of other stores, who in turn may have gotten it from Apple directly or may have gone through a distributor.  Apparently, for Cisco gear, it’s much more complicated than that.  The U.S. government buys from “approved” vendors, who might then buy from multiple tiers of sub-contractors.  In one case, one person bought shady gear from eBay and resold it to the government, moving a total of $1M in gear before he was caught.  In a more complicated case, Lockheed Martin won a bid for a U.S. Navy project.  They contracted with an unauthorized Cisco reseller who in turn contracted with somebody else, who used a sub-contractor, who then directly shipped the counterfeit gear to the Navy. (The slides say that $250K worth of counterfeit gear was sold; duplicate serial numbers were discovered.)

Why is this happening?  The Government wants to save money, so they look for contractors who can give them the best price, and their contracts allow for subcontracts, direct third-party shipping, and so forth.  There is no serious vetting of this supply chain by either Cisco or the government. Apparently, Cisco doesn’t do direct sales except for high-end, specialized gear.  You’d think Cisco would follow the lead of the airline industry, among others, and cut out the distributors to keep the profit for themselves.

Okay, on to the speculation.  Both the New York Times and the FBI presentation concern themselves with Trojan Horses.  Even though there’s no evidence that any of this counterfeit gear was actually malicious, the weak controls in the supply chain make it awfully easy for such compromised gear to be sold into sensitive parts of the government, raising all the obvious concerns.

Consider a recent paper by U. Illinois’s Sam King et al. where they built a “malicious processor”.  The idea is pretty clever.  You send along a “secret knock” (e.g., a network packet with a particular header) which triggers a sensor that enables “shadow code” to start running alongside the real operating system.  The Illinois team built shadow code that compromised the Linux login program, adding a backdoor password.  After the backdoor was tripped, it would disable the shadow code, thus going back to “normal” operation.

The military is awfully worried about this sort of threat, as well they should be.  For that matter, so are voting machine critics. It’s awfully easy for “stealth” malicious behavior to exist in legitimate systems, regardless of how carefully you might analyze or test it. Ken Thompson’s classic paper, Reflections on Trusting Trust, shows how he designed a clever Trojan Horse for Unix.  [Edit: it’s unclear that it ever got released into the wild.]

Okay everybody, let’s put on our evil hats.  If your goal was to get a Trojan Horse router into a sensitive military environment, how would you do it and how would it behave?  Clearly, the weak supply chain is an excellent vector for getting the gear into place.  Given the resources of a nation-state intelligence agency, you could afford to buy genuine Cisco parts and modify them, rather than using low-cost, counterfeit gear.  Nobody would detect you; you wouldn’t screw up and ship multiple boxes with the same serial number.

How will you implement your Trojan Horse logic?  Pretty much any gear you’ll ever find of any modest complexity will have software running inside it.  Even line cards have embedded processors of some sort.  For all that hardware, there’s software, and that’s what you’d go to install your logic bomb.  The increasing use of FPGAs in industrial designs means you could also “rewire” those parts to behave arbitrarily, much like the Illinois hack; you’d really want to get a hold of the original VHDL “source code”, leveraging your aforementioned spying prowess, to simplify the design and implementation of your malicious behavior.  Hacking the raw netlists (the FPGA-equivalent of machine code) would be possible, but would be far more painful. [See Sidebar.]

What sort of behavior would you build in?  The New York Times raises the idea of a kill switch.  I send your router a magic packet and it dies.  That’s too easy.  How about I send your router a magic packet, it then forwards it on to all of its peers, repeatedly, and then they all die a few seconds later?  That’s a pretty good denial of service attack (nevermind a plot device that was the basis of a popular science fiction television series). Alternatively, following the Illinois idea, we could imagine that the magic packet turns on a monitoring feature, allowing our intelligence agency to gather all kinds of information, reconfigure the router, and so forth.  If they don’t want to generate extra traffic, which might be detected, they could instead weaken the encryption of a VPN tunnel, perhaps publishing the session key through a subliminal channel of some sort, acquiring the ciphertext through “other” means.

In summary, it’s probably a good thing, from the perspective of the U.S. military, to discover that their supply chain is allowing counterfeit gear into production.  This will help them clean up the supply chain, and will also provide an extra push to consider just how much they trust the sources of their equipment to ship clean software and hardware.

[Sidebar: Xilinx supports a notion of “encrypting” a netlist.  Broadly speaking, the idea behind the technology is to encrypt the description of your FPGA configuration with a crypto key, such that anybody who reads the file out of your board gets encrypted garbage.  However, the FPGA has the key material to decrypt the configuration and then initialize itself normally.  This sort of technology is meant to serve an anti-piracy / anti-reverse-engineering purpose.  It could ostensibly also serve an anti-Trojan Horse purpose, although at that point it’s really no more or less secure, semantically, than Microsoft’s Authenticode.  This technology, more broadly, is also an active research area (see, for example, Roy et al.’s EPIC: Ending Piracy of Integrated Circuits).  Again, if we’ve got a nation-state intelligence service tampering with the system, none of this is going to provide meaningful protection for the end-user against Trojan Horses.]

One Laptop Per Child (New Version), Reviewed by 12-Year-Old

[Today we welcome back SG, a twelve-year-old who previously reviewed the B2 version of the One Laptop Per Child computer. SG had a chance to examine the latest (B4) version of the OLPC machine and write a new review. As before, the review is unedited, just as SG wrote it. – Ed]

After my first review, the administrators at OLPC were kind enough to send Mr. Felten the newer model of the computer, the B4, for me to review. The difference between the two models was quite dramatic. Between new games, new applications, design changes, and a few touch ups for the system, the B4 clearly outshines the B2. I didn’t even know about a bunch of problems in the B2 until they got fixed in the B4!

The minute I picked the new computer up, I saw the physical differences. There are bumps on the handle of the B4. The B2 has none. The flip- up antenna on the B2 was encased in hard plastic, and on the B4, it’s just thick rubber. The keyboards are pretty much the same, apart from a few minor differences along the top. Once I opened it up and started it, I noted how much quicker it booted up than the B2. Then I saw the icons. The B2 has less than half the icons than the B4, which has 13!

As for games, entertainment, and the internet, this computer has bountiful resources. There were many new and fun programs. One of them, called “Block party”, is just plain old tetris with a different name. As I am not really gifted in tetris, I had a lot of fun losing repeatedly. The internet was a lot better on this newer laptop. In my last review, I complained about how slow it was and how the connection was so-so. In the B4, both of those problems have been fixed. It is quick, always connects, and is really very nice. If you don’t want to go on the web to read the new Freedom to Tinker article, “News Reader” lets you subscribe to websites’ feeds. In the games category, “Connect” is a game which can only be played on two separate OLPC laptops . The game is a little like tick tack toe. If you’ve ever played “Connect 4”, that’s the same game. If you want to watch some video clip from the web, “watch and listen”, OLPC’s media player, has you covered. Want some music? Use “tamtam”. This application is similar to Garageband, but not quite the same. Last but not least is “Record”. On the B2, “record” just took pictures with an okay camera. On the B4, you can take pictures with a pretty good camera AND record video with no time limit (as far as I can tell). I was surprised and overjoyed to discover I could take video with the new one.

One of the coolest applications is called simply “Chat”. It is basically an IM-ish kind of thing that works between all OLPC laptops. Since I got two laptops from OLPC, I could test out the chat application with my friends and family. I spent a lot of time having silent conversations with the friend sitting across the room, so that was fun. Etoys is another cool application, and it is definitely the program of a genius technologist. Although it is difficult to understand and use, once you get into the swing of things, it’s awesome. To use Etoys you make a “sketch” on the computer, then save it, and that’s where the fun begins. You can write “scripts” that make the sketch move around the screen in the way that you want. You can put it in “books” that have multiple pages for a flip book or make animations with it (ie. a bouncing ball, flying bird, eating kid, etc.). In Turtle Art, you get a chance to write a simple program that makes the turtle in the middle of the screen move. It’s very cool.

Last review, I said that my main problem with the computer was its slow speed and its battery charge. And I am happy to say that both of those problems have been fixed in the new version. It has more applications, higher quality camera, more games, a few design changes for the better, and much more. I tested how long it would stay alive by opening it and leaving it open. Surprisingly, it stayed awake for more than four hours! And some other testing revealed that the B4 does, in fact, auto save your documents and stuff if it runs out of battery while an unsaved document is on it. I like that feature, because there were many times with the B2 that I was typing and it just died, leaving me rather stunned for a couple seconds until I came to my senses and wearily plugged it in. Then it would take hours to charge up again. But in the B4, it charges up really quickly. Another minor turn for the better is the plug. Now they are greener, more round, easier to hold, and they have the XO sign on them.

I thought that this version was way better than the last one. It was just easier to figure out, more fun to spend time on, just better. It’s going to be hard to send it back to OLPC, but I’m going to have to. It’s great that they’re going to start selling them to the public. (You have to buy two, and you send one to a needy kid in a third world country and keep one for yourself. Read about it in the New York Times… …) I hope I can get one!

For a regular laptop, this would be the paragraph about its problems, its deficiencies. But the thing is, there aren’t any problems with this computer! Congratulations, OLPC. You’ve done it. Or will you come out with yet better laptops? Is that even possible? We’ll have to see…

OLPC Review Followup

Last week’s review of the One Laptop Per Child (OLPC) machine by twelve-year-old “SG” was one of our most-commented-upon posts ever. Today I want to follow up on a few items.

First, the machine I got for SG was the B2 (Beta 2) version of the OLPC system, which is not the latest. Folks from the OLPC project suggest that some of the problems SG found are fixed in the latest version. They have graciously offered to send an up to date OLPC machine for SG to review. SG has agreed to try out the new machine and review it here on Freedom to Tinker.

Second, I was intrigued by the back-and-forth in the comments over SG’s gender. I had originally planned to give SG a pseudonym that revealed SG’s gender, but a colleague suggested that I switch to a gender-neutral pseudonym. Most commenters didn’t seem to assume one gender or the other. A few assumed that SG is a boy, which generated some pushback from others who found that assumption sexist. My favorite comment in this series was from “Chris,” who wrote:

Why are you assuming the review was written by a boy?
At 12 we’re only two years from 8th grade level, the rumored grail (or natural default) of our national publications. SG, you’re clearly capable of writing for most any publication in this country, you go girl! (even if you are a boy)

Third, readers seem to be as impressed as I was by the quality of SG’s writing. Some found it hard to believe that a twelve-year-old could have written the post. But it was indeed SG’s work. I am assured that SG’s parents did not edit the post but only suggested in general terms the addition of a paragraph about what SG did with the machine. I suggested only one minor edit to preserve SG’s anonymity. Otherwise what you read is what SG wrote.

Though sentences like “My expectations for this computer were, I must admit, not very high.” seem unusual for a twelve-year-old, others show a kid’s point of view. One example: “Every time you hit a key, it provides a certain amount of satisfaction of how squishy and effortless it is. I just can’t get over that keyboard.”

SG is welcome to guest blog here in the future. Kids can do a lot, if we let them.

[Update (June 2012): I can reveal now that SG is a girl: my daughter Claire Felten.]