June 13, 2024

Counterfeits, Trojan Horses, and shady distributors

Last Friday, the New York Times published an article about counterfeit Cisco products that have been sold as if they were genuine and are widely used throughout the U.S. government.  The article also raised the concern that these counterfeits could well be engineered with malicious intent, but that this appears not to have been the case. There was an immediate Slashdot thread as well, but a number of issues are still worth commenting on.

First things first: the facts, as best we understand them.  The New York Times reports that approximately 3500 counterfeit Cisco components (worth $3.5M) have been discovered as a result of a two-year FBI investigation.  A Cisco spokesman is quoted saying that they found “no evidence of re-engineering.”  In other words, we’re talking about faithful knock-offs of legitimate products.

If you go to the FBI’s unclassified PowerPoint presentation (dated January 11, 2008), you’ll see all the actual information.  This is a fascinating read.  For starters, let’s talk about the cost.  The slides claim you can get a counterfeit router for approximately 1/6 the cost of a genuine router.  (You can do similarly well buying used gear on eBay.)  The counterfeit gear looks an awful lot like the genuine article.  Detecting differences here is as difficult as detecting counterfeit money, counterfeit Rolex watches, or counterfeit signatures from sports stars.  Given the apparent discrepancy between component cost and street value, we should be no more surprised to find knock-off Cisco gear than we are to find knock-off everything else.

Counterfeit vs. Original Cisco line card

It’s claimed that these counterfeits are built to lower manufacturing standards than the original equipment, causing higher failure rates. One even caught fire due to a faulty power supply.  Likewise, the fakers are making stupid errors, like building multiple components with the same MAC address.  (MAC addresses, by design, are meant to be unique – no two ever the same.)

The really interesting story is all about the supply chain. Consider how you might buy yourself a new Mac.  You could go to your local Apple store.  Or you could get it from any of a variety of other stores, who in turn may have gotten it from Apple directly or may have gone through a distributor.  Apparently, for Cisco gear, it’s much more complicated than that.  The U.S. government buys from “approved” vendors, who might then buy from multiple tiers of sub-contractors.  In one case, one person bought shady gear from eBay and resold it to the government, moving a total of $1M in gear before he was caught.  In a more complicated case, Lockheed Martin won a bid for a U.S. Navy project.  They contracted with an unauthorized Cisco reseller who in turn contracted with somebody else, who used a sub-contractor, who then directly shipped the counterfeit gear to the Navy. (The slides say that $250K worth of counterfeit gear was sold; duplicate serial numbers were discovered.)

Why is this happening?  The Government wants to save money, so they look for contractors who can give them the best price, and their contracts allow for subcontracts, direct third-party shipping, and so forth.  There is no serious vetting of this supply chain by either Cisco or the government. Apparently, Cisco doesn’t do direct sales except for high-end, specialized gear.  You’d think Cisco would follow the lead of the airline industry, among others, and cut out the distributors to keep the profit for themselves.

Okay, on to the speculation.  Both the New York Times and the FBI presentation concern themselves with Trojan Horses.  Even though there’s no evidence that any of this counterfeit gear was actually malicious, the weak controls in the supply chain make it awfully easy for such compromised gear to be sold into sensitive parts of the government, raising all the obvious concerns.

Consider a recent paper by U. Illinois’s Sam King et al. where they built a “malicious processor”.  The idea is pretty clever.  You send along a “secret knock” (e.g., a network packet with a particular header) which triggers a sensor that enables “shadow code” to start running alongside the real operating system.  The Illinois team built shadow code that compromised the Linux login program, adding a backdoor password.  After the backdoor was tripped, it would disable the shadow code, thus going back to “normal” operation.

The military is awfully worried about this sort of threat, as well they should be.  For that matter, so are voting machine critics. It’s awfully easy for “stealth” malicious behavior to exist in legitimate systems, regardless of how carefully you might analyze or test it. Ken Thompson’s classic paper, Reflections on Trusting Trust, shows how he designed a clever Trojan Horse for Unix.  [Edit: it’s unclear that it ever got released into the wild.]

Okay everybody, let’s put on our evil hats.  If your goal was to get a Trojan Horse router into a sensitive military environment, how would you do it and how would it behave?  Clearly, the weak supply chain is an excellent vector for getting the gear into place.  Given the resources of a nation-state intelligence agency, you could afford to buy genuine Cisco parts and modify them, rather than using low-cost, counterfeit gear.  Nobody would detect you; you wouldn’t screw up and ship multiple boxes with the same serial number.

How will you implement your Trojan Horse logic?  Pretty much any gear you’ll ever find of any modest complexity will have software running inside it.  Even line cards have embedded processors of some sort.  For all that hardware, there’s software, and that’s what you’d go to install your logic bomb.  The increasing use of FPGAs in industrial designs means you could also “rewire” those parts to behave arbitrarily, much like the Illinois hack; you’d really want to get a hold of the original VHDL “source code”, leveraging your aforementioned spying prowess, to simplify the design and implementation of your malicious behavior.  Hacking the raw netlists (the FPGA-equivalent of machine code) would be possible, but would be far more painful. [See Sidebar.]

What sort of behavior would you build in?  The New York Times raises the idea of a kill switch.  I send your router a magic packet and it dies.  That’s too easy.  How about I send your router a magic packet, it then forwards it on to all of its peers, repeatedly, and then they all die a few seconds later?  That’s a pretty good denial of service attack (nevermind a plot device that was the basis of a popular science fiction television series). Alternatively, following the Illinois idea, we could imagine that the magic packet turns on a monitoring feature, allowing our intelligence agency to gather all kinds of information, reconfigure the router, and so forth.  If they don’t want to generate extra traffic, which might be detected, they could instead weaken the encryption of a VPN tunnel, perhaps publishing the session key through a subliminal channel of some sort, acquiring the ciphertext through “other” means.

In summary, it’s probably a good thing, from the perspective of the U.S. military, to discover that their supply chain is allowing counterfeit gear into production.  This will help them clean up the supply chain, and will also provide an extra push to consider just how much they trust the sources of their equipment to ship clean software and hardware.

[Sidebar: Xilinx supports a notion of “encrypting” a netlist.  Broadly speaking, the idea behind the technology is to encrypt the description of your FPGA configuration with a crypto key, such that anybody who reads the file out of your board gets encrypted garbage.  However, the FPGA has the key material to decrypt the configuration and then initialize itself normally.  This sort of technology is meant to serve an anti-piracy / anti-reverse-engineering purpose.  It could ostensibly also serve an anti-Trojan Horse purpose, although at that point it’s really no more or less secure, semantically, than Microsoft’s Authenticode.  This technology, more broadly, is also an active research area (see, for example, Roy et al.’s EPIC: Ending Piracy of Integrated Circuits).  Again, if we’ve got a nation-state intelligence service tampering with the system, none of this is going to provide meaningful protection for the end-user against Trojan Horses.]


  1. To be clear, the bitstream (not netlist) encryption prevents one from decrypting the bitstream. Security keys from xilinx are provided in plain text to the CM who manufactures the PCB. That’s how the key gets in there. For small volumes, sure, you could use the xilinx tools yourself to program the key in and then CM wouldn’t see it. But for large scale companies that’s not practical. This is where the gray market comes from, ex-employees from the CM or insiders at the CM have access to these files.
    Encryption helps from reverse engineering your bitstream by a third party, it does nothing to help gray market copies.

  2. Dan, I’m afraid you’ve moved the bomb line. This article now looks good, but some comments made since your hosting change have differently-incorrect characters where apostrophes or accented characters should be (see e.g. Bryan Feir’s May 26th response or the “username” on the following trackback comment to the “The Microsoft Case: A Window Into the Software Industry” article, or the quoted text in Anonymous Coward’s May 21st response to “The Microsoft Case, Ten Years Later” for examples).

    Data: I normally read f-t-t in FF2.0.0.12, but FF3b5, IE7, and IE6 all have the same problem with the badly-encoded chars in comments. FF3b5 shows the problem characters as 0xfffd, which, if I’m reading my Unicode documentation right, suggests that what you’re sending in the problem spots isn’t valid UTF-8. (And now you probably have a very good idea what kind of work I do for a living…)

  3. The “garbage characters” seem to be where quotation marks used to be. Odds are, this issue happened throughout the transition from the old site to the new one. Hopefully we can get this fixed soon.

  4. Dan, what happened to this article? Since you changed hosting providers, it’s full of garbage characters.

  5. supercat says

    Using battery-backed storage for bitstream encryption means that when the battery dies, so does the device. Some arcade games from the 1980’s are believed to be lost forever because of the use of such techniques.

  6. @Whoever: you’re right, I got the terms backward. More on bitstream encryption here:

  7. I think you are confusing netlists with bitstreams (or BIT files).

    Netlists are an intermediate format in the design process. The design starts with RTL, this is synthesized into a netlist and then the netlist is placed and routed. It is the result of the place and route that is actually used to program the FPGA

    The manufacturing site won’t see the netlists. The encryption of netlists is aimed at IP sales — where the IP supplier provides the design in an encrypted netlist to the design house that is doing the system design. The IP provider writes RTL, synthesizes it to a netlist and sells the netlist to the system house. The system designer combines the netlist with the rest of the design, synthesizes the rest and runs P&R on the complete design.

    Encrypting netlists prevents reverse engineering of the IP (which is difficult anyway with a netlist) by the system house and hence limits the use of the IP.

    There may also be encryption of bitstreams, but this is unrelated to encryption of netlists.

  8. How much money did the FBI spend to find the $3.5M of fake Cisco gear? Would Cisco have spent the similar amount of money on the investigation?

    As far as I see we mostly have trademark infringement here… functional equipment was provided to the US government.

  9. Ultimately, the question isn’t whether you should trust Cisco or any other particular company. The question is whether you should trust your supply chain to get you the genuine article that you thought you were buying, and whether your supply chain could be used as a vector to attack you.

    If official, legitimate Cisco gear is insecure or otherwise broken, that’s a separate problem.

  10. Dark Side says

    This discussion of course also brings up the question whether individuals and organizations outside the US should trust products from the US, for example from Cisco. I think not. Rule number one in security is not to trust a third party to address all your security concerns. Besides, Cisco tried to cover up vulnerabilities of its equipment in the past. Same applies to software, like Microsoft software.

  11. Just want to point out that Thompson did not actually plant the trojan in any real systems, he just wrote a paper on how he could have.