Update: More details have continued to come out, and I think that they generally support the less-paranoid version of events. There continues to be discussion on the mozilla.dev.security.policy list, Turktrust has given more details, and Mozilla has just opened up for public viewing their own detailed internal response documentation (including copies of all of the certs in question). None of this changes the fundamental riskiness of subordinate certificates, or the improvements that should be made to the CA system. It just means that in this case, the failure didn’t progress to a full-blown meltdown.
Today, the public learned of another failure by a Certificate Authority–one of of companies that certifies SSL-encryption for our internet communications. (See the end of this post for a catalogue of our past writing on problems with this “CA” system.) This time, the company Turktrust was revealed to have issued two subordinate certificates (also known as “intermediate” certificates) to entities that should not have had them. Subordinate certificates are very powerful. They give the holder the ability to issue SSL certificates for any domain name as though they have control of the parent CA’s “root” certificate. In this case, Google discovered that one of Turktrust’s previously undisclosed subordinate certificates had issued SSL certificates for the domain gmail.com, and that these certificates had been used to intercept Gmail users’ traffic… somewhere. This is where the details get foggy, but Turktrust has begun to describe their version of events.
There is a less paranoid and a more paranoid way of interpreting what happened.
According to Turktrust, they made some configuration errors in late 2011 that caused them to inadvertently hand out subordinate certificates to two entities–the Turkish government and a Turkish bank. They claimed that these users expected to receive ordinary SSL certificates that could be used to secure their own web sites, not what amounts to a master “skeleton key” to the internet. This assertion is somewhat supported by the fact that one of these certificates appears to have been used for precisely that until recently, at www.ego.gov.tr (which seems to be the web site for the capital city of Ankara or one of its offices, but I’m sure that Zeynep will clear this up in the comments). Turktrust says that the certificate was used by the government to provide SSL-encrypted webmail–presumably for government employees (As a further side-note, it is unclear whether deploying subordinate certificates as though they are ordinary SSL certificates will cause some or all browsers to generate an error, but perhaps it worked for them). In any case, Turktrust says that a government system administrator recently loaded the certificate onto a “firewall” that proceeded to perform man-in-the-middle attacks on individuals who attempted to browse to secure web sites like gmail.com. It’s not clear what network this device was on (although it was evidently manufactured by Check Point), and which individuals were affected. The least paranoid version of suggests that the device sat between the government’s internal network and the public internet, and that the only individuals affected were government employees in that office.
The more paranoid version of events is perhaps less likely, but it helps to highlight some fundamental shortcomings in the Certificate Authority system. Subordinate certificates have long been identified as a point of weakness in the CA system. They are typically granted unconstrained power to issue certificates for any domain name. Thus, a leak of one subordinate certificate is seen as equivalent to a leak of authority equivalent to all CAs combined. Worse, subordinate certificates need not be explicitly trusted by the software that authenticates encrypted SSL connections (typically your web browser). They inherit their trust from the explicitly trusted CAs that have been vetted by your browser vendor. Browser vendors have slowly been trying to reign in the practices of CAs that issue subordinate certificates to third parties. For instance, when CA Trustwave admitted that it had been selling subordinate certificates to third parties for the purpose of performing man-in-the-middle attacks, Mozilla made clear that this was unacceptable. That being said, depending on your browser, there may be a couple hundred trusted CAs. Hoping that they all comply with a policy is probably not enough. I have spent the past couple of years trying to convince Mozilla to strengthen their requirements so that CAs must disclose all subordinate certificates that they have issued, so that researchers can better map the risk landscape and so that users can make more informed trust decisions and detect unexpected subordinates. They’re getting closer, but it is taking an awfully long time. Several years ago, security researchers Christopher Soghoian and Sid Stamm described the “compelled certificate creation attack” in which a government would compel a local CA to produce a subordinate certificate that it would then use to conduct mass or targeted man-in-the-middle surveillance on its citizens.
We are justified in indulging in a bit of professional paranoia, given that the Turktrust subordinate certificate in question was controlled by the Turkish government, and that it was used to perform man-in-the-middle attacks. It also happens to be the case that Turktrust satisfied its annual compliance audits by having the Turkish government audit their operations, as opposed to the more common practice of obtaining an audit from a commercial auditing firm (although they claim to have recently obtained a commercial audit, they have yet to publish any documentation of this as far as I can tell). Turktrust has begun to give some answers, but it’s worthwhile to keep probing.
Mozilla revoked trust in one of Turktrust’s three “root” CA certificates, as did Google–but oddly the revoked root is not the certificate that issued the subordinate certificates in question. It is a new root certificate (included in Firefox only in beta releases so far), and it is intended to be used only for issuing higher-standard “extended validation” certificates. It appears that the browsers’ strategy is to provide some mild punishment to Turktrust without disrupting the operation of legitimate sites that have obtained standard SSL certificates from them in the past. Representatives from both browser vendors have stated that it is unclear how they will proceed, and one can imagine that they would restore trust in this root at some point in the future. Is Turktrust’s behavior sufficient that trust in all three of their roots should instead be removed permanently? The facts are still emerging.
Relevant past posts:
- Mozilla Debates Whether to Trust Chinese CA
- Web Security Trust Models
- Web Certification Fail: Bad Assumptions Lead to Bad Technology
- A Major Internet Milestone: DNSSEC and SSL
- General Counsel’s Role in Shoring Up Authentication Practices Used in Secure Communications
- The Flawed Legal Architecture of the Certificate Authority Trust Model
- Web Browsers and Comodo Disclose A Successful Certificate Authority Attack, Perhaps From Iran
- Building a better CA infrastructure
- DigiNotar Hack Highlights the Critical Failures of our SSL Web Security Model
For those of you following along at home, I posted a short update at the top of the post.
“Trust” is trust, as the meaning of the word should imply. Even under the assumption of lack of malicious intent, such a gross incompetence is enough basis for me to loose trust to a party (sometimes lack of skills causes more damage than malicious intent). Problem is most browser users don’t have the skills to remove all Turktrust CA certs from their trust chain. Why all certs? As average user can’t discern the number of compromised certs and amount of exposure, they would like to fail-to-safe based on what’s at stake which is trust and privacy.
You can create your own key pins, using an open source webapp I created, and importing them manually into chrome to protect yourself, or add them to your website headers to protect your site users. http://certpins.appspot.com
What do you think about SSLCop ? http://www.security-projects.com/?SSLCop
Well, EGO(gov.tr) supposed to run public busses and gas services around the capital Ankara. I can’t imagine why they need or how they get a subordinate certificate. Doesn’t a plain SSL cerificate suffice for webmail access or so. Although turktrust is primarily responsible for the action, also EGO should take responsibility.
Personally I’ll put TurkTrust’s further root certs to my non-trusted list and in my opinion, the browser vendors should revoke all the root certificates that belong to TurkTrust. This is a serious punishment for a “trust selling” company which will probably lead to bankrupcy. But hey, morning sunshine, Certification is also a serious business, and this is what you will face when you don’t give the business the proper attention.
Another conspiracy thoery pushes this incident into another dimension. The Mayor of Ankara, which -by structure- the president of EGO is tied to, is conducting a huge social media campaign. So the question around Turkish society arises as the government increases the push against media on freedem of speech: “Did somebody use the fraud SSL certificate to make a man-in-the-mid operation among citizens (or at least politically involved ones) ?”
Ps: Mayor is politically involved in the governing Party.
You are paranoid 🙂
This mayor?
9 Nov 2012 – Melih Gökçek, Ankara’s Mayor, has taken legal action against 600 of his Twitter followers who insulted him [http://www.neurope.eu/article/ankara-major-sues-followers-after-insults] Nice. lol
You’re Turkish, paranoia is not optional. It’s essential.
thats the one.. and yeah nuke them certs from orbit…
Looking at Chrome’s pinning implementation further, it does appear likely that Google does have the capability to receive reports for certificate pinning failures for Google’s own domains, at least if the user has Chrome’s diagnostic reporting turned on. See TransportSecurityState::ReportUMAOnPinFailure() in chromium’s transport_security_state.cc (https://github.com/adobe/chromium/blob/master/net/base/transport_security_state.cc#L677). The collected data can be seen by entering about:histograms in Chrome’s URL bar; I see increasing values for Net.PublicKeyPinSuccess when I access google.com via HTTPS. It would be nice if someone from Google could clarify whether this mechanism is how the sub-CA was discovered.
Since chromium’s pinning system is bypassed for user-installed root CAs (largely to facilitate corporate MITM proxies, parental control systems, etc…), a certificate pinning failure in Chrome must have occurred outside of a corporate proxy environment where a company-issued certificate is preinstalled on clients to facilitate interception. It is, of course, entirely possible that the Checkpoint appliance was being used in this way, and that the failures Google saw were the result of a few unconfigured clients that lacked the company root. If certificate pinning failure reports were collected by Google here, they should also have in their logs information about the users and networks impacted, which could help determine exactly who the targets of the interception were.
Of course, a truly evil attack would block Chrome’s diagnostic reporting mechanism, preventing Google from learning about the unauthorized sub-CA through that channel!
Given the shady nature of many players in the CA business and the recent interest and outpouring of funds in the area of deep packet inspection, I suspect (and I’ll freely admit that this is partially mere conjecture) that funny business around intermediate CAs is not at all uncommon. Trustwave basically said as much, acknowledging that selling intermediate certificates to enterprises is a common industry practice for a number of root CAs. Fears about national security letters and government coercion seem unnecessary when one can simply buy a sub-CA for a couple tens of thousands of dollars at most.
Even the classic “authorized” MITM schemes where a corporate network appliance intercepts SSL traffic by substituting a self-signed certificate which is preinstalled on client machines through a system image is inherently deceptive to users. While one cannot truly offer much defense when a company controls both the network and the client machines (the enterprise could just as easily surreptitiously install software to record and store full video of the user’s actions), users are deceived by our old friend the padlock into thinking that their communications are more secure than normal. It might be interesting to explore informing users when their connection relies on a CA installed without the knowing consent of the user, just as Firefox takes steps to warn users when extensions are installed by third-party malware or adware.
Finally, I am curious if we know more about the mechanism Google used to learn of this situation. I know that Chrome takes some steps toward certificate pinning (really, public key pinning) for some sites, mostly Google’s own properties (see http://www.imperialviolet.org/2011/05/04/pinning.html). I’m not sure if there’s some kind of diagnostic functionality that is capable of communicating failures back to Google, but it would be interesting if this were the case. Of course, such pinning is still vulnerable, as I understand it, if one of Google’s pinned root CAs (Verisign, Equifax, GeoTrust, etc…) have issued a sub-CA that falls into the hands of an attacker.