The topic of how to handle security vulnerabilities has been discussed for years. Wikipedia defines responsible disclosure as:
Responsible disclosure is a computer security term describing a vulnerability disclosure model. It is like full disclosure, with the addition that all stakeholders agree to allow a period of time for the vulnerability to be patched before publishing the details. Developers of hardware and software often require time and resources to repair their mistakes. Hackers and computer security scientists have the opinion that it is their social responsibility to make the public aware of vulnerabilities with a high impact. Hiding these problems could cause a feeling of false security. To avoid this, the involved parties join forces and agree on a period of time for repairing the vulnerability and preventing any future damage. Depending on the potential impact of the vulnerability, this period may vary between a few weeks and several months.
Facebook understands the value of responsible disclosure, noting:
If you give us a reasonable time to respond to your report before making any information public and make a good faith effort to avoid privacy violations, destruction of data and interruption or degradation of our service during your research, we will not bring any lawsuit against you or ask law enforcement to investigate you.
Microsoft has a similar scheme, which they call Coordinated Vulnerability Disclosure.
I believe in responsible disclosure, and have practiced it both when I was working as a vendor (at which point I worked with people who found vulnerabilities in my company’s products, ensuring that they got credit and that security problems were fixed promptly), and when I’ve found vulnerabilities in products.
So I was surprised by a recent interaction with Blackboard.com, a provider of services to schools and universities.
The problem started when I forgot my password for Fairfax County Public Schools, where my daughter is a student. Clicking the “I forgot my password” link sent me an email – but instead of an email with a link, it sent my actual password. This was common five or 10 years ago but has largely been eliminated in recent years.
So I wrote a letter to the FCPS technical support line, explaining the problem, and why it’s a bad idea to send passwords instead of a link. (In brief, because (1) if the site can send the password, that means they’re not storing it in a one-way hashed form as is best practice, (2) many people use the same password for multiple sites so any compromise of the site containing the password could compromise the user on multiple sites, and (3) email is insecure.) I got back a pleasant reply saying they would investigate, and after a few days was told that the problem was in Blackboard’s product, and they would follow up with Blackboard. It’s not the most critical security problem, but since site compromise causing password theft is a common source of identity theft, it’s an important problem to solve promptly.
Feeling that I, as a security professional, probably knew more about how to make this work than FCPS, I contacted Steve Feldman, the Vice President of Performance and Security Engineering at Blackboard (whom I found with LinkedIn. I explained to Feldman why storing and sending passwords is a problem, and asked him to investigate. Feldman responded promptly, understood the problem, and forwarded me to one of his lieutenants with the title “Director, Security, Blackboard Learn”, who also promised to investigate and respond promptly, and forwarded me to someone in technical support for Blackboard Connect, which I assume is a different product. And that’s where things went downhill.
The technical support person, after some investigation, informed me that “I have requested that my VP of Product Management generate a quote to perform this custom change.” I asked whether this was a problem in the Blackboard product or FCPS’s integration into their existing systems (which may have been performed by Blackboard, FCPS, or a third party), and what Blackboard is doing to solve this problem for all its customers – as well as questioning why a customer is being charged for a “custom change” to fix a security problem. (That’s a battle that Microsoft lost years ago, when they refused to give security patches to people with pirated copies of Windows.) At that point, the technical support person said that since I don’t work for FCPS, he can’t answer any more questions.
That’s fair enough, but when I popped the stack, I ran into the same problem – neither Feldman nor his director of security would tell me anything about whether this is a problem in their product or in the local integration, and how any resolution is being communicated to other Blackboard customers, if others are affected.
Which takes me back to my original point – responsible disclosure. It’s surprising that a vendor, in particular a company whose mission is to support education, is so uninterested in solving this problem. It’s now been almost two months since I notified them, which is not an unreasonable period of time – but since they’ve made it clear they have no intention of telling me anything, I think it’s long enough. I’m not a hacker by any stretch of the imagination – last time I did any amateur hacking was 15 years ago, and 30 years ago for any serious hacking – but I know the rules of the game.
Vendors, in general, justifiably complain when vulnerabilities are publicly disclosed without an opportunity to address the problem. This experience brought home to me that responsible disclosure isn’t the whole story – responsible handling of the disclosure by the vendor is also critical to the security ecosystem.
I laugh at this, I suspect that Fairfax County Public Schools is a government educational entity. It has been my personal experience that the U.S. Government at all levels from federal, down to state, down to county, down to city, don’t have a clue as to security practices. So this doesn’t surprise me.
It has now been over a year, so in accordance with “reasonable disclosure” I think I can now publicly disclose this security hole–not that it isn’t obvious to anyone with even a basic knowledge of the Internet.
When attempting to secure an H1-B Visa for a feign specialty occupation worker, I was directed to setup an account with the U.S. Department of Labor to file certain forms in the iCERT system, that would be checked, signed by them and sent back to be filed with the paperwork going to ICE.
The rules detailed in their terms of use included a clause that all passwords if written down must be stored in an encrypted form, and full disclosure rules in case an account or password has been compromised.
However, the irony was that their website portal (login screen) wasn’t encrypted at all. Clear text HTTP. Not just the password and username at login screen either, but when I finally did setup the account–despite the obvious security flaw–everything on the site was sent/received unencrypted HTTP traffic.
And, to this day, this website still fails to load HTTPS–to say nothing of how any data is stored.
icert.doleta.gov/index.cfm
I suspect that FCPS won’t bother to ever fix the issue with sending passwords in the clear via e-mail, let alone bother hashing them, and so long as the government doesn’t require it, the venders won’t bother adding it to their products.
As to why the venders wanted more money for security fix. Well, the key to understanding that there is again, they are a vender to our dear “government” and they don’t do anything without proper bidding documents in the which they expect to be paid for everything they do. So long as they don’t sell to the general public (which was MS’s problem) they will get away with charging even for security fixes, when and only when the government requests it.
I find this hard to believe. If you look at the ‘users’ table in a live environment, it’s pretty clear that the passwd column that column is MD5 hashed.
Additionally, Blackboard’s database schema is public.
http://library.blackboard.com/ref/726424e9-b664-4ce2-bbcb-dafc60691049/tables/as_core/users.html
That link is the public documentation for the ‘users’ table and the passwd column has the description “This is the encrypted user’s password.”
Clearly, FCPS is mistaken. The problem isn’t with Blackboard. The problem is with whatever authentication provider (http://library.blackboard.com/ref/df5b20ed-ce8d-4428-a595-a0091b23dda3/Content/_admin_server_authentication/authentication_framework_about.htm) FCPS has configured on their environment.
Has FCPS actually contacted Blackboard about this? With the power of Google and dnstools.com combined, it’s not hard to see that FCPS is using Blackboard’s managed hosting service, where Blackboard hosts the environment, handles backups, maintenance, etc., and provides assistance when requested.
In any case, I have a hard time this was intentionally (or unintentionally) configured by Blackboard.
Colin, thanks for your knowledgeable reply. I believe FCPS has contacted Blackboard, and Blackboard as much as admitted that they had been talking to FCPS about it. Whether they’ve agreed where the problem lies is beyond me, though – I think I’ll reach out to FCPS and see what I can get them to tell me, since Blackboard obviously isn’t being forthcoming (even if, as your evidence suggests, it’s not their fault).
You’re welcome.
I’m not sure what relationship FCPS has with Blackboard, but I expect that the technician that said something along the lines of “I have requested that my VP of Product Management generate a quote to perform this custom change” didn’t exactly understand the scope of this issue.
Blackboard doesn’t classify security vulnerabilities as “custom changes”. A custom change would be something along the lines of “the authentication provider that is not made by Blackboard has a security vulnerability present in it and we need part of the authentication framework changed to resolve this vulnerability until it is fixed upstream”. When Blackboard has to rewrite part of their product and recompile the modified files and Java classes, for something that isn’t a security vulnerability or an outright bug, it’s a custom change.
I don’t mean to come off as rude, but you make a lot of broad assumptions about Blackboard as a company and how secure their product is as if they’re absolute fact. As a security professional myself, I understand the points your making, but as someone also familiar with Blackboard Learn, it’s database, and how it works and how Blackboard operates as a company, a lot of what you’ve written isn’t true.
I have a simpler explanation: internal budgeting problems.
When engineers talk about change requests and billing estimates, that is a sign there is a history of overbooking them. It sounds like this engineering group is being told to implement more things than they have time for, and now they are supposed to fix this security problem on top of everything else. One way or another, if they are to spend time on this problem, then they need to be authorized to drop or defer some other specific bundle of work.
Keep in mind that you really are asking them to eat a cost here. No sound business is going to be excited by the prospect. Furthermore, keep in mind that sometimes the legitimate right answer really is to say no, we’re going to live with the security problem.
Pragmatically, maybe back up a little on your long chain of delegation in your communication? Explain to people earlier in the chain that people later in the chain are balking about the budget impact.
As soon as you heard “I have requested that my VP of Product Management generate a quote to perform this custom change” you should have recognized that the company was not acting in good faith. Because that translates to “We are trying to decide how much to charge your school district for fixing this flaw in our product, and don’t plan to roll the fix out to any other school district.”
Paul, I entirely agree – I did realize it (potentially) wasn’t good faith when they said they were asking for money, and challenged them on why they were charging for a security fix. I say “potentially” because it’s possible that the problem isn’t with their product, but rather with something the school system did to integrate the product into their existing infrastructure, and maybe the charge would be to improve the integration to reduce the risk – and if that were the case, perhaps the problem isn’t universal to their customers. (Giving Blackboard the benefit of the doubt.) But since Blackboard won’t tell me anything, it’s hard to say.
I appreciate all the feedback – I’m trying to decide whether my next step is to escalate the issue in FCPS (I have some contacts at high levels) or to bring this discussion to the attention of the Blackboard folks, so they see that their actions are not well received, or both. Opinions?
I wonder if its really just “responsible handling of the report”. I have sent in multiple reports, and after a while its the same pattern “oh thank you! we will fix it, great!” and they are all very polite. But, there is no real change in their software development styles: at that point, do they deserve the public shaming and the the threat of full disclosure?
No quarter ever for a software vendor who uses litigation or law enforcement in an effort to prevent publication of a security vulnerability. Any vendor who has used that tactic should forever after face only “full disclosure”, immediately, in full detail, and to the entire public.
A software vendor should certainly get no more protection for their reputation than is given to classified military documents. In the Pentagon Papers case (“New York Times v. United States”, US Supreme Court, June 30 1971, https://www.gwu.edu/~nsarchiv/NSAEBB/NSAEBB48/decision.pdf) the Court denounced prior restraint. The US argued that these were military secrets with lives on the line. A software vendor risks only their reputation, and then only because of their own error in creating the vulnerability in the first place.
A vendor who turns to litigation AFTER publication should be evaluted on a case-by-case basis, just as government secrets are protected by sanctions after publication.
If a vendor has ever sued a security researcher or has involved law enforcement, in an effort to prevent a disclosure, they have no right to “responsible disclosure”. Providing them with advance notice is simply giving them time to pull the same stunts again, to the detriment of the public.
I reserve judgment for vendors who sue or involve law enforcement AFTER a disclosure. Those should be evaluted on a case-by-case basis. Doing so in an effort at prior restraint should always trigger a permanent change to “full disclosure” for any future incidents involving that vendor.
These are the same rules that the US Government plays by. Prior restraint is almost never sanctioned by the courts, though sanctions after the fact are quite possible. If it’s protection enough standard for classified military material (see the Pentagon Papers case, “New York Times v. United States”, US Supreme Court, June 30 1971, https://www.gwu.edu/~nsarchiv/NSAEBB/NSAEBB48/decision.pdf), then it’s enough for a commercial software vendor who risks only their reputation.
I think you hit the nail on the head here–vendor actions are important. I’d go a step further, and comment that Blackboard’s action here carries massive externalities. They alter your willingness, and the willingness of others, to bother going through the effort of private communications.
Wait a minute! You wrote a piece of exactly this event:
https://freedom-to-tinker.com/blog/felten/security-research-muzzled-georgia/
Wow, I didn’t realize that Ed had written about the same topic – funny that 9 years later, Blackboard is still making the same mistakes.
Blackboard has a history of suing security researchers:
http://www.theregister.co.uk/2003/04/15/dmca_threats_gag_security_researchers/
Unfortunately, I’m not surprised. It is my experience that this is standard M.O.
I’m sad that they have this attitude — whether it be in IT or other realms of our lives.
You vs. policy. It was an easy way to either dismiss your concern or cut you out and handle it themselves — none of your beeswax after this moment. After all, you exposed a weakness which might affect sales.
Now If I were in charge of that environment, I’d call you personally to thank you for stopping a possible avalanche.
But with good PR Firm behind them, they can patch your fix, call it a new version “now with enhanced security” and up the price 20 percent.