Andrew wrote last week about the stock market’s May 6 “flash crash”, and whether it might have been caused by a denial-of-service attack. He points to a detailed analysis by nanex.com that unpacks what happened and postulates a DoS attack as a likely cause. The nanex analysis is interesting and suggestive, but I see the situation as more complicated and even more interesting.
Before diving in, two important caveats: First, I don’t have access to raw data about what happened in the market that day, so I will accept the facts as posited by nanex. If nanex’s description is wrong or incomplete, my analysis won’t be right. Second, I am not a lawyer and am not making any claims about what is lawful or unlawful. With that out of the way …
Here’s a short version of what happened, based on the nanex data:
(1) Some market participants sent a large number of quote requests to the New York Stock Exchange (NYSE) computers.
(2) The NYSE normally puts outgoing price quotes into a queue before they are sent out. Because of the high rate of requests, this queue backed up, so that some quotes took a (relatively) long time to be sent out.
(3) A quote lists a price and a time. The NYSE determined the price at the time the quote was put into the queue, and timestamped each quote at the time it left the queue. When the queues backed up, these quotes would be “stale”, in the sense that they had an old, no-longer-accurate price — but their timestamps made them look like up-to-date quotes.
(4) These anomalous quotes confused other market participants, who falsely concluded that a stock’s price on the NYSE differed from its price on other exchanges. This misinformation destabilized the market.
(5) The faster a stock’s price changed, the more out-of-kilter the NYSE quotes would be. So instability bred more instability, and the market dropped precipitously.
The first thing to notice here is that (assuming nanex has the facts right) there appears to have been a bug in the NYSE’s system. If a quote goes out with price P and time T, recipients will assume that the price was P at time T. But the NYSE system apparently generated the price at one time (on entry to the queue) and the timestamp at another time (on exit from the queue). This is wrong: the timestamp should have been generated at the same time as the price.
But notice that this kind of bug won’t cause much trouble under normal conditions, when the queue is short so that the timestamp discrepancy is small. The problem might not have be noticed in normal operation, and might not be caught in testing, unless the testing procedure takes pains to create a long queue and to check for the consistency of timestamps with prices. This looks like the kind of bug that developers dread, where the problem only manifests under unusual conditions, when the system is under a certain kind of strain. This kind of bug is an accident waiting to happen.
To see how the accident might develop and be exploited, let’s consider the behavior of three imaginary people, Alice, Bob, and Claire.
Alice knows the NYSE has this timestamping bug. She knows that if the bug triggers and the NYSE starts issuing dodgy quotes, she can make a lot of money by exploiting the fact that she is the only market participant who has an accurate view of reality. Exploiting the others’ ignorance of real market conditions—and making a ton of money—is just a matter of technique.
Alice acts to exploit her knowledge, deliberately triggering the NYSE bug by flooding the NYSE with quote requests. The nanex analysis implies that this is probably what happened on May 6. Alice’s behavior is ethically questionable, if not illegal. But, the nanex analysis notwithstanding, deliberate triggering of the bug is not the only possibility.
Bob also knows about the bug, but he doesn’t go as far as Alice. Bob programs his systems to exploit the error condition if it happens, but he does nothing to cause the condition. He just waits. If the error condition happens naturally, he will exploit it, but he’ll take care not to cause it himself. This is ethically superior to a deliberate attack (and might be more defensible legally).
(Exercise for readers: Is it ethical for Bob to deliberately refrain from reporting the bug?)
Claire doesn’t know that the NYSE has a bug, but she is a very careful programmer, so she writes code that watches other systems for anomalous behavior and ignores systems that seem to be misbehaving. When the flash crash occurs, Claire’s code detects the dodgy NYSE quotes and ignores them. Claire makes a lot of money, because she is one of the few market participants who are not fooled by the bad quotes. Claire is ethically blameless — her virtuous programming was rewarded. But Claire’s trading behavior might look a lot like Alice’s and Bob’s, so an investigator might suspect Claire of unethical or illegal behavior.
Notice that even if there are no Alices or Bobs, but only virtuous Claires, the market might still have a flash crash and people might make a lot of money from it, even in the absence of a denial-of-service attack or indeed of any unethical behavior. The flood of quote requests that trigged the queue backup might have been caused by another bug somewhere, or by an unforeseen interaction between different systems. Only careful investigation will be able to untangle the causes and figure out who is to blame.
If the nanex analysis is at all correct, it has sobering implications. Financial markets are complex, and when we inject complex, buggy software into them, problems are likely to result. The May flash crash won’t be the last time a financial market gyrates due to software problems.
Curt Sampson — By saying HF traders “are quite often saving money for the folks who would make those trades anyway”, you’re implying that they net benefit long-term investors. But if HF traders are really saving some long-term investors money, either the HF traders are losing money themselves (in which case can you explain why any of them do it?) or they’re costing even more money from other long-term investors.
If you can explain why millisecond-level liquidity is so valuable that society overall is getting a worthwhile payback on all the profits HF traders take away from long-term investors, then go ahead and try, but please don’t try and act like they’re a direct a net savings for long-term investors.
Now that the NSA is launching an expansive program dubbed “Perfect Citizen” to detect cyber assaults, should we be rejoicing that we are more secure against cyber warfare or worried about government intrusion into internet privacy?
Now that the NSA is launching an expansive program dubbed “Perfect Citizen” to detect cyber assaults, should we be rejoicing that we are more secure against cyber warfare or worried about government intrusion into internet privacy?
http://online.wsj.com/article/SB10001424052748704545004575352983850463108.html?mod=WSJ_hpp_MIDDLENexttoWhatsNewsTop
It seems that there are some market participants that are inserting large numbers of transactions into the queues, and then canceling them before they are transacted. They may be doing this to manipulate market prices, or may be doing it to give themselves favorable queue positions if prices change in certain directions. I don’t know very much about the whole thing, but it certainly seems like there is some deliberate (and still on-going!) “cheating” going on.
Some articles I’ve read suggest this activity is illegal, you aren’t allowed to submit a bid or offer that you have no intention of fulfilling, but that’s all 3rd hand.
It’s not possible in a properly run market to submit a bid or offer that you don’t have any intention of fulfilling. Once your quote is in, it’s not up to you whether or not someone trades against it: it’s up to the others in the market who see that quote. I don’t see why it’s unreasonable for someone to be able to pull a quote out of the market if, after some amount of time, it’s not matched. Just because they wanted to sell at a given price an hour or a minute or a second ago doesn’t mean that they still want to sell at that price now.
It certainly would be possible to set up a market where, for example, once you entered a quote you had to leave it in the market for a given amount of time (at least a minute, or an hour, or the rest of the day). But that’s not going to help anybody, since all it means is that someone who’s willing to sell something for $10 right now, but doesn’t know if he will still be willing to sell it at that price later, probably won’t put in the order, or will put it in at a higher price. So someone else who’s willing to buy for $10 right now will either have to pay more than that, or won’t be able to make that trade at all.
When the guy running your pension fund wants to hedge something, do you want him to pay more than he does now?
(That said, inserting huge numbers of quotes in order to try to overwhelm the system is certainly unfair, but traders like that are already going to get the boot in most markets. E.g., on the exchange where my system was trading, we had a certain minimum ratio of executions to orders we needed to maintain, and if we didn’t maintain that, we’d get in trouble, and eventually cut off.)
As best I understand it, high-frequency trading is a practice whereby certain large financial institutions, typically hedge funds, pay exchanges to allow them to see and create quotes in milliseconds, before public markets are allowed to see them. So armed, they suck tens of billions of dollars out of trades that would happen anyway without them, on the proposition that they “improve liquidity”. What does ethics have to do with any of this?
You misunderstand the term. High-frequency trading merely refers to having computer programs analyze the market and enter orders at speeds faster than humans are capable of doing. In general, these programs would react to market changes in well under a second; reacting within milliseconds is not uncommon.
Nobody (if the market is working correctly and is fair) should be seeing quotes ahead of anybody else, except to the degree that the latency of your connection to the exchange affects how quickly you see things. With some exchanges there may be issues of access (e.g., one firm is allowed to colocate a server in the same building as the exchange computers, but another is not, and ends up with a cross-town leased line), but this is not a new problem: it’s existed ever since people not on actually the trading floor have been trading.
HFT is not limited to large institutions: for the first two years when I was writing an HFT system the company consisted basically of two people: a trader and a developer.
As for the money that HFT traders suck out, it’s not just an improvement in liquidity, but some of that money should also be compensation for giving better prices to non-market-makers (I use this term in the technical sense). See my comment on Andrew’s post for more information about this. (You can find it at this site under /blog/appel/did-denial-service-attack-cause-stock-market-flash-crash#comment-110530 ; I’ve not put in a link because apparently links to this site in comments mark the comment as spam. (!))
The short summary, HFT itself is not specifically the problem here, and banning HFT would be a lot more difficult than you think, anyway. Rather than taking aim at that, we should be trying to create markets that are fair to everyone, and be especially careful to make sure that “everyone” includes the small traders as well.
I do not see how Mark misunderstands the term high frequency trading (although it is a simplification). Whenever I have heard the term high frequency trading it is in conjunction with those high frequency traders that are paying extra to have their servers co-located with the exchange servers, and it is these servers that highlight the problem. All request might be treated fairly by the exchange, but high frequency traders that are co-located get those quotes milliseconds ahead of everyone else and can send orders milliseconds faster than everyone else. So in effect they have milliseconds more of computing time to implement trading strategies.
In my view, we now have actually made the exchanges too efficient, therefore we have to slow things down to get a more level playing field. As you say the way to solve this problem is not by banning HFT. But to think about how the system works and how it should ideally work for everyone to have a level playing field.
One strategy I have heard is to let all bids have a minimum duration of 1 second. This removes the possibility of issuing a bid, and then removing it before anyone else has a chance to react to it. Other possibilities would be to have the exchange work on fixed intervals, such as 0,1s, so that all trades where buffered and only executed once every 0,1s.
I do not think any of these redesigns will be implemented as the exchange itself is a financial entity which makes more money the more trades that are made, and therefore has an self interest in getting ore high frequency trades.
Here’s how Mark misunderstands the term “high frequency trading.”
He says it’s, “a practice whereby certain large financial institutions….” This is incorrect; small traders can also (at least when things are working properly) co-locate their servers near the exchange. The system I built was started and run with two people and less capital than the yearly salary of many traders, and is to this day co-located at the exchange at about the same “distance” as anybody else who’s not a specially-regulated member of the exchange. (There are often traders with better access to the exchange than most, but they’re almost invariably saddled with certain other disadvantages, such as having to make a market (i.e., being forced to trade even when it’s not advantageous to them) to compensate for that. They provide a service, such as allowing the market to remain more liquid than it would be otherwise, and they need to be compensated for that.)
He says that, HF traders “pay exchanges to allow them to see and create quotes in milliseconds, before public markets are allowed to see them.” No. They see quotes at the same time as any other member of the public at a similar distance from the exchange sees them.
He claims that they “they suck tens of billions of dollars out of trades that would happen anyway without them.” In fact, in a good market, they are quite often saving money for the folks who would make those trades anyway. It’s not like they come in when they see someone selling something for $10, and manage to sell it for $11 instead. If they get the trade instead of a non-HF trader, it’s because they were cheaper: $9 instead of $10.
The idea of adding delays or having a minimum bid of 1 second or anything like that doesn’t make any difference, if you think about it. First, the one second rule would affect relatively few quotes in many trading systems, since unless the market is moving abnormally quickly, many quotes tend to stay in for quite some time. (It’s preferable to keep your quotes in until you really need to take them out, because when you take out a quote and then re-enter it, you move to the “back of the queue” of people bidding or offering that quote.) Second, even if everybody were given something like a fixed one second delay, good HFT traders would still be ahead because they’re still relatively faster than others: whether orders are entered into the market after 50 and 51 milliseconds or 1050 and 1051 milliseconds, the one guy is ahead of the other.
In fact, by trying to slow down a market, what you’re really saying is that when a price is incorrect you want that price to stay incorrect for a longer period of time. This in fact is going to give an even greater advantage to more experienced traders, and hurt “the little guy” even more.
While there are certainly problems in the system that we should try to solve, most of the off-the-cuff solutions spit out by people that people who don’t understand the system are likely to hurt the people that they think they’re trying to help.
Here’s a quick heuristic to see if your solution is even worth thinking about: if you can’t write a solid 2000-3000 word essay explaining in reasonable detail just what the effect of your proposed solution will be on the various classes of traders in the market, you probably haven’t thought it through enough to make it worth proposing in public.
If you’re interested in how these sorts of markets work, a good book to check out is Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris.
. . . “open the pod-bay doors HAL.”
“I’m sorry, Dave, I can’t do that.”
@rp; In AI – behavior *specifically* incorporates probability calculations, in order to make decisions. By the very nature, it is non-deterministic.
In the engineering of non-deterministic systems, we hope that all possible outcomes are planned for by the engineer. But that’s simply not the case. Instead, its most likely outcomes, ones that the engineer can foresee. And often, in very complex systems, there will be one or more independent analyses to document the possibility of other outcomes. The responsible party makes these documents part of their engineering documents, and ensures that contingencies have been planned for. That there’s an ethical human intent to the designed operation of the system.
But no engineer will tell you that that’s a 100% guarantee of perfect safety of the system. Even in a completely deterministic non-AI system, like an automobile airbag controller, or a machine-tool controller. There will always be unforseen input conditions, and output results. As long as the designers of this system can show that they did their due-diligence in the engineering of the system, their legal liability is mostly protected.
What we’re wondering here – is in the fast-paced and competitive world of software engineering for finance and securities trading systems, IS there a rigorous engineering process that pays more than lip service to ethical “public safety” of the market? Or when designing trading systems, is the primary goal of the coder simply to get the code to run, and make the owner rich? Rigorous software engineering process is VERY expensive, and rarely profitable, rarely cutting-edge.
> The May flash crash won’t be the last time a financial market gyrates due to software problems.
Yup. “Program trading” was the hot item a while ago. There’s a great book to be written here, sort of the financial equivalent of the bridges which are claimed to collapse due to resonance.
From the data that are available, there pretty clearly has to be at least one Alice or Dave in the mix. And yeah, you pretty much have to assign culpability to Dave, the same way you would assign culpability to someone who built a robotic debt collector that learned to break people’s kneecaps. Not because Dave is necessarily aware of the AI’s actions and approving of them, but because without culpability for Dave the temptation to let AIs or other kinds of agents “learn” to game the system is way too great.
In civil law, this is a well-established principle: you’re responsible for your employee or agent’s actions.
Lets look at Dave. Dave has created a system that takes input, creates an action, and then judges the outcome. Better outcomes reinforce the action with the inputs. Lets call it a basic AI. At some point the AI has to process a large number of requests and notices a favorable outcome as a result. This slowly morphs to the point where it sends a large number of errroneous requests in order to game the system. Now Dave at this point may or may not see and understand what is happening, but definitely did not program it to react in such a manner. Is Dave culpable?