I recently interviewed Ernie Chan, Managing Member of QTS Partners. Roger Hunter is the Chief Technology Officer of QTS Partners and he was kind enough to share his wealth of knowledge and experience implementing a highly sophisticated algorithmic trading strategy.
1. Can you tell me about your background and how you ended up at QTS as the CTO?
I grew up in New Zealand and my family moved to Hobart, Tasmania (Australia) when I was 16. I did my undergraduate BSc (Hons) at the University of Tasmania and PhD in Abelian Group Theory (a very abstract branch of mathematics) at the Institute for Advanced Studies, Australian National University, Canberra, Australia.
After teaching for a year at the University of Sydney, I was awarded a travelling postdoc fellowship to visit the leading lights in my field around the world. It turned out that the world center for my field was here in Las Cruces, NM. I intended to stay for just a year, but the research environment was wonderful and they asked me to stay. I became a tenured full professor in the Department of Mathematical Sciences, New Mexico State University.
During this time, two colleagues and I started a scientific software company and by 1993 it had grown to 35 employees and I resigned my university position to manage the company full time. I sold the company to ITP (now Thomson Reuters) and stayed on as CEO until 1998. I become interested in investing/trading during the 1990s and developed software for backtesting and research which was then used for a small hedge fund, CMM Primero.
Through the 2000's I worked with a couple of colleagues on developing and implementing trading systems. In 2010 I came across Ernie Chan's blog and first book. We contracted with Ernie for some statistical arbitrage strategies we were developing. A few months after that work finished, Ernie contacted me asking if I would like to have him manage an account using a new strategy he had developed. I agreed.
Over time, I worked on improving the strategy with my colleagues and encouraged Ernie to start a fund employing the strategy and others under development, which he did. Eventually, I developed the production trading programs and backtest system for the strategies. Ernie's investment management company QTS Capital Management was a Commodity Pool Operator at first and became a Commodity Trading Advisor as well in 2014. At that time, he invited me to be CTO. I accepted. Since the second half of 2014, assets under management have grown. I have been kept very busy developing the production trading code for the fund and for the separately managed accounts.
2. What does a typical day look like for you?
Our strategies run on servers for most of each 24 hour day, starting on Sundays at the start of the forex session and ending on Fridays at the end of the futures session. The trading programs, written by me, send out emails reporting various phases of operation and any warnings. Critical errors are notified using texts and emails.
I am usually awake for the shutdown messages from our overnight forex session trading, which can occur anytime between 4am and 8am Mountain Time. If there are critical errors at any time, then either I, or one of the other people "on duty" need to log on to the server and fix the problem. For the most part, the programs start, trade, and shut down automatically so there's little need to have someone monitoring them all the time so my time during the day is taken up researching and backtesting new strategies or fixing problems that show up in the production programs.
I am in constant email, Skype and phone contact with Ernie and our interns. A great deal of time is taken up with historical data. We get this from several sources, but the tick data we're currently using is from AlgoSeek made available under a rental scheme through QuantGO.
Downloading and extracting this data to a QuantGO Virtual Quant Lab (VQL) in the Amazon cloud takes considerable work --- for example, a single day of option data for Apple in readable (not compressed) form is often 100GB. I usually spend about an hour at the gym in the morning. I have two development stations in my office (at home), one of which is at a treadmill desk.
Most of the time I am at the computer, I am on the treadmill, logging 4-5 hours at around 1.5 mph most days, and alternating with the second system at a normal desk. I am also working with several interns in Canada, California, and Scotland. This involves Skype calls and many emails. I am frequently on Skype calls with institutional clients here in the US, in Europe, and in Australia, and I work with several local people here in New Mexico.
3. Do you have any observations on hardware, cloud services, or programming language trends in the high performance algorithmic trading space?
We're not doing HFT, although some strategies require timely executions and responses. Our servers are located at NJ1 and have about a 2 millisecond round trip to the brokers we use. QuantGO is the main cloud service we use, which is for institutional quality tick data. Renting the data from QuantGO is considerably cheaper than purchasing it outright.
4. Do you have any experience using algorithmic trading platforms you can comment on and do you have any observations on the space in general?
Our trading platform was designed and implemented by me. It is used for both backtesting and production trading. I have tried a number of systems offered by others, but they always seem to have limitations in what they can backtest and trade. It may be that some of the very expensive systems would do everything needed but having a system to which I can add features at any level makes it extremely flexible. It is often the case that a new strategy being tested requires more of the system to be built out or new features to be added.
5. Are there any specific technology trends in the algo space that you are excited about or dreading?
I'm excited about the possibilities opening up with more powerful machine learning. There's nothing I know of that instills dread.
6. What are the key benefits to building your strategy in C# and why is this such a popular choice?
I spent many years developing scientific software in C++. I switched to C# .NET because I felt it would reduce development time and debugging significantly, and that has been the case. There is little if any speed penalty with C#, especially if you study the book "Writing High Performance .NET Code" by Ben Watson, who shows that performance problems are program design issues in almost all cases. So far, there have been no instances where we've felt the need to go to C++ unmanaged code.
7. Can you comment on any unique benefits associated with other programming languages for inclusion in your technology stack?
Ernie uses Matlab for quickly trying out new ideas, and for initial full backtests. Our interns often use Python for researching ideas. When a strategy looks good, it is always verified using my system, and we find that having completely different systems in this way catches many bugs, problems and misconceptions early that might otherwise make it through to initial production testing.
8. What kind of technology considerations are needed for those building a feed handler to handle upwards of hundreds of price updates per second from one or multiple venues?
So far, our current code and servers are handling the data feeds and order processing that we are using with no difficulty. At peak times, the servers are handling several thousand messages per second and yet they are only using about 10% of the processor.
9. Do you write your live production trading strategy and backtesting engine in different languages for performance measures?
As mentioned above, our live production system is also our final arbiter for backtesting. Different languages are used for much of the backtesting because they are the preferences of different individuals. It is the case that some strategies are only easily testable on the production system --- this is usually because of the need to use terabytes of tick data.
10. What types of unexpected events occur that might create problems for a live trading system?
I believe that attempts to anticipate and correct every contingency are unlikely to be successful, and in fact can lead to major problems. Because of this, it is vital to have humans in the loop to handle unexpected events. An example is due to an Interactive Brokers (IB) bug which is still not fixed as far as I know --- if an IB gateway is running over a weekend of daylight savings change, then on Sunday startup, market data is available and orders are accepted and execute but status and executions are not reported.
In the past, code that, after many different checks, made the assumption that those orders did not arrive at the IB servers due to, for example, a momentary internet outage, would end up repeatedly sending the same order which would be executed but not reported. This happened to us early on and we were lucky to check at the right time before our accounts hit their forex leverage limit of 50x.
In general, anything and everything can go wrong!
A recent example that happened in an extreme market move was that forex market orders from some of our programs were cancelled --- we were unaware that that could happen. Internet outages can happen at any time, so you need alternative ways to take accounts out of positions. Brokers can make changes that suddenly leave you without live market data. And many more!
11. What is the biggest challenge you face as a CTO in this space?
Simply having enough time to get everything done.
12. Is there any question I should have asked you that I didn’t? If so, what would that question and answer be?
What does your background and history bring to the table?
I feel my background as a mathematician, and managing a scientific software company and its development group, has given me quite a different perspective from those who go straight into this business.