2

A bit expanding the question asked here, I want to try and test a strategy using a simulated(paper) trading. As a first step, I have tried to find any brokers who provide a convenient simulated trading platform. However, after some thought, I have decided to use simple manual "excel" approach. My strategy is based on the below assumptions:

  1. The decision whether to enter/leave a position will be made by EOD, when the closing price for the same day is known.
  2. The buy/sell action will be performed the day later, once trading starts.

So basically I'm going to wait for next trading day, see the actual prices on that day and write down the price I would have paid/received. Needless to say, that my main concern here is to create a simulation as much close to the real thing as possible. So I thought of what factors must be taken into account to make this simulation process a bit more life-like. I came up with the below list:

  1. Brokerage fees. Can be easily deducted after each performed transaction.
  2. Tax. Can be easily calculated.
  3. Availability of buyers/sellers???

The last(third) issue seem problematic since I can't figure out how to take that into an account. I'm perfectly fine to just use the close price for decision making stage, but when I execute the order the day after, I'd want to be sure that I actually can buy/sell the desired papers and I don't want to assume the day's open price to be equal to the last day's close price. So, according to my understanding, I will need to have an access to the order book to see the actual bid/ask requests, including the number of shares(or anything else). It seems to me that utilizing the order book records in my simulations, will make the results more real, thus it will incorporate all unsuccessful/partially executed orders into the calculations.

Do you agree with that? Can I actually access this info or maybe it can be calculated indirectly in some way? What other factors could I incorporate into my calculations to make it more robust?

2

I think you're on the wrong track. Getting more and more samples from the real world does not make your backtest more accurate, it just confirms that your strategy can withstand one particular sample path of a stochastic process.

The reason why you find it simple to incorporate fees, commissions, taxes, etc. is because they're a static and constant process -- well they might change over time but most definitely uncorrelated to the markets.

Modelling overnight returns or the top levels of the order book the next day is serious work. First you have to select a suitable model (that's mostly theoretical work but experience can help a lot). Then, in order to do it data-driven, you'd have to plough through thousands of days of sample data on a set of thousands of instruments to get a "feeling" (aka significant model parameters).

Apropos data mining, I think Excel might be the wrong tool for the job. Level-2 data (even just the first 10 levels) is a massive blob. For example, the NYSE OpenBook historical data weighs in at a massive 15 TB compressed (uncompressed 74 TB) for the last 10 years, and costs USD 200k.

Anyway, as for other factors to take into account:

  • Slippage! Even when the book's first 3 levels easily support your, say, 20k order, someone else might be there just before you and you'd fall through to levels 3, 4 and 5. (can be modelled with a Poisson process).
  • Position limits! You didn't say what instruments you're trading, but particularly in the commodity futures section there are absolute position limits (across all maturities) you must obey.
  • Margin calls! They are broker-initiated sudden position limits, for instance as a result of changing margin requirements (can be modelled with a Poisson process)
  • Corporate Actions. In case of bond or stock equity instruments you should know that the next day open price can be ex-rights, ex-dividend, ex-interest, ex-spin-off, etc. Such actions typically affect the quotes but don't affect your investment (as you're compensated with cash or more instruments).
  • Special corporate actions: squeeze-out, liquidation, bankruptcy, worthlessness declaration; they are so called tail events and typically modelled with a heavy-tail distribution. In your real-world backtest you will never find these because you suffer from the so called survivorship bias

So how to account for all this in a backtest? Personally, I would put in some penalty terms (as % on a return basis) for every factor you want to consider, don't hardcode them. You can then run a stress test by exploring these parameters (i.e. assign some values in the range of 0 to whatever fits). Explore them individually (only set one penalty term at a time) to get a feeling how the strategy might react to stress from that factor. Then you can run the backtest with typical (or observed) combinations of penalty factors and slowly stress them altogether.

Edit
Just to avoid confusion about terminology. A backtest in the strict sense (had I implemented this strategy X years ago, what would have happened?) won't benefit from any modelling simply because the real-world "does the sampling" for us.

However, to evaluate a strategy's robustness you should account for the additional factors and run some stress tests. If the strategy performs well in the real-world or no-stress scenario but produces losses once a tiny slippage occurs every now and again, you could conclude that the strategy is very fragile. The key is to explore the maximum stress the strategy can handle (by whatever measure); if a lot you can call the strategy robust.

The latter is what I personally call a backtest; the first procedure would go by the name "extension towards the past" or so.

Some lightweight literature:

  • Hi and thank your for your detailed answer. Modelling overnight returns was never my intention. My basic strategy is 1.Make a decision based on EOD price. 2.Buy\sell the paper from step one the next day morning. So what I really wanted to avoid is a situation when I incorrectly assume I could have bought\sold the paper for a certain price(which I just can read from any, even slightly delayed source). I do want to take into account the factors like slippage that you have suggested. – Eugene S Sep 6 '13 at 3:13
  • Well, and my point was that you're generating only one backtest path if you simply use real-world data, which is a good start to rule out right away those strategies that wouldn't have worked in the real world. On the other hand, you still don't know how the strategies work under stress. And to apply stress it's easier to have a model (slippage for instance is a Poisson process). – hroptatyr Sep 6 '13 at 5:51
  • Thank you for your insight! It is very valuable. But will artificially applied stress conditions still represent real results? I was under impression that using real data will get me closer to the actual "real world" results. Could you please point me towards any information sources where I can read more on modelling topic? What stochastic processes I can use to model other factors? (like Poisson for slippage). Thanks again! – Eugene S Sep 6 '13 at 7:02
  • 1
    I extended my factors a bit and included some modelling suggestions. Also I clarified our terminology. – hroptatyr Sep 6 '13 at 8:03
  • @hroptatyr Great answer. A lot of people leave out sensitivity analysis when testing strategies. – John Bensin Sep 6 '13 at 11:09
3

You said the decision will be made by EOD. If you've made the decision prior to the market close, I'd execute on the closing price. If you are trading stocks with any decent volume, I'd not worry about the liquidity.

If your strategy's profits are so small that your gains are significantly impacted by say, the bid/ask spread (a penny or less for liquid stocks) I'd rethink the approach. You'll find the difference between the market open and prior night close is far greater than the normal bid/ask.

  • Hi and thank you for your answer. However using the real (definitive) close price is crucial in my case since I use a specific algorithm for decision making, and it is based on this price. Hence I don't want to choose just a random close price before the actual(official) close price. My gains should not be significantly impacted by the spread but I just think that it will give me more insight and more place for improvement. – Eugene S Sep 5 '13 at 1:18
  • If you need the day's closing price to make the decision, I understand. I was not concerned about the bid/ask, but rather, the difference between today's close and tomorrow's open. – JoeTaxpayer Sep 5 '13 at 3:16
  • I understand. Well, that's probably for the simulations to find out(the significance of the difference between today's close and tomorrow's open). So I can just use the current price without being concerned of bid/ask spread. – Eugene S Sep 5 '13 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.