In the case of Forex Thor, however, which has been sitting proudly at the top of the MellyForex leaderboard for several weeks, I have to be honest at the outset and say that I don't think that those backtests are going to be of much use at all.
To explain my reasoning behind this, I feel firstly that I need to discuss something which isn't specifically related to Forex Thor and which some readers will already be fully aware of. I can only apologise in advance, therefore, if you feel that what I am about to say covers some old ground.
The problem comes about because of the exceptionally short trade duration of all of Forex Thor's trades. A quick peek at the stats page shows that the average trade duration of Forex Thor's trades in its forward test is less than 1 minute, with the longest lasting trade being all over in less time than it takes to boil an egg!
So what difference does that make exactly?
Well, the biggest limitation is that the history data used in a MetaTrader strategy test consists of just the open, high, low and close prices on the 1-minute timeframe. MetaTrader doesn't store data on a 'per-tick' basis, and this means that we have no way of knowing the correct sequence of price movements in the time between a 1-minute bar opening and the bar closing. Sure, we know what the high and low prices were in between the open and close, but did the market make its 1-minute high price before it made its low price? Or vice versa?
MetaTrader attempts to overcome this fundamental problem by using a method called Fractal Interpolation to 'guess' the sequence of ticks within each and every 1-minute price bar. But how do we know if MetaTrader has 'guessed' the tick sequence correctly? The simple fact is that we don't know, and this matters a great deal when we are trying to evaluate an EA such as Forex Thor whose typical trade duration is less than a minute, because we've no way of knowing if any of the trades taken in the tester would have actually been taken a real environment. In turn, this means that the backtest results aren't worth the paper they're printed on.
To make matters even more complicated, whenever you restart the MT4 client terminal the strategy tester will 'guess' a totally different sequence of ticks because the Fractal Interpolation algorithm employs a random process. This means that you could run two strategy tests on the same EA with the same broker using the same history data and the same price spread, and get two totally different sets of results. Confusing, or what?
I suppose I should say that, for 90% plus of EAs, this MT4 limitation won't matter in the slightest. Expert Advisors with longer trade durations and with SLs and TP of 10 pips or more are, generally speaking, unaffected by the lack of 1-minute tick data.
Now, before I get 101 emails from readers, I should also point out that I am well aware of the excellent work that Birt has done in preparing his tick data scripts and modules which are aimed at improving the accuracy of backtest results. Birt has put a lot of hard work and effort into providing something which wouldn't otherwise be available to retail traders, and I honestly think the guy deserves a medal for his dedication to the cause. The limitation of any 99% modelling quality data, however, is that the tick data should really come from the same broker as the EA is going to run live with. There is no guarantee whatsoever that the Dukascopy tick data is going to be the same as that of any other broker and, for all of Birt's good work, we still can't say for sure precisely how accurate and indicative a Dukascopy 99% modelling quality tick data backtest really is when it comes to evaluating EAs like Forex Thor which trade inside a 1-minute price bar. Most people have witnessed first hand the extent to which some EAs are broker dependent and, for such an EA, it wouldn't matter whether a strategy test used tick data or not because the tester's results could never be relied upon. Unfortunately, until a majority of brokers start releasing their own tick history data (and I should add that there is absolutely no incentive whatsoever for them to do so!), I fear the average retail punter will have to continue playing this game on an uneven surface.
About Forex Thor
Now that my rant is over, I should make it clear that the above mentioned limitations are not the fault of the Forex Thor team in any way, nor are they the fault of any other EA developer for that matter - this is, purely and simply, an MT4 problem which serves to make life difficult for EA users and developers alike!
Despite the reservations that I've attempted to explain above, I'm still going to conduct an analysis of the Forex Thor developer's own 99% Dukascopy strategy tests going back to 2007 (tick data is unavailable prior to then) for the purposes of this review, although there are some caveats which I'll try to highlight as I proceed.
Before looking at those backtests, I suppose that I should firstly say a little bit about Forex Thor and explain what little I can about how it works. The EA runs without any time filtering, 24/5 on the 30-minute chart timeframe of the EURUSD pair, and comprises of three independent strategies which it calls 'cores'. Users can use either just one core, or they can use a combination of cores, but each core needs to run in a separate chart window. Despite the fact that the EA runs on 30-minute charts, that doesn't in itself mean that it uses 30-minute trade signals. As a part of its inner workings, the EA copies the rates of the 1-minute timeframe across to its DLL which is where it conducts all its hocus pocus and spell-casting behind a mysterious veil of secrecy.
The fact that the EA does its work inside its DLL means that I can only hazard a guess as to Forex Thor's exact modus operandi, and I can't be 100% sure of what it's actually doing. That said, there are several EAs which work in a similar manner to Forex Thor. Their strategy generally involves waiting until the price hits either the top or bottom of its range and then detecting the early signs of a price reversal which the EA attempts to latch onto. Very often, the reversal signals manifest in the form of a two-way battle between the bulls and the bears which can be detected by an increase in the frequency at which incoming price ticks flow. Having taken its signal and entered a trade, the Forex Thor then runs a very tight trailing stop-loss and the vast majority of its trades result in a mere 2 or 3 pip profit or loss.
Occasionally, however, Forex Thor manages to latch onto the start of decent price move and will end up bagging itself a profit of perhaps 20 or even 30 pips.
The fact that Forex Thor runs a very tight SL also means that it can afford to take larger than norm lot sizes and still only risk a small percentage of the account. Some users might initially be a little perturbed by the trade sizes being larger than they're used to, but they needn't be alarmed.
The larger lot sizes do mean, however, that when the EA does latch onto one of its 20 or 30 pippers, the outcome will be a very healthy upwards spike in the account balance as can be seen from my own forward test of the EA on the left!
It's also worth saying that Forex Thor's trade filtering method means that it can often go a week or two without trading at all before suddenly bursting into like, and taking several trades within a few short minutes before falling back into snooze-mode for another week.
Bearing in mind that the vast majority of Forex Thor's trades involve a profit of just a couple of pips, broker selection is going to be very important otherwise the performance of these 'bread and butter' trades will suffer. The best performance will invariably be obtained by using a true ECN broker who charges a commission on each trade, rather than one who pads their commission within the spread.
When selecting a broker, users will also need to ensure that the broker's stops level is very tight. Forex Thor enters its trades using stop orders and, if you choose a broker with a 5 pip stops level, the EA will never manage to get any trades away. You really do need to be using a broker whose stops level is 0.01 pips! It's also my personal suspicion that the EA will work better with 5-digit brokers, rather than with older style 4-digit ones and, finally, the developer also suggests that good latency may improve the EA's performance in terms of trade execution speeds.
Now that we've learned a little of how Forex Thor works, it's probably time to look at the developer's backtests. The tests have been conducted using the latest version Forex Thor 2, which is only different from the original version of the EA in so far as it includes a new 'Aggression' parameter. Following a number of user requests for a modification which made the EA trade more frequently, the developer has introduced an updated version containing this new parameter. Set the aggression value to FALSE, and the EA will trade identically to the original version; Set it to TRUE, it will take the extra trades that some users were asking for, and everybody's happy. Easy, innit?
There are separate strategy tests for each of Forex Thor's cores, and they can be viewed in detail by clicking each of the balance curves below. Be aware though that the file size for the Core 1 strategy test is quite large (nearly 14MB), so the report may take a while to load which could cause a problem if you're on a slow connection.
Forex Thor Core 1 Strategy Test
Forex Thor Core 2 Strategy Test
Forex Thor Core 3 Strategy Test
A close inspection of the test settings shows that the aggression parameter has been set to false, meaning that these tests represent the original version of Forex Thor. To analyse these results, my first step was to merge all three cores into a single set of results, and I then standardised the trades to the same 0.1 lot size. This enabled me to conduct some risk simulation tests, and I have summarised the important bits of the analysis below.
It seems that Core 1 is responsible for around 59% of all of Forex Thor's trades, although all 3 cores exhibit very similar levels of performance with similar win rate and trade expectancies. Very often an analysis like the one above will show that best performance can be obtained by disabling either certain symbols or strategies, but there's nothing to suggest that is the case with Forex Thor.
Having bored most readers to death with my earlier rant about the failings of MetaTrader 4, I also thought it appropriate to parse the trades taken in the test from the date my MellyForex forward test started, and compare the tester's trades with the forward test's trades. This would give me some sort of idea of just how reliable the strategy test results were likely to be. My original Forex Thor forward test started on 14th December 2011 and the developer's strategy tests ended on 9th March 2012, so there is a period of around 3 months during which the forward tests can be compared against the 99% modelling quality strategy tests.
I've adjusted the start deposit to $1,000 and standardised the lot sizes to 0.1 lots to make the comparison as straightforward as possible. For anybody who wants to compare the physical trades, this can be done by clicking the relevant summary image below.
I don't intend to go into too much detail about the differences between my forward test trades and the Dukascopy strategy test trades for the corresponding period, other than to say that I'm sure everybody is able to spot that the strategy tester is showing around 16% more trades than have been taken in my forward test during the corresponding period, and this clearly results in better all-round performance. This simply reinforces my earlier assertion that there's no effective way of reliably predicting the future performance of an EA of Forex Thor's nature from its backtests.
Despite the obvious statistical differences, however, both equity curves do go in pretty much the same direction and the basic concept of a system which exhibits short trade durations, a low trade expectancy and a win rate of around 75% - 80% is apparent in both the strategy and forward test. In addition, it's difficult to be sure precisely how representative the comparison actually is when the sample is less than 40 trades taken over a relatively short period of time.
The other point to note is that the combined strategy tests for all 3 cores suggest a total of 4,165 trades in 1,799 days of test - in other words, over 16 trades on average each week. In addition, the longer term win rate seems to be around 55%. So far, after around 4 months in forward test, my copy of Forex Thor has been averaging just 2 trades each week and winning over 75% of its trades. Again, it's quite difficult draw any firm conclusion from this. The initial suggestion is certainly that the tester is likely to overstate the number of trades that are seen in reality, but it's also quite possible that the EA has been able to successfully reject the worst trade entries in recent weeks and only select the good ones. That may explain the anomaly.
The original version of Forex Thor has been in forward test at MellyForex for quite a while now. It has been running on a an FXCC true-ECN demo account and has already built up a reasonable amount of trade history which is currently being analysed in detail at MellyForex on this link - Forex Thor Performance Analysis and Charts.
As I stated earlier, the only difference between Forex Thor 2 and the original version is the new 'Aggression' parameter. Given the unreliability of the strategy tests, the Forex Thor developer has kindly agreed to allow me to publish two sets of the results to MellyForex.
The original version will continue to publish with its aggression parameter set to FALSE. The version that I am referring to as Forex Thor 2 has its aggression parameter set to TRUE. This will enable readers to monitor and compare the effect of this new parameter. Otherwise, the settings I'm using are identical to the original version's settings.
Both copies of the EA appear on the MellyForex leaderboard, and a detailed analysis of Forex Thor 2 is available on MellyForex.
It's very difficult to draw any firm conclusions about this EA given the unreliability of the strategy tests, and I should repeat that this is a fundamental MetaQuotes issue rather than a fault of the EA's developer. Despite those problems, I do believe that Forex Thor offers tremendous potential if used in the right environment, and I find it encouraging that my forward test is so far heading in a broadly similar direction to the strategy tests, albeit there are clear differences in the route being followed.
We do know from both the strategy and forward tests that the EA's trade expectancy is low, and that the choice of broker is very important. Users should be prepared to see the EA possibly take different trades with different brokers. I'm sure that some users will become frustrated by this aspect of the EA but, over the fullness of time (and I mean a year or more, not just a few weeks!), I would hope to see those difference even themselves out to produce a similar end result.
The other caveat is that there are no results available to call upon which might suggest how beneficial the new aggression parameter is likely to be. This means that any user who opts to set the aggression parameter to TRUE is likely to be flying blind. We do know that setting the parameter to TRUE will result in the EA taking more trades, but I suspect that its extra signals will be of a poorer quality and users could see the win rate fall and the drawdowns increase if they opt to use aggression. That's where I hope MellyForex readers will benefit from being able to compare copies of the EA running both settings in identical environments.
For anybody who is interested, the developer has recently started offering a free trial of Forex Thor 2. There are some minor limitations and, rather than go into detail about the trial here, readers can find out more about the offer by visiting the Forex Thor website.