Article - Measuring Technical Indicator Effectiveness
See the full list of articles.
2006-08-11
Introduction
Measuring the effectiveness of a buy signal, based on technical indicators, with back-testing is not an easy task. It's made even more complicated by the fact that traditional back-testing inherently tests the strength of two components: the buy and sell signals.
In an effort to specifically test just the buy signal we tried out a different kind of back-testing. We call it the "future value" test and it works like this:
- Every trading day the market is scored and ranked for the buy signal
- The test looks ahead in the data a certain number of days to see how the top few stock picks performed
- Statistics are gathered, such as: Maximum Price Change, Days Until Max, Win %, Win By At Least 1%, 2%, ..., 5%. These are taken based as the average of the top-5 scored stocks by the signal.
By isolating the buy signal we get better data for comparison of indicators. Also, we potentially get more accurate results for a situation where a trader is using a buy-signal to help filter the market for buys but then makes a manual sell decision.
Indicators Used
The following indicators were used in various combinations during the testing:
- Bollinger Bands [5 days look-behind, 2.0 band multiple]
- Relative Strength Index [14 days look-behind]
- Commodity Channel Index [20 days look-behind]
- MACD [10 days / 5 days]
- Fastest Decreasing Price [3 days look-behind]
- Price Filter [$1.0 to $5.0]
These were chosen because the first four are among the most popular indicators. The other two will be used to augment the first ones. This page may be useful: technical indicator information.
Test Parameters
The test was run over all of 2005 using a maximum look-ahead of 30 days to check the maximum value attained by the stock, after the pick.
Testing the core indicators
The first step of the analysis is to run the test with the four "core" indicators. This should give a good starting point as well as insight into how naturally effective they are.
Here were the results:
| Indicator | Max Change % | Days Until Max | Win % | Win % Over 1% | Win % Over 3% | Win % Over 5% |
|---|---|---|---|---|---|---|
| BB | 21% | 9 | 86% | 74% | 54% | 42% |
| CCI | 6% | 9 | 83% | 71% | 53% | 40% |
| RSI | 6% | 8 | 84% | 72% | 51% | 40% |
| MACD | 16% | 8 | 81% | 77% | 66% | 55% |
Explanation of statistics
Since this is the first time presenting these types of results, they should be explained:
- Max Change % - The average maximum price reached, as a percentage of original, over the 30 days immediately after the signal "picked" the stock (the "look-ahead").
- Days Until Max - The average days until the maximum price was reached.
- Win % - The percentage of "picks" that had positive price movement over the look-ahead period.
- Win % Over 1% (and 3%, 5%) - The percentage of "picks" that had positive price movements, by at least the specific percent of price, over the look-ahead period.
Analysis
Notice how both CCI and RSI had much lower Max Change % than the other two but their "real" performance, as measured by the win rates, is almost identical. The win rates seem decent, although when remembering the note above shouldn't surprise us since it's a bull market.
Trying to catch the bottom
The results of the core indicators were encouraging but the most important measure, win rate over 5%, was under 50%. If we assume that stocks should rebound by at least 5% after a big hit (the subject for another article) then we're not catching the bottom. Using the Fastest Decreasing Price indicator is a popular method for trying to catch closer to the bottom.
Let's run the test again and add FDP to each core indicator as an aggregate buy signal. The original results are grouped in for comparison.
| Indicator | Max Change % | Days Until Max | Win % | Win % Over 1% | Win % Over 3% | Win % Over 5% |
|---|---|---|---|---|---|---|
| BB | 21% | 9 | 86% | 74% | 54% | 42% |
| BB + FDP | 20% | 8 | 79% | 72% | 58% | 48% |
| CCI | 6% | 9 | 83% | 71% | 53% | 40% |
| CCI + FDP | 10% | 8 | 78% | 72% | 60% | 50% |
| RSI | 6% | 8 | 84% | 72% | 51% | 40% |
| RSI + FDP | 12% | 7 | 78% | 72% | 61% | 51% |
| MACD | 16% | 8 | 81% | 77% | 66% | 55% |
| MACD + FDP | 16% | 8 | 78% | 73% | 63% | 54% |
Analysis
The overall win rate seems to suffer but the win rate over 5% gets a big boost for all except MACD. One theory is that MACD might be better at catching the bottoms by itself.
Adding a price filter
Groupings of indicators are sometimes much more effective when restricted to certain areas of the market. Small cap stocks and generally lower-priced stocks tend to have higher risk/reward ratios. We'll use a price filter (PF) in this case but keep out of the pennies, so in the range of $1-$5.
| Indicator | Max Change % | Days Until Max | Win % | Win % Over 1% | Win % Over 3% | Win % Over 5% |
|---|---|---|---|---|---|---|
| BB | 21% | 9 | 86% | 74% | 54% | 42% |
| BB + FDP | 20% | 8 | 79% | 72% | 58% | 48% |
| BB + FDP + PF | 15% | 7 | 81% | 75% | 66% | 58% |
| CCI | 6% | 9 | 83% | 71% | 53% | 40% |
| CCI + FDP | 10% | 8 | 78% | 72% | 60% | 50% |
| CCI + FDP + PF | 16% | 7 | 81% | 76% | 68% | 60% |
| RSI | 6% | 8 | 84% | 72% | 51% | 40% |
| RSI + FDP | 12% | 7 | 78% | 72% | 61% | 51% |
| RSI + FDP + PF | 16% | 6 | 78% | 74% | 66% | 59% |
| MACD | 16% | 8 | 81% | 77% | 66% | 55% |
| MACD + FDP | 16% | 8 | 78% | 73% | 63% | 54% |
| MACD + FDP + PF | 14% | 7 | 81% | 76% | 65% | 57% |
Analysis
Somewhat surprisingly, this was another big boost for BB, CCI and RSI. MACD improved mildly. Going back to the original comment on the max change %, notice how the addition of FDP and PF increased it drastically for CCI and RSI but lowered it for BB and MACD.
It's also interesting that when each of the core indicators is grouped with FDP and PF their results are almost identical. These indicators are all fundamentally looking at price movement (momentum) so that's not a surprise in the end.
Conclusion
We've seen some good data on levels of performance we might expect when using a semi-automated buy signal and a manual sell signal. The indicators we used here were just the tip of the iceberg, so keep on testing new ideas.
Feedback
Please send us any feedback or requests for other analysis to feedback.