Predicting future sales figures has an important role in production planning, determining stock levels and making decisions about new investments. Numerous forecasting methods have been developed over the years, ranging from very simple ones to advanced methods based on artificial intelligence. The Greek mathematician Spyros Makridakis has been organising forecasting competitions since 1982 to find the best methods. His research led to some interesting insights.
Simple forecasting methods
Quantitative forecasting methods rely on historical data to make predictions. The simplest method of forecasting is called the naive method. This predicts that the value in the next period will be the same as in the last known period. Despite its simplicity, this method often turns out to work surprisingly well, especially in the short term. For example, the weather in 15 minutes will probably be the same as it is now.
Another simple method is to calculate the average of the last N periods. This method smooths out the fluctuations but gives the same importance to the value from the last period as one from the more distant past. As a result, this method will systematically misjudge trends. An alternative is to calculate a weighted average, in which a greater weight is assigned to recent data. A popular variant of this is 'exponential smoothing' in which weights are assigned according to an exponentially decreasing function. The 'exponential triple smoothing' method is a further refinement that takes into account seasonal effects and trends. This function is also available in Excel ("FORECAST.ETS").
Surprising power of simple methods
Surprisingly, during the first Makridakis competition in 1982, it turned out that the simplest methods outperformed the advanced methods of that time. For example, a simple exponential-smoothing method outperformed the sophisticated Box-Jenkins ARIMA method. This conclusion was confirmed several times in subsequent competitions. A second insight was that one could reduce forecast error by simply averaging multiple forecast methods. This insight was surprising at the time because it was thought that there was a best-fit method for each time series that could be determined by inspection of the data.
It wasn't until the 2018 contest that a sophisticated method developed by Uber's Slawek Smyl managed to outperform the benchmark method (an average of a few simple exponential-smoothing methods). Smyl's method was able to reduce the forecast error from 12.6 percent to 11.4 percent. Forty years of research by hundreds of scientists had finally led to a 1.2 percent improvement in the forecast error, which is a relative improvement of 9.4 percent. Smyl's method was a hybrid approach that combined exponential smoothing with weights determined by machine learning. It was also striking that the pure machine learning methods underperformed the benchmark method in that competition. Only one pure machine learning entry could surpass the naive forecast method.
Triumph of machine learning
The most recent Madridakis competition took place in 2020. This required participants to predict the sales figures of 3,049 products from ten Walmart stores over a period of 28 days. In addition to the sales figures for the last five years, they also had information about special days (holidays, sports competitions) and promotional campaigns. A total of 88,136 entries were submitted by 5,507 teams from 101 countries.
The best method, a submission by the South Korean Yeonjun In, improved the relative forecast error by 22.4 percent compared to the exponential smoothing benchmark. The top 50 entries all improved forecast error by at least 14 percent, which was significantly better than the previous competition. For the first time, the best methods were all based on pure machine learning. The best entries calculated the averages of various machine learning methods, mainly "lightGBM" methods.
Reservations
The last competition clearly demonstrates the superiority of advanced methods, but with some remarks, namely:
- The vast majority of the teams (92.5 percent) failed to do better than a simple exponential-smoothing method. In fact, only 48 percent of the teams outperformed the naive method and only 35 percent outperformed a naive method with a seasonal adjustment.
- Many teams submitted several entries, but often failed to indicate in advance which entry would score best.
- The improvement in the forecast error was assessed over various levels of aggregation (sales per store, per category, per product). This showed that the improvement depended greatly on the aggregation level. For example, while total sales per store could be predicted 31 percent better with the best method, the prediction for sales per item in a store was only 3 percent better. The more specific the prediction, the smaller the improvement.
Conclusions
The best advanced methods do significantly better than simple methods, but they’re not a panacea. It’s not easy to select a superior method in advance, and the added value - only a few percent for a specific prediction - must be weighed against the extra effort involved. For big companies with large inventories (supermarkets, large web shops, etc.) advanced methods can pay off, but for most production companies simple, easily comprehensible methods will be the best choice.
Find out more
Forecasting helps to align capacity with demand, resulting in short lead times that greatly benefit your customers. Would you like to know more about reducing lead times? Quick response manufacturing (QRM) is the strategy to grow your company by reducing lead times. Explore this strategy during our upcoming QRM Silver courses starting in Ghent (from 27 October 2023) and in Diepenbeek (from 17 November 2023).
