Most demand planners are evaluated on their ability to deliver accurate forecasts. There are plenty of metrics — the Mean Absolute Percentage Error (or MAPE) is the most well-known and widely used — that can help companies understand how well its sales forecasting team is performing. But forecasting is complex; using only MAPE to evaluate your demand planner’s performance is an oversimplification that can lead to an undervaluing of extremely capable employees. Thus we caution against using it as the main or sole key performance indicator.
Outside factors impact accuracy
It’s important to recognize MAPE’s limitations. Context is everything, and before you can use your forecast accuracy (or lack thereof) to make business and personnel decisions, you need to understand the bigger picture. There may be factors beyond your demand planner’s control that are leading to a final forecast with a lower degree of accuracy. These include:
- Low quality data — It may seem obvious that basing a forecast off inaccurate or incomplete data is a recipe for disaster, but demand planners have to work with what they’re given. Unfortunately, many companies rely on manual processes to collect, store, and share information, such as human data entry or passing Excel spreadsheets back and forth via email. Such an approach increases the chances that data points will be missed or errors will be introduced along the way, inevitably skewing your forecasts.
- High industry volatility — Some industries experience higher volatility in demand than others. Companies working in travel, hospitality, or food services have a harder time anticipating demand than those dealing with precious metals or insurance, for example, as there can be a number of reasons for a consumer’s choice to travel to or dine at a particular place. Since consumer demand fluctuates widely from season to season and year to year, companies in these industries will inevitably have higher MAPEs.
- Low forecastability of certain products — Simply put, certain products lend itself to forecasting; others do not. Perhaps the clearest examples of products with low forecastability are new products or product lines, since planners don’t have a lot of historical data to draw from when determining a statistical baseline. It would have been difficult, for instance, to forecast demand for the iPad when it was first introduced in 2010.
Not all MAPEs are created equal
How are you measuring your MAPE? Different approaches yield different results. Measuring the MAPE on product- or customer-specific basis, for instance, can mean you’re evaluating what specific item has been sold to a specific customer. But given that you probably have numerous SKUs and many customers, this makes it difficult to get a sense for the big picture and leaves you putting out small fires instead of gaining a more comprehensive understanding of how accurate your forecasts are on the whole.
The goal can be arbitrary
When viewing forecast accuracy through the MAPE lens, it’s easy for things to become black and white, where raw accuracy is the end in itself and the goal is a MAPE of 0 or another arbitrary target. This is not only an impossible goal, it misses the point: it’s more important to see improvement than perfection. If your forecast accuracy is improving from one month to the next, it’s more important than reaching a specific number.
There are other KPIs that can provide a fuller picture
While forecast accuracy is indeed a valuable metric to track, companies shouldn’t judge success or failure on this alone. Another major metric to evaluate is bias. Unlike MAPE, which only tells you the extent of the error in absolute terms — that your forecast was 30% wrong, for instance — bias tells you if you’ve made a misstep in a specific direction. Since you ultimately use these forecasts to determine your ideal inventory levels, evaluating bias helps you determine whether you’re likely to be carrying too much inventory or whether you may be at risk for a stock out.
Companies should also measure the Forecast VAlue Add (FVA). With FVA, you’re not looking at the end result, so much as examining how the error rate has changed throughout the forecasting process. This allows you to determine how effective step was in strengthening the accuracy of the final forecast — perhaps the best indicator as to how much value your demand planner is adding to the process.
Of course, since forecasting is complex, it can seem easier to focus on one outcome and one number (i.e., MAPE). A thorough analysis of the entire process using a range of metrics can be overwhelming. But it doesn’t have to be. Click here to engage with one of John Galt’s supply chain management consultants and learn how our best-in-class software solutions, Atlas Planning and ForecastX, can help you easily implement techniques that will monitor — and drive — improvements in your sales forecasts.
