This means what now? How to use outputs from Time Series Forecasts to make decisions

To keep it simple, the value of AI/ML models is in the decisions they help drive. To make a decision based on the insights provided by a model you need to understand how to interpret the outputs of the models. From my perspective, the internet is filled with articles on how to build models, but few articles explain how to leverage outputs to make a decision. As such, I thought I'd start a series of explaining how to interpret the outputs of common AI/ML approaches & drive decisions.

This page will be dedicated to Time Series Techniques forecasting, and like other pages in the series is intended to be a running document, to be updated as I learn more & make edits

What are Time Series Forecasting Techniques used for :

In simple terms, you use time series analysis because you believe whatever you are forecasting is impacted by factors that change over time. For example, if you are trying to forecast # of iPods sold or the amount of rain in inches, you know that the time of the year for which you are trying to forecast matters, due to the unique presence of things like the holiday season or high temperatures during that specific period. Time series try to capture these seasonal and cyclical trends to better improve the accuracy of the overall forecast.

The standard output from a time series model is whatever you are trying to forecast, which can be a whole list of things but it's always just one variable & the output looks exactly like the input. Some examples of things people try to forecast:

• Temperature readings
• Heart rate monitoring (EKG)
• Brain monitoring (EEG)
• Quarterly sales
• Stock prices
• Automated stock trading
• Industry forecasts
• Interest rates

There are many kinds of time series forecasting techniques out there, but when it comes to how you interpret & use the outputs, it’s pretty standard. You simply compare the actuals vs predicted & calculate the error. However, given there are different approaches to calculating & representing the error, I have created a scenario to explain the approaches I prefer to use.

Scenario - Determining when to sell my house

Say you wanted to forecast the price of your house in the upcoming year so you can decide when to sell and for that someone you build a time series model for the house. To evaluate how accurate the model is & if you want to use it to make a decision you for 4 months & collect data on the actual vs predicted price

Metric # 1 : Mean Absolute Error

Actual Price = $500K

Predicted Price = $480K

Error => 500k–480k = 20k.

Say we did this for 4 months & calculated the errors

$30K, $10K ,$20K, $10K

First, you calculate the differences between actuals & predicted and take the absolute value of each difference. Then you calculate the average of the differences

($30K+ $10K +$20K+$10K)/4 = 17.5

$17.5K which means on average the predicted price undervalued or overvalued the house at 17.5K

Metric # 2 : MAPE- Mean Absolute Percentage Error

As the name implies, you aren’t looking at how off your prediction is in terms of absolute dollars but really %.

The approach is simple for 4 months take absolute percentage difference

(10% + 5%+7%+5%)/4 = 6.75%

What this means is that on average our house is being under or overvalued by 6.5% compared to its actual price

Residuals Plotting:

One issue with the above approaches is that an average can sometimes be misleading. To arrive at an average error of 17.5 K means the model could have

1. Either consistently been off by 17.5K
2. Could have been off 30K one month and right on the dot the next month

It’s important to know which scenario is happening so my suggestion is to simply plot the differences to observe what is happening

There are alternate approaches like MSE that penalize outliers but they aren’t interpretable so I prefer to simply plot the residuals

The forecaster is usually within 5-degree error each time as opposed to being wildly off by 10 degrees sometimes & on the dot rest of the time

Interpretation to help with Decision Making

Given what we have discussed so far, here is how I would use the outputs from a time series such as the predicted prices & the error metrics to make a decision on when to sell the house.

First I’d set a minimum on how much I’d need the house to go for to make the ROI wanted. Assuming the number is $460K, I’d look at the outputs and do the following

1. I’d look to see how often is the model predicts my house will be above $480 K given that on average my model over or undervalues the house by around $20K.
2. Then I’d look at both my MAPE & my residual plots to see how volatile are my predictions. From a % standpoint am I close in general to the actual value of my house, am I sometimes wildly off & sometimes wildly accurate
3. If I felt that my residuals showed few outliers & consistency in outputs and there we enough predicted points where the house would fetch me my minimum I’d move forward in considering the outputs as an input into my decision on when to sell

However, I wouldn’t simply look at the outputs alone, I’d make sure I understood historically why the price had peaked & fallen, and how that informed the model's outputs as well.

To get this insight, there are a few things I would have requested the data scientist who worked on the model to provide me with starting with:

ETS decomposition.

Time series are full of patterns and relationships & decomposition aims to identify and separate them into distinct components.

• Trend — general movement over time
• Seasonal — behaviors captured in individual seasonal periods
• Residual — everything not captured by trend and seasonal components

The reason I’d want to see this is to make sure I understood the patterns. If the price for my house just trended upwards over time, did I understand why, and was I comfortable I could explain the rate of the trend. If saw a strong seasonal component, did I also understand why could I explain that? Also, looking at the ETS decomposition would allow me to ask my data scientist informative questions such as how he was considering what he had seen from the decomposition when developing his model.

The second thing I’d ask for is a page that details the results from various models. It’s best not to put all your eggs in one basket & try out multiple approaches before proceeding forward. As an end-user, it’s not important to know the nitty-gritty of VAR vs Holtz but what would be important to ask which models capture just trend, which capture trend & seasonality, and which factor in exogenous variables. Understanding this is important before you start comparing the outputs.

Decision Making

Very rarely should you do something cause a model tells you & or because you think it’s the most sophisticated approach you have. Rather the model should serve as a key input into your overall decision-making process. In the case of the housing scenario, I’d effectively craft out a presentation that covers the following

1. What I understand about historical trends & factors that have impacted the price of my house until now
2. What upcoming events are likely to have a positive & negative impact on my house price
3. How & why do I feel comfortable that the price of my house will be within a certain range at the time I want to sell

These points would need to be covered regardless if you built a model with help from MIT or consulted a magic 8 ball. Having a story in place first, you can then best understand how to use the insights from your ETS decomposition to address points #1 & #2 and the accuracy metrics for the model (MAE, MAPE, Residual plotting) to address point #3. If you or others feel your presentation is strong, then you have best leveraged your model properly. If not, then you need to revisit whether more analytics is needed or an alternate approach is to be considered.

Hopefully, you all found this article to be useful in helping better understand the outputs of time series models. Look forward to hearing your feedback & continue to create similar content for other AI/ML techniques.