World Class On-Call & Alerting - Free 14 Day Trial: Start Here.
To use machine learning for time series forecasting there are many things to take care of. Some of these are common to machine learning. Others are specific to time series forecasting. To get started, let’s briefly go over the basics of building a machine learning model.
The general process for adding machine learning to your system is as follows:
But, due to the nature of data in time series forecasting, we need to be careful. First and foremost, when building machine learning into your time series forecasting model, you need to take precautions to ensure data persistence and accuracy. Take a look at some of the common things you should look out for:
Be careful in performing data splits for other problems also solved by machine learning. But, in the case of time series data, you need to make sure that we do the splits based on time.
During feature engineering, you need to take care of features based on time. Maybe business problems have seasonality that affect relevant business metrics. Maybe older data is just not relevant; then you’ll need to age your data properly. Maybe there are special days of the week or holidays that affect your business metrics. Even something as trivial as day-to-day weather could affect you.
When testing the chosen model we need to decide the testing strategy. Do we do a simple test by comparing predictions with actual values? Should we do a walk forward validation? If we do a walk forward validation then how do we scale in case datasets are large? By walk forward hourly or daily basis? We can decide the granularity based on our belief of the granularity’s effect on the changes in behavior. And, we need to test our beliefs by checking the model performance via A/B tests.
You need to be able to test your assumptions because of the precautions taken when developing a machine learning model for time series forecasting. We wrote some pseudo code to give you an idea about a possible structure of a test harness that can be used to test out various models.
class MyTestHarness(object): We'll be using this convention and process for the datasets: - train -> dataset used to train the model - valid -> dataset used to check the "goodness" of the model during parameter search - test -> dataset used to check the final "goodness" of the selected model def __init__(self, data_root, model_algo, model_parameter_search_space): Here we'll initialize the data root from where data needs to be read in. We'll need to use some conventions to maintain a consistent structure of files so this test harness can be re-used with different input data (maybe for different dates etc.) model_algo - a function that will return the model we need to train model_parameters_search_space - a dictionary containing the search space boundaries for the model - the key would be the parameter - the value can be different depending on how you would want to do the search. For example: - a list containing the search values - a function defining the search strategy """ self.data_root = data_root self.model_algo = model_algo self.model_parameter_search_space = model_parameter_search_space # ... Initialize any other parameter def read_data(self): """ This will read in the train, test datasets and maintain data frames in attributes. The train/valid split will be done in the train_model method. """ def train_metric(self): """ The metric used to define the "goodness" of models during training process. This is used to search over the model parameter space. """ def test_metric(self): """ The metric used to define the goodness of models at final test. This could be different or the same as the train metric. It's also possible to have this be specific to models based on what's easier for the model to optimize for """ def train_model(self, model_algo, model_parameter_search_space): """ This trains a model after searching through the search space. You might also push the intermediate models, the parameters, train metric value into a permanent storage. This could be used to debug things if you want Particular care needs to be taken here to split the train/valid data in case of time series forecasting model. Don't do random splits, do time-based splits. """ def save_model(self, model): """ Save model, metric value in permanent storage for future reference and debugging. """ def test_model(self, model): """ This is used to evaluate the model. This can be as simple as checking your metric on the test set directly or doing a walk-forward test. """ def check_model(self): """ This is the main method used to define the workflow of this test harness. Pseudo code is given below: """ self.read_data() trained_model = self.train_model(self.model_algo, self.model_parameter_search_space) self.save_model(trained_model) self.test_model(trained_model)
After reading through this code you should have some idea about how we can use a test harness to develop a machine learning model. Such a test harness can be used to quickly iterate over various algorithms of your choice.
I’d now like to cover a few other things that weren’t discussed earlier:
Metrics are important. Training metrics, testing metrics as well as business metrics. Just like well-defined incident management metrics need to be chosen carefully, choosing your ML model metrics correctly is required for building an effective machine learning model for time series forecasting.
While developing a model, you may want to have different metrics for training as well as testing.
Sometimes you’ll want to add additional metrics during testing other than the primary testing metric. The other metrics could be used for reporting purposes. Those could help you to compare the various models that you tested based on different business concerns.
When talking about adding machine learning into a system, people talk about developing the models. They skip the part of actually using it in the system. You need to consider how you’ll integrate the model into your system before you start developing it. You need business metrics to decide whether adding machine learning is beneficial to your business or not. This metric has to be something that business people can look at and understand. This business metric cannot be technical numbers like accuracy, precision, latency, etc. This has to be something like product sales, user engagement, views, etc. – data relevant to business people.
For a machine learning model for time series forecasting, saving the following into permanent storage is required:
The metrics need to be in a format that is readable by a machine. This way, they can be read via code and checked to see whether there were any problems.
Software engineers are used to versioning the code base. But, when adding machine learning in a system, you need to consider all of the above as items you need to version. Maybe you could create a bucket in S3 and version things using directories to represent different versions.
After designing your test harness, you can move over to testing various different models.
Effective use of machine learning behind time series data can improve engineering practices from CI/CD to incident response – giving people the information they need when they need it. Check out the Incident Management Buyer’s Guide to learn all about the other tools you need in order to level up your incident management practices.