Chronos Feedback

[abdulfatir]

This is a forum to provide feedback on Chronos. Please share your thoughts on the performance of Chronos models, both in terms of the quality of forecasts and the inference speed. We are looking for both success and failure cases of Chronos. Of these, the failure cases are more important as they will help us improve the models further. Did Chronos work very well for your data? Please share this here. Did it not work so well on some time series (or the entirety of your dataset)? Definitely share it with us. Is Chronos too slow (or fast 😉) for your use case? Share it here. Whenever possible, please share the time series, metrics, and forecast plots with us.

Let's take this forward together! 👫

[AzulGarza]

congrats on the new model! here is a benchmark that compares against a statistical ensemble that might be helpful.

[abdulfatir]

Thank you for your analysis! We have extended this benchmark to all the datasets in our paper's benchmark II, making it more complete. Please check our updates in this pull request. In short, on 200K+ unique time series across 28 datasets, we show that zero-shot Chronos performs comparably to this strong ensemble of 4 statistical models while being significantly faster on average.

[Donkeyfish87]

How do I know I'm not using future information if I forecast a series? E.g. if data from 2018 is included in the training data, does that mean I cannot really forecast anything from 2017 and earlier?

[abdulfatir]

Chronos models were not trained with and do not use absolute time information, so information leakage beyond the training datasets should be a limited concern. Please check Appendix B in the paper for details on the training datasets.

[AdamDudley]

Hi, Thanks for the great model, and making it so easy to try out too.

Here is what looks like a success:
Real ecommerce store order counts, you can see COVID lockdown in March 2020 - started the prediction from the month before lockdown to view impact 4 years later and you can see it still keeps the same/similar cadence and understand the pattern)

Fail? Perhaps you can assist?
This is real data. I've cleaned it, ensured there are only ever 52 weeks even, and it just fails to pic up even simple things, such as the first week of the year the number is no higher than 20.

predicting 52 weeks sooner with overlap...

I can't get it to do more than a doodle for this dataset :(
Any ideas, I've tried many combinations of top_k, samples top_p.

[abdulfatir]

Thanks for sharing @AdamDudley!

My first guess regarding the failure cases is that it's a representation issue. Concretely, using mean scaling and quantization makes things tricky for spiky time series like the ones you have. Please check Fig. 16 in the paper for a discussion on why spiky time series are problematic. In the current tokenization framework this is a limitation and there's no quick fix. However, there's one thing that you could potentially try (I haven't tested it myself, so I am not sure if it would work): normalize the time series in some way before sending to pipeline.predict. For instance, you could try standardizing the time series x' = (x - mean)/std and then feed x' into Chronos. After getting the prediction, y', scale it back by y' * std + mean.

[AdamDudley]

Thanks for your suggestion @abdulfatir

I see, I remember that part of the paper, yes that makes sense, although you did get it to track some spikey data.

What I find odd, is that the range isn't so huge, I thought with your 4096 tokens it would cope with 4096 unique values as a range, and since the min and max of the data is 16 to under 4000, (just over 3000) I assumed we'd be ok.

so far Normalising as you suggest and putting it back, I can't yeild any improvements. However, that's likely to be equal parts my lack of time, compounded with lack of experience in this exact area, than a direct reflection of your suggestion. I will try again at some later point. But in the meantime, it's good to be clear of the limitations, thank you for that.

Am I right in thinking that if we scale the data to 'fit' within just 4096 values, it should be ok?

[abdulfatir]

@AdamDudley the 4096 bins are equally spaced between -15 to 15 after mean scaling. If after mean scaling your values fall between -15s and 15s then they can be represented. Here, s is the absolute mean scale of the time series.

[tomar-s]

Hi @AdamDudley
For the plots you shared above, I am unable to reproduce them correctly using the forecasted values. Can you please share the script for creating them, if possible?

[rkkfactspan]

Hi @abdulfatir,

I tried the model with my own dataset and have few observation, if could please comment on this :

1. When I tried this model with respect to healthcare and retail dataset , it worked fine but when worked with other time series dataset, results were not good.
2. Is there any way where we can pass the other variables because currently it is working with single variable. when we pass multiple variable the model is able to run but how will it interpret that which is the target variable in this case.
3. And last, Is there any steps to fine tune this model.

Thanks.

[abdulfatir]

@rkkfactspan thanks for trying.

1. It would be great to know what these other time series datasets are and how the results were not good. How exactly did the model fail? Was it a representation issue due to quantization as in the comment above? Are these datasets heavily dependent on covariates? etc.
2. Current Chronos variants are univariate models, i.e., they only look at the target time series independently and cannot incorporate information from other series. When you pass multiple time series in a batch, it is just doing batched univariate inference NOT multivariate forecasting. We are looking into possibilities of extending Chronos for multivariate/covariate modeling.
3. We are working towards release training/fine-tuning scripts on custom datasets.

[Otsutsukii]

Hi, I noticed, smoothing the features/context_history has some impact on the accuracy. Not exactly removing outlier values per say, but smoothing the input with for example gaussian kernel filter, or trained a CNN to learn the optimal filter. I was able to change the MAE from ~879.62 to ~453.53 ( for a single time series)

Mostly some prompt engineering stuff, could some sort of CNN filter be integrated into the chronos architecture to automatically learn the projection?

(The experiment was done without fine-tuning the chronos-mini model )