Input dataframe for statsforecast

[AFilipaVEL]

Hi all,

I'm new to Python, PySpark and StatsForecast... i'm now trying to run a simple forecast example to get familiar with this module. My first step was to create a dataframe with the mandatory columns unique_id (string), ds (date yyyy-mm-dd) and y (float).

I converted it to a pyspark.pandas.frame.DataFrame and it looks good. However, when i apply the function StatsForecast.plot(pd_df) (being pd_df my pandas dataframe), i get the following error:

AttributeError: 'NoneType' object has no attribute 'Date'

Function StatsForecast and forecast method work ok.

Do you have any idea of what might be happening here?

Thanks in advance!

[jmoralez]

Hey @AFilipaVEL, thanks for using statsforecast. Can you provide the code that you're running or the full stacktrace of the error?

[AFilipaVEL]

I apologize but now i'm confused. I read that statsForecast should receive a Pandas dataframe and that Fugue would "communicate" with Spark to speed things up. Sorry the not-so-technical terms...

Now you're saying that statsForecast is fast enough if it receives a spark dataframe? Even when there are millions of time series to model and predict?

If this is true, should i use n_jobs = -1 in statsForecast to take full advantages of my cluster?

[jmoralez]

It automatically dispatches to the corresponding backend:

• If you provide a pandas dataframe it will run locally. Here you can set n_jobs>1 to use multiprocessing in your local machine.
• If you provide a spark dataframe it will run in your cluster, here n_jobs will automatically be set to 1 to avoid nested parallelism (spark handles the parallelism here). To maximize the utilization of your cluster you should make sure that you at least have as many partitions as executors (or a multiple of them) so that spark can schedule at least one task per executor.

[AFilipaVEL]

Ok, great! So simple!

Another thing, if i'm allowed... if parallelism is ensured by providing a spark dataframe, then i assume i don't need module statsforecast.distributed.utils ? I tried to run

from statsforecast.distributed.utils import forecast

but i get the error

ModuleNotFoundError: No module named 'statsforecast.distributed.utils'

[jmoralez]

No, you don't need it. Just provide a spark dataframe to StatsForecast.forecast

[AFilipaVEL]

Wow! Many, many thanks! This was really useful!! I wish you all the best and keep up the good work!