Time series forecasting Timeout and Seed issues

[leonardo95]

Hi,

I have some questions regarding the timeout and seed features about the Fedot.
Although I set the timeout, sometimes the FEDOT surpasses that timeout. How can i prevent this? Furthermore, I'm trying to replicate some experiments by defining seeds, but the output is always different even with the same seed.

Here is my code and data:

target_column = "frequency"
task = Task(TaskTypesEnum.ts_forecasting, TsForecastingParams(forecast_length=720))
dataset_path = "data.csv"
train_input = InputData.from_csv_time_series(file_path=dataset_path, task=task, target_column=target_column, delimiter=";")

train_data, test_data = train_test_data_setup(train_input)
model = Fedot(problem='ts_forecasting', task_params= task.task_params, timeout=1, cv_folds= 2, seed = 0)
pipeline = model.fit(train_data)

out_sample_forecast = model.forecast(test_data)
print("First out:", out_sample_forecast)
print('Metrics for out-sample forecast:',
model.get_metrics(metric_names=['rmse', 'mae', 'mape']))
data.csv
model.plot_prediction()

[Lopa10ko]

Hi @leonardo95. Thanks again for reaching out.

To reproduce the results in your case, you might want to try using the set_random_seed method:

from fedot.core.utils import set_random_seed


RANDOM_SEED = 0

if __name__ == '__main__':
    set_random_seed(RANDOM_SEED)
    # ... your code

Could you elaborate more regarding the timeout issue, or perhaps provide some logs proving that the timeout is ignored by composition methods?

Also I think your code snippet is facing the same problem as earlier in the discussion #1254.

[nicl-nno]

Hi!

"seed in the Fedot class" is aimed to fix seed for all parts of AutoML (including stochastic evolutionary operators).

"set_random_seed" function is aimed to fix seed for the training of machine learning models.

If it still not helps, we will re-check that all stochastic parts are handled properly in our side.

[leonardo95]

Hi again,

I used the seed in the Fedot class with seed 0, and the results were different. Here is the log:

Run: 2
2024-02-28 15:59:14,242 - CSV data extraction - Used the column as index: "date".
2024-02-28 15:59:14,458 - LaggedTransformationImplementation - Window size of lagged transformation was changed by WindowSizeSelector from 0 to 726
2024-02-28 15:59:14,524 - AssumptionsHandler - Memory consumption for fitting of the initial pipeline in main session: current 4.4 MiB, max: 14.6 MiB
2024-02-28 15:59:14,526 - ApiComposer - Initial pipeline was fitted in 0.1 sec.
2024-02-28 15:59:14,528 - AssumptionsHandler - Preset was changed to best_quality due to fit time estimation for initial model.
2024-02-28 15:59:14,548 - ApiComposer - AutoML configured. Parameters tuning: True. Time limit: 5.0 min. Set of candidate models: ['adareg', 'ar', 'diff_filter', 'dtreg', 'ets', 'exog_ts', 'fast_ica', 'gaussian_filter', 'glm', 'isolation_forest_reg', 'lagged', 'lasso', 'lgbmreg', 'linear', 'locf', 'normalization', 'pca', 'poly_features', 'polyfit', 'ransac_lin_reg', 'ransac_non_lin_reg', 'rfr', 'ridge', 'scaling', 'sgdr', 'smoothing', 'sparse_lagged', 'svr', 'topological_features', 'ts_naive_average'].
2024-02-28 15:59:14,694 - ApiComposer - Pipeline composition started.
Generations: 0%| | 0/10000 [00:00<?, ?gen/s]2024-02-28 15:59:20,544 - MultiprocessingDispatcher - 4 individuals out of 4 in previous population were evaluated successfully.
2024-02-28 16:01:15,369 - MultiprocessingDispatcher - 21 individuals out of 21 in previous population were evaluated successfully.
2024-02-28 16:01:15,419 - GroupedCondition - Optimisation stopped: Time limit is reached
Generations: 0%| | 0/10000 [02:00<?, ?gen/s]2024-02-28 16:01:15,430 - ApiComposer - Hyperparameters tuning started with 3 min. timeout

2024-02-28 16:01:15,769 - SimultaneousTuner - Initial graph: {'depth': 3, 'length': 4, 'nodes': [ridge, ts_naive_average, ridge, sparse_lagged]}
ridge - {}
ts_naive_average - {'part_for_averaging': 1.0}
ridge - {}
sparse_lagged - {'window_size': 42, 'n_components': 0.5, 'sparse_transform': True, 'use_svd': False}
Initial metric: [12.587]
100%|██████████| 10/10 [00:03<00:00, 2.85trial/s, best loss: 12.586689277537635]
0%| | 456/100000 [02:55<10:51:06, 2.55trial/s, best loss: 11.774240647088714]
2024-02-28 16:04:14,654 - SimultaneousTuner - Final graph: {'depth': 3, 'length': 4, 'nodes': [ridge, ts_naive_average, ridge, sparse_lagged]}
ridge - {}
ts_naive_average - {'part_for_averaging': 1.0}
ridge - {}
sparse_lagged - {'window_size': 42, 'n_components': 0.5, 'sparse_transform': True, 'use_svd': False}
2024-02-28 16:04:14,657 - SimultaneousTuner - Final metric: 12.587
2024-02-28 16:04:14,674 - ApiComposer - Hyperparameters tuning finished
2024-02-28 16:04:14,819 - ApiComposer - Model generation finished
2024-02-28 16:04:15,174 - FEDOT logger - Final pipeline was fitted
2024-02-28 16:04:15,176 - FEDOT logger - Final pipeline: {'depth': 3, 'length': 4, 'nodes': [ridge, ts_naive_average, ridge, sparse_lagged]}
ridge - {}
ts_naive_average - {'part_for_averaging': 1.0}
ridge - {}
sparse_lagged - {'window_size': 42, 'n_components': 0.5, 'sparse_transform': True, 'use_svd': False}
2024-02-28 16:04:15,178 - MemoryAnalytics - Memory consumption for finish in main session: current 8.9 MiB, max: 96.2 MiB
Metrics for out-sample forecast: {'rmse': 21.34, 'mae': 18.779, 'mape': 1.291}
Time spent: 5.02 min
Run: 3
2024-02-28 16:04:15,335 - CSV data extraction - Used the column as index: "date".
2024-02-28 16:04:15,561 - LaggedTransformationImplementation - Window size of lagged transformation was changed by WindowSizeSelector from 0 to 726
2024-02-28 16:04:15,620 - AssumptionsHandler - Memory consumption for fitting of the initial pipeline in main session: current 4.4 MiB, max: 14.6 MiB
2024-02-28 16:04:15,622 - ApiComposer - Initial pipeline was fitted in 0.1 sec.
2024-02-28 16:04:15,624 - AssumptionsHandler - Preset was changed to best_quality due to fit time estimation for initial model.
2024-02-28 16:04:15,639 - ApiComposer - AutoML configured. Parameters tuning: True. Time limit: 5.0 min. Set of candidate models: ['adareg', 'ar', 'diff_filter', 'dtreg', 'ets', 'exog_ts', 'fast_ica', 'gaussian_filter', 'glm', 'isolation_forest_reg', 'lagged', 'lasso', 'lgbmreg', 'linear', 'locf', 'normalization', 'pca', 'poly_features', 'polyfit', 'ransac_lin_reg', 'ransac_non_lin_reg', 'rfr', 'ridge', 'scaling', 'sgdr', 'smoothing', 'sparse_lagged', 'svr', 'topological_features', 'ts_naive_average'].
2024-02-28 16:04:15,786 - ApiComposer - Pipeline composition started.
Generations: 0%| | 0/10000 [00:00<?, ?gen/s]2024-02-28 16:04:21,837 - MultiprocessingDispatcher - 4 individuals out of 4 in previous population were evaluated successfully.
2024-02-28 16:06:13,679 - MultiprocessingDispatcher - 21 individuals out of 21 in previous population were evaluated successfully.
2024-02-28 16:06:13,730 - GroupedCondition - Optimisation stopped: Time limit is reached
Generations: 0%| | 0/10000 [01:57<?, ?gen/s]2024-02-28 16:06:13,739 - ApiComposer - Hyperparameters tuning started with 3 min. timeout

2024-02-28 16:06:14,050 - SimultaneousTuner - Initial graph: {'depth': 3, 'length': 4, 'nodes': [ridge, ts_naive_average, ridge, sparse_lagged]}
ridge - {}
ts_naive_average - {'part_for_averaging': 1.0}
ridge - {}
sparse_lagged - {'window_size': 42, 'n_components': 0.5, 'sparse_transform': True, 'use_svd': False}
Initial metric: [12.587]
100%|██████████| 10/10 [00:03<00:00, 3.08trial/s, best loss: 12.586663952633288]
0%| | 453/100000 [02:57<11:04:32, 2.50trial/s, best loss: 11.897545563741527]
2024-02-28 16:09:15,089 - SimultaneousTuner - Final graph: {'depth': 3, 'length': 4, 'nodes': [ridge, ts_naive_average, ridge, sparse_lagged]}
ridge - {'alpha': 6.714934360611504}
ts_naive_average - {'part_for_averaging': 0.30722245545836235}
ridge - {'alpha': 8.737109070930972}
sparse_lagged - {'window_size': 41, 'n_components': 0.10155865872877783, 'sparse_transform': True, 'use_svd': True}
2024-02-28 16:09:15,091 - SimultaneousTuner - Final metric: 11.898
2024-02-28 16:09:15,102 - ApiComposer - Hyperparameters tuning finished
2024-02-28 16:09:15,238 - ApiComposer - Model generation finished
2024-02-28 16:09:15,603 - FEDOT logger - Final pipeline was fitted
2024-02-28 16:09:15,605 - FEDOT logger - Final pipeline: {'depth': 3, 'length': 4, 'nodes': [ridge, ts_naive_average, ridge, sparse_lagged]}
ridge - {'alpha': 6.714934360611504}
ts_naive_average - {'part_for_averaging': 0.30722245545836235}
ridge - {'alpha': 8.737109070930972}
sparse_lagged - {'window_size': 41, 'n_components': 0.10155865872877783, 'sparse_transform': True, 'use_svd': True}
2024-02-28 16:09:15,607 - MemoryAnalytics - Memory consumption for finish in main session: current 8.5 MiB, max: 95.8 MiB
Metrics for out-sample forecast: {'rmse': 25.059, 'mae': 22.246, 'mape': 1.535}
Time spent: 5.01 min

[leonardo95]

Hi,

Do you know the reason why the same seed leads to different results?

Thank you for the help!

[nicl-nno]

Hi!

I still researching on it, but the current guess that you are limiting the optimisation run by the timeout (so the performance is affected by backgound load). It is more deterministic to fix the number of generations.

auto_model = Fedot(problem='ts_forecasting', seed=42,
                   timeout=None,
                   num_of_generations=10, pop_size=10,
                   preset='fast_train',
                   n_jobs=-1,
                   with_tuning=False)

Of couse, it is mostly "expertimental" setup and "timeout" is more applicable for real taks.

[nicl-nno]

FYI possible solution on structural learning reprodubility problem is under development here - #1278

I will notify you when this PR will be merged.

[nicl-nno]

Hi @leonardo95.

You can also try to limit the fit time for individual pipelines during AutoML using the max_pipeline_fit_time parameter of the Fedot class. Otherwise, the large training time of intermediate candidate solutions can lead to exceeding the common timeout.

[leonardo95]

Hi,

Thank you for the suggestion! I tried limiting the max_pipeline_fit_time but I still have to wait some time. I'm probably doing something wrong. The code used is in the previous response.

Appreciate the help!

[valer1435]

Hi, @leonardo95. I have not run your code but I see what might cause a problem. You want to forecast 720 points that means in terms of multi-target regression that you will fit 720 different models (for example pipeline lagged->rfr will produce 720 random forest regressor instances) And unfortunately it's unreal to fit it in 0.5 or 1 minute.

Hope it helps

[leonardo95]

Hi,

Thank you for the response!

Makes sense. I'm using a large forecast length to test FEDOT's ability to forecast 1 year. For that, I'm using different seeds. How can I do this considering the constraint of some models taking more time than others to train?

Thank you for the help!

[VadimsAhmers]

Hi, @leonardo95!

The changes improving reproducibility in FEDOT (#1278, #1351) have been implemented and merged into the master branch. Your issue should now be resolved. Please update FEDOT to the latest version from master and check if reproducibility works as expected in your case.
For better reproducibility, we recommend not limiting the timeout parameter, as time constraints may introduce variability in the results.

Let us know if you encounter any further issues!