Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Implement cache for predictions #1334

Draft
wants to merge 28 commits into
base: master
Choose a base branch
from
Draft

feat: Implement cache for predictions #1334

wants to merge 28 commits into from

Conversation

DRMPN
Copy link
Collaborator

@DRMPN DRMPN commented Sep 10, 2024 •
edited
Loading

This is a 🙋 feature or enhancement.

Summary

⚠WIP⚠

  • Implements DataCache for composer.
    • Adds cache for metrics.
    • Adds cache for node's predictions.
  • Renames pipelines_cache to operations_cache.
  • Removes ancient .pyc files.

Context

Resolves #1291

@DRMPN DRMPN added enhancement New feature or request in progress task in progress composer Related to GP-composition algorithm labels Sep 10, 2024
@DRMPN DRMPN self-assigned this Sep 10, 2024
@pep8speaks
Copy link

pep8speaks commented Sep 10, 2024 •
edited
Loading

Hello @DRMPN! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 2:1: F401 'typing.List' imported but unused
Line 2:1: F401 'typing.Union' imported but unused

Line 4:1: F401 'os.getpid' imported but unused
Line 5:1: F401 'typing.List' imported but unused
Line 5:1: F401 'typing.Tuple' imported but unused
Line 5:1: F401 'typing.TypeVar' imported but unused
Line 7:1: F401 'numpy as np' imported but unused

Line 134:9: F841 local variable 'predicted_train' is assigned to but never used

Line 97:121: E501 line too long (122 > 120 characters)

Line 7:1: F401 'cProfile' imported but unused
Line 8:1: F401 'pstats' imported but unused
Line 9:1: F401 'pstats.SortKey' imported but unused

Comment last updated at 2024-12-17 08:51:30 UTC

@DRMPN DRMPN changed the title Implement cache for predictions WIP: Implement cache for predictions Sep 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 10, 2024 •
edited
Loading

Code in this pull request still contains PEP8 errors, please write the /fix-pep8 command in the comments below to create commit with automatic fixes.

Comment last updated at Tue, 17 Dec 2024 11:52:09

@DRMPN DRMPN changed the title WIP: Implement cache for predictions feat: Implement cache for predictions Sep 10, 2024
kasyanovse
kasyanovse reviewed Oct 13, 2024
View reviewed changes
fedot/core/caching/data_cache_db.py
Comment on lines +31 to +32
with closing(sqlite3.connect(self.db_path)) as conn:
with conn:
Copy link
Collaborator

@kasyanovse kasyanovse Oct 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Почему бы не реализовать DataCacheDB как синглтон, подключаясь к БД один раз при инициализации (реинициализации в новых инстансах питона в многопотоке)?

DRMPN added 14 commits November 20, 2024 00:49
@DRMPN
Copy link
Collaborator Author

DRMPN commented Nov 27, 2024

Для pipeline и nodes уже есть кэширование, если правильно понимаю, поэтому сделал кэширование для метрик.

По логам можно сделать вывод, что композирование работает быстрее без этого кэша. Я смотрел на параметр s/gen - количество затрачиваемых секунд на одну генерацию.

Возможно для более точных результатов эксперимента нужно увеличить количество запусков хотя бы 10 для каждого варианта, увеличить timeout и запускать на сервере, а не на локальном компьютере.

С кэшом метрик:

2024-11-27 22:35:45,122 - Topological features operation requires extra dependencies for time series forecasting, which are not installed. It can infuence the performance. Please install it by 'pip install fedot[extra]'
2024-11-27 22:35:45,587 - ApiComposer - Initial pipeline was fitted in 0.3 sec.
2024-11-27 22:35:45,587 - ApiComposer - Taking into account n_folds=5, estimated fit time for initial assumption is 1.5 sec.
2024-11-27 22:35:45,591 - ApiComposer - AutoML configured. Parameters tuning: False. Time limit: 3 min. Set of candidate models: ['adareg', 'catboostreg', 'dtreg', 'fast_ica', 'isolation_forest_reg', 'knnreg', 'lasso', 'lgbmreg', 'linear', 'normalization', 'pca', 'poly_features', 'ransac_lin_reg', 'ransac_non_lin_reg', 'rfr', 'ridge', 'scaling', 'sgdr', 'svr', 'xgboostreg'].
2024-11-27 22:35:45,608 - ApiComposer - Pipeline composition started.
Generations:   0%|                                                                                               | 0/10000 [00:00<?, ?gen/s]2024-11-27 22:35:47,836 - MultiprocessingDispatcher - 2 individuals out of 2 in previous population were evaluated successfully.
2024-11-27 22:36:18,285 - MultiprocessingDispatcher - 21 individuals out of 21 in previous population were evaluated successfully.
2024-11-27 22:36:44,500 - MultiprocessingDispatcher - 14 individuals out of 14 in previous population were evaluated successfully.
2024-11-27 22:37:06,453 - MultiprocessingDispatcher - 18 individuals out of 18 in previous population were evaluated successfully.
Generations:   0%|                                                                                   | 1/10000 [01:20<224:33:35, 80.85s/gen]2024-11-27 22:37:49,802 - MultiprocessingDispatcher - 24 individuals out of 24 in previous population were evaluated successfully.
2024-11-27 22:37:58,044 - MultiprocessingDispatcher - 7 individuals out of 7 in previous population were evaluated successfully.
Generations:   0%|                                                                                   | 2/10000 [02:12<176:44:30, 63.64s/gen]2024-11-27 22:38:32,890 - MultiprocessingDispatcher - 31 individuals out of 31 in previous population were evaluated successfully.
Generations:   0%|                                                                                   | 3/10000 [03:04<162:15:07, 58.43s/gen]2024-11-27 22:38:50,374 - GroupedCondition - Optimisation stopped: Time limit is reached
Generations:   0%|                                                                                   | 3/10000 [03:04<171:01:37, 61.59s/gen]
2024-11-27 22:38:50,508 - ApiComposer - Model generation finished
2024-11-27 22:38:50,535 - FEDOT logger - Final pipeline was fitted
2024-11-27 22:38:50,536 - FEDOT logger - Final pipeline: {'depth': 2, 'length': 2, 'nodes': [linear, scaling]}
linear - {}
scaling - {}
{'rmse': 0.0}
                           days  hours  minutes  seconds  milliseconds
Data Definition (fit)         0      0        0        0            34
Data Preprocessing            0      0        0        5           551
Fitting (summary)             0      0        3        5           260
Composing                     0      0        3        5           102
Train Inference               0      0        0        0            25
Tuning (composing)            0      0        0        0             0
Tuning (after)                0      0        0        0             0
Data Definition (predict)     0      0        0        0             6
Predicting                    0      0        0        0             6

Без кэша метрик:

2024-11-27 22:40:13,935 - Topological features operation requires extra dependencies for time series forecasting, which are not installed. It can infuence the performance. Please install it by 'pip install fedot[extra]'
2024-11-27 22:40:14,493 - ApiComposer - Initial pipeline was fitted in 0.5 sec.
2024-11-27 22:40:14,493 - ApiComposer - Taking into account n_folds=5, estimated fit time for initial assumption is 2.3 sec.
2024-11-27 22:40:14,498 - ApiComposer - AutoML configured. Parameters tuning: False. Time limit: 3 min. Set of candidate models: ['adareg', 'catboostreg', 'dtreg', 'fast_ica', 'isolation_forest_reg', 'knnreg', 'lasso', 'lgbmreg', 'linear', 'normalization', 'pca', 'poly_features', 'ransac_lin_reg', 'ransac_non_lin_reg', 'rfr', 'ridge', 'scaling', 'sgdr', 'svr', 'xgboostreg'].
2024-11-27 22:40:14,515 - ApiComposer - Pipeline composition started.
Generations:   0%|                                                                                               | 0/10000 [00:00<?, ?gen/s]2024-11-27 22:40:16,628 - MultiprocessingDispatcher - 2 individuals out of 2 in previous population were evaluated successfully.
2024-11-27 22:40:45,262 - MultiprocessingDispatcher - 21 individuals out of 21 in previous population were evaluated successfully.
2024-11-27 22:41:09,459 - MultiprocessingDispatcher - 14 individuals out of 14 in previous population were evaluated successfully.
2024-11-27 22:41:29,641 - MultiprocessingDispatcher - 18 individuals out of 18 in previous population were evaluated successfully.
Generations:   0%|                                                                                   | 1/10000 [01:15<208:40:15, 75.13s/gen]2024-11-27 22:42:14,612 - MultiprocessingDispatcher - 27 individuals out of 27 in previous population were evaluated successfully.
2024-11-27 22:42:20,201 - MultiprocessingDispatcher - 3 individuals out of 3 in previous population were evaluated successfully.
2024-11-27 22:42:21,971 - MultiprocessingDispatcher - 3 individuals out of 3 in previous population were evaluated successfully.
Generations:   0%|                                                                                   | 2/10000 [02:07<171:24:25, 61.72s/gen]2024-11-27 22:42:59,166 - MultiprocessingDispatcher - 28 individuals out of 28 in previous population were evaluated successfully.
Generations:   0%|                                                                                   | 3/10000 [03:02<163:16:10, 58.79s/gen]2024-11-27 22:43:17,349 - GroupedCondition - Optimisation stopped: Time limit is reached
Generations:   0%|                                                                                   | 3/10000 [03:02<169:14:14, 60.94s/gen]
2024-11-27 22:43:17,455 - ApiComposer - Model generation finished
2024-11-27 22:43:17,480 - FEDOT logger - Final pipeline was fitted
2024-11-27 22:43:17,480 - FEDOT logger - Final pipeline: {'depth': 3, 'length': 3, 'nodes': [linear, scaling, resample]}
linear - {}
scaling - {}
resample - {'balance': 'expand_minority', 'replace': False, 'balance_ratio': 1}
{'rmse': 0.0}
                           days  hours  minutes  seconds  milliseconds
Data Definition (fit)         0      0        0        0            42
Data Preprocessing            0      0        0        5           322
Fitting (summary)             0      0        3        3           452
Composing                     0      0        3        3           324
Train Inference               0      0        0        0            22
Tuning (composing)            0      0        0        0             0
Tuning (after)                0      0        0        0             0
Data Definition (predict)     0      0        0        0             7
Predicting                    0      0        0        0             7

Еще раз с кэшом метрик:

2024-11-27 22:44:42,984 - Topological features operation requires extra dependencies for time series forecasting, which are not installed. It can infuence the performance. Please install it by 'pip install fedot[extra]'
2024-11-27 22:44:43,455 - ApiComposer - Initial pipeline was fitted in 0.4 sec.
2024-11-27 22:44:43,455 - ApiComposer - Taking into account n_folds=5, estimated fit time for initial assumption is 1.8 sec.
2024-11-27 22:44:43,460 - ApiComposer - AutoML configured. Parameters tuning: False. Time limit: 3 min. Set of candidate models: ['adareg', 'catboostreg', 'dtreg', 'fast_ica', 'isolation_forest_reg', 'knnreg', 'lasso', 'lgbmreg', 'linear', 'normalization', 'pca', 'poly_features', 'ransac_lin_reg', 'ransac_non_lin_reg', 'rfr', 'ridge', 'scaling', 'sgdr', 'svr', 'xgboostreg'].
2024-11-27 22:44:43,476 - ApiComposer - Pipeline composition started.
Generations:   0%|                                                                      | 0/10000 [00:00<?, ?gen/s]2024-11-27 22:44:45,754 - MultiprocessingDispatcher - 2 individuals out of 2 in previous population were evaluated successfully.
2024-11-27 22:45:16,021 - MultiprocessingDispatcher - 21 individuals out of 21 in previous population were evaluated successfully.
2024-11-27 22:45:40,506 - MultiprocessingDispatcher - 14 individuals out of 14 in previous population were evaluated successfully.
2024-11-27 22:46:01,911 - MultiprocessingDispatcher - 18 individuals out of 18 in previous population were evaluated successfully.
Generations:   0%|                                                          | 1/10000 [01:18<217:51:55, 78.44s/gen]2024-11-27 22:46:45,402 - MultiprocessingDispatcher - 24 individuals out of 24 in previous population were evaluated successfully.
2024-11-27 22:46:53,188 - MultiprocessingDispatcher - 7 individuals out of 7 in previous population were evaluated successfully.
Generations:   0%|                                                          | 2/10000 [02:09<173:28:16, 62.46s/gen]2024-11-27 22:47:39,401 - MultiprocessingDispatcher - 32 individuals out of 32 in previous population were evaluated successfully.
Generations:   0%|                                                          | 3/10000 [03:07<167:40:59, 60.38s/gen]2024-11-27 22:47:51,160 - GroupedCondition - Optimisation stopped: Time limit is reached
Generations:   0%|                                                          | 3/10000 [03:07<173:43:41, 62.56s/gen]
2024-11-27 22:47:51,275 - ApiComposer - Model generation finished
2024-11-27 22:47:51,740 - FEDOT logger - Final pipeline was fitted
2024-11-27 22:47:51,741 - FEDOT logger - Final pipeline: {'depth': 4, 'length': 4, 'nodes': [linear, ridge, scaling, isolation_forest_reg]}
linear - {}
ridge - {}
scaling - {}
isolation_forest_reg - {}
{'rmse': 0.0}
                           days  hours  minutes  seconds  milliseconds
Data Definition (fit)         0      0        0        0            37
Data Preprocessing            0      0        0        5           166
Fitting (summary)             0      0        3        8           651
Composing                     0      0        3        8            74
Train Inference               0      0        0        0           463
Tuning (composing)            0      0        0        0             0
Tuning (after)                0      0        0        0             0
Data Definition (predict)     0      0        0        0             5
Predicting                    0      0        0        0             7

@nicl-nno
Copy link
Collaborator

Для pipeline и nodes уже есть кэширование, если правильно понимаю

Ты же вроде его модифицировал на кэширование данных?

По логам можно сделать вывод, что композирование работает быстрее без этого кэша.

А фактически сколько попаданий в кэш происходит? И сколько занимает одно обращение к нему?

Если много - то кэш метрик можно держать в памяти, а не на диске. Да и данных тоже, если в этом проблема.

Также можно задать какую-то специфическую начальную популяцию, где пайплайны сильно пересекаются по структуре. Так эффект кэша будет более заметен

@DRMPN
Copy link
Collaborator Author

DRMPN commented Nov 28, 2024

Сделал кэширование для промежуточных метрик, в текущем применении не заметил триггер.

Сделал кэширование node для fit и predict.
Количество сохранений и загрузок кэша совпадает с количеством операций для метрик.

Прогнал несколько раз, при маленьком timeout досчитывает, а при большом заканчивается всегда так:

2024-11-29 00:35:26,356 - PipelineObjectiveEvaluate - --- save evaluate metrics cache
2024-11-29 00:35:26,478 - PipelineNode - -- load fit node data_cache
2024-11-29 00:35:26,479 - PipelineObjectiveEvaluate - Pipeline is not fitted yet
2024-11-29 00:35:26,479 - MetricsObjective - Objective evaluation error for graph {'depth': 3, 'length': 3, 'nodes': [ridge, resample, scaling]} on metric rmse: Metric can not be evaluated because of: Pipeline is not fitted yet
2024-11-29 00:35:26,479 - PipelineObjectiveEvaluate - --- save evaluate metrics cache

30_min_out.txt
30_min_out_1.txt

Возможно для глубоких/широких пайплайнов (length >3) накапливается/появляется ошибка в кэше.
Мое предположение: либо можно опустить кэш для fit, либо придумать обработку посложнее.

@DRMPN
Copy link
Collaborator Author

DRMPN commented Nov 28, 2024 •
edited
Loading

Ты же вроде его модифицировал на кэширование данных?

Я operations_cache.py имел ввиду. Тогда в каких случаях он работает?

А фактически сколько попаданий в кэш происходит? И сколько занимает одно обращение к нему?

Для метрик в одном случае получилось 135 загрузок на 805 сохранений ~= 16.7%
Для node должно получиться также, ибо их количество совпадает.

Также можно задать какую-то специфическую начальную популяцию, где пайплайны сильно пересекаются по структуре.

Вроде видел, что можно создать список из нескольких пайплайнов и дать Федоту прогнать только их. Тогда он запустится без композирования и выберет самый оптимальный, да? Данный кэш именно с композированием сейчас работает.

@nicl-nno
Copy link
Collaborator

Я operations_cache.py имел ввиду. Тогда в каких случаях он работает?

Ты написал что кэш на уровне узлов и пайплайнов уже есть. Но в этом PR-е он модифицирован, поэтому это тоже может влиять не эффективность.

Возможно для глубоких/широких пайплайнов (length >3) накапливается/появляется ошибка в кэше.

Сохрани такую историю оптимизации и попробуй по ней конкретную ситуацию воспроизвести.

@nicl-nno
Copy link
Collaborator

Вроде видел, что можно создать список из нескольких пайплайнов и дать Федоту прогнать только их. Тогда он запустится без композирования и выберет самый оптимальный, да? Данный кэш именно с композированием сейчас работает

Я про то, что можно задать такие начальные условия для структурной оптимизации, которые максимизируют пользу от кэша (если это нужно для тестирования).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kasyanovse kasyanovse kasyanovse left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@DRMPN DRMPN

Labels
composer Related to GP-composition algorithm enhancement New feature or request in progress task in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

enh: Better caching
4 participants
@DRMPN @pep8speaks @nicl-nno @kasyanovse