Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cli/notebooks for (performance) evaluation #78

Open
5 tasks
chenkins opened this issue Jul 24, 2024 · 0 comments · May be fixed by #88
Open
5 tasks

Cli/notebooks for (performance) evaluation #78

chenkins opened this issue Jul 24, 2024 · 0 comments · May be fixed by #88
Labels
enhancement New feature or request

Comments

@chenkins
Copy link
Contributor

chenkins commented Jul 24, 2024

Objectives
First benchmark tooling. Further appetite will come with eating...

Todos

  • Log current commit/git diff
  • ability to run with cProfile / and document who to visualise the profils with snakeviz
  • cli/notebook for matrix training/evaluation (see Flatland policy evaluation and training cli. #74), write logs of one "run" to folder and upload to S3
  • cli/notebook for generating a matrix of results from one run (e.g. comparing average epoch time across different algorithms/policies)
  • cli/notebook for comparing different results (e.g. timeline of average epoch times before and after a certain refactoring)

Minimal testing requirements
Think about what behavior shall be covered by a test (qualitatively describe each test).

Additional context
Preparation for #59
Depends on #77 #74

@chenkins chenkins added the enhancement New feature or request label Jul 24, 2024
@chenkins chenkins linked a pull request Nov 20, 2024 that will close this issue
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant