Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tool testing improvements #117

Open
5 of 9 tasks
mvdbeek opened this issue Jan 3, 2019 · 3 comments
Open
5 of 9 tasks

Tool testing improvements #117

mvdbeek opened this issue Jan 3, 2019 · 3 comments

Comments

@mvdbeek
Copy link
Member

mvdbeek commented Jan 3, 2019

These are a couple of things I found while implementing tool tests:

  • tests only run of the test data is available from where the tests are triggered (this should be fixed with a galaxy-lib update containing Unprivileged remote tool tests galaxy#7160)
  • tests seem to require an admin API key, that shouldn't be necessary with Unprivileged remote tool tests galaxy#7160
  • We create a new history for every test, even if this test is for the same tool
  • We just raise JobOutputExceptions, these need to be displayed properly
  • We should have some reporting, ideally re-using some things from planemo (manual planemo test_reports tool_test_output.json works fine)
  • Optionally delete test histories after run
  • Skip tests that use reference data
  • Move to next test when job becomes paused
  • Expose test timeout as command line option
@selten
Copy link

selten commented May 15, 2020

@mvdbeek would it also be an idea to add the ability to remove the histories after all tests are finished (or at least as a command line option)?

@mvdbeek
Copy link
Member Author

mvdbeek commented May 15, 2020

Yep, that would be a good option. I've added this along with some other things we've recently come across

@selten
Copy link

selten commented Jun 17, 2020

shed-tools test should have an option to exit with exit code 1 in case of failed tests. For me it seems to exit with exit code 0 even if it has failed tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants