-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reconsider relation of usage examples and unit tests #86
Comments
What I forgot: We should also find a way for the test files to define a range of LilyPond versions they are intended for. So far there is the provision of header fields specifying minimal and maximal versions a "snippet" is known to work with. But that is something only LilyPond can determine when a library file is included. A Python script doesn't know about this when using a test file that includes such an incompatible file. But I'd like to keep such compatibility information in one place exclusively, so I can think of another strategy:
|
I agree with you that we should have the possibility to separate test and examples and I think that the chages to the current code will be minimal to introduce such a feature. However I think that we should keep the example files among the tested ones by default, not considering them as unit tests, but just to check that they compile on all supported versions.
I don't have yet an opinion on how to implement this, but I agree that the information should be included only in one place. As for capturing the error message of the lilypond script you can parse the standard error stream from python. For instance take a look at line 275 of |
Am 28.02.2015 12:38, schrieb Matteo Ceccarello:
OK, I'll do the following then:
However, I think .ily files should by default not be compiled. So one Library maintainers are encouraged to cover all possible combinations
Oh yes, I vaguely had this idea too but somehow didn't think further in
The only thing we wouldn't be able to cover with this is a test that is |
OK.
I'm fine with this, one can just include them by hand if necessary.
Could you please wait a little before making these changes? I'm thinking of an alternative approach that should handle both this issue and the module version declartion. |
Oh, one more thing: how should we handle expected failures? |
I'll think about this, but I have some ideas. First of all, this should be rather rare cases and should not happen in We could either add a flag inside the file, which is determined in |
As for versions (minimal, maximal, and module version) I was thinking about something like the following \declareModule "gridly" \with {
author = "Matteo Ceccarello"
short-description = "Simple segmented grid framework"
version = "0.4.0"
minimal-lilypond-version = "2.18.2"
maximal-lilypond-version = "2.19.15"
description = "A brief descrtption of the library."
category = "..."
tags = "...."
} in place of the current However, I don't know how much tooling there already is around openLilyLib |
Sounds good. I'll think about it when family duties are over later tonight.Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet. |
I think defining such a command is a good idea as it would avoid conflicts of purpose between header fields and module definition (e.g. it's not necessary anymore to use prefixes). However, version conflicts should still be performed in Existing code is mainly present in the doc-generation script - which has to be redone anyway, so that shouldn't be an issue. Please not that minimal and maximal LilyPond versions are optional and should only be entered when there is a known limit (e.g. "this module won't work prior to 2.19.3") |
@Cecca Please have a look at the
The process of
This allows the main file to respond to the presence/values of any options set in What do you think? I think the versioning should be done on library level while e.g. compatibility with LilyPond versions should (could) be on a lower per-module level. |
(See the |
Thank you Urs. Right now I'm busy, I'll have a look at this later this evening. Anyway, I linke this approach.
I don't quite get the difference between library level and module level. |
This doesn't come as a surprise, as GridLY doesn't expose this difference. But take ScholarLY as an example. This is a library, but it contains a number of modules / items. So far there is |
OK, I understand. I think that having different modules of the same library support different lilypond versions can be potentially confusing for a user. Im my opinion if a module has different requirements on the LilyPond version with respect to the rest of the library, then it should be in a separate library. |
This "issue" is meant for discussion. We should agree upon an approach here before starting any coding.
#85 brings up an issue I wanted to open anyway (just didn't find the opportunity until now).
I think we still have a problematic mix of purposes with the automated tests.
We collect all files from a
usage-examples
directory for the automated tests if they aren't explicitly excluded. Sounds like a good idea but it turns out that it is problematic because the usage examples we have so far (for GridLY and e.g. partial compilation) don't cover all situations.Usage examples are part of the documentation, which is particularly obvious with the GridLY example. With such usage examples it is a natural approach to provide alternatives that the user can experiment with by uncommenting certain lines to activate alternative behaviour.
So it seems usage examples are similar but not identical with unit tests in their organization. Therefore I propose a different policy for usage examples and unit tests:
Libraries can (i.e. are strongly encouraged to) have
usage-examples
directory (usually on library top-level)All
*.ly
files in this (recursively) will be used for the auto-generated documentation(which hasn't been implemented yet. We'll have to think about the explicit in-/exclude options in that context separately)
unit-tests
directoryAll
*.ly*
files in this will be used for automated tests with Travis. These tests are like LilyPond's regression tests, and library maintainers are responsible for keeping the tests up to date and comprehensive. All relevant commands and constellations should be covered by tests. Usually it is a good idea to write one*.ily
file containing the main includes and a bunch of smaller*.ly
files covering individual tests or coherent groups of tests. This will also make possible failures point more directly to the cause.In cases where the usage examples are appropriate as unit tests it is not necessary to duplicate them as test files. Instead they can simply be included through the
.automated-tests-include
approach.I think this approach would reduce some collisions of concerns while still being straightforward and not imposing too much overhead or complexity on library maintainers.
And it shouldn't be complicated to change the implementation. The moment would be a good one because we a) do have a few examples we can use as proof-of-concept and b) we don't have too many examples that would have to be updated.
The text was updated successfully, but these errors were encountered: