[Spec] Specify if there should be a trailing end-of-phrase line

[Tuupertunut]

Suggestion

Specify in the Ultrastar format specification whether there must/may/must not be an end-of-phrase (-) line at the end of a song, right before the end-of-file (E) line or at the end of each player's notes in a duet song.

Use case

The Ultrastar format specification does not specify whether a trailing end-of-phrase line is allowed. Currently there are tools in the Ultrastar ecosystem that produce trailing end-of-phrases and others that fail to read them.

Examples:

• Ultrastar Play 0.9.0 song editor always saves files with a trailing end-of-phrase line.
• If Ultrastar Deluxe 2023.6.0 has any songs in its collection with a trailing end-of-phrase line, it complains at startup that there is an error loading a song. It interprets a trailing end-of-phrase as a new phrase without any notes.
• If Ultrastar Deluxe 2023.6.0 encounters a trailing end-of-phrase line in a two player duet song, it will abruptly quit the song at the last note.

Extra info/examples/attachments

No response

[Baklap4]

Im not at a computer but i think the end of file E is/should be scrapped. Its unnecessary as the end of file is easily detected
As for end of line the - can be used, but noy sure if it should be a must

[bohning]

I agree that this should be specified - I would vote against a trailing -, both right before the final E (which I like and would vote for - it explicitly shows the end of the song, indicating that nothing has been cut off or lost at the end and it is complete) and between tracks of a duet. I believe UltraStar Manager doesn’t handle trailing line breaks well either. The Syncer will remove them when downloading from usdb.

[Baklap4]

So in conclusion (and correct me if i'm wrong) :
❌ We shouldn't use both - and E
❌ We shouldn't use - to determine the end of P1 text. That's where the P2 delimiter is for
✅ We should use only E at the end of file
✅ Only use - for line-breaks where a next line is coming

[codello]

I‘m also not a big fan of the trailing line-breaks or the trailing E.

I definitively agree that there should not be a trailing line-break. But I also think that the existence of a trailing line-break should not cause programs to reject a file. Maybe the spec could allow trailing line breaks syntactically but define them as semantically irrelevant and recommend not to use them?

As for the trailing E, maybe it could be optional? I think that‘s how USDX handles it currently.

[codello]

The current phrasing of the spec includes an optional E at the end of the input (or rather everything after an optional trailing E is ignored). Note that the in the current phrasing the E does not need to be on its own line. If a line starts with E everything else on the line and all subsequent lines are ignored.

format/spec.md

Lines 86 to 88 in 23bf930

	song = file-header
	file-body
	[ %x45 *char ] ; E

The current phrasing also allows for end-of-phrase markers (I use this term instead of line breaks to differentiate them from newlines within the file) to appear anywhere within the body of a song.

format/spec.md

Lines 722 to 725 in 23bf930

	body = *( note /
	end-of-phrase /
	voice-change /
	empty-line )

By this spec it is syntactically valid to have end-of -phrase markers anywhere (even before the first note, after the last note or multiple times in a row). The spec includes this addition which discourages the use of leading or trailing end-of-phrase markers:

format/spec.md

Lines 841 to 842 in 23bf930

	An end-of-phrase SHOULD NOT appear between the start beat (inclusive) of a note and its end beat (exclusive).
	An end-of-phrase marker SHOULD NOT appear before the start time of the first note or after the start time of the last note.

Does this capture the consensus of this issue? I'm not quite sure how we can best vote on this. Maybe we split this into two votes?

[marwin89]

Yes this a good description of the problems in this issue.

Let's vote about this like:

Handling End-of-File Marker "E"	Handling End-Of-Phrase Marker "-"
E only has to be in it's own line - yes 👍🏻	End-Of-Phrase Marker has to be placed only once for each phrase - ❤️
E only has to be in it's own line - no 👎🏻	End-Of-Phrase Marker can be used anywhere multiple times - 👀

[bohning]

I would consider the final E the final end-of-phrase marker, thus it is discouraged to have an end-of-phrase marker after the last line.

[codello]

I'm personally strongly in favor of allowing end-of-phrase markers anywhere within the file. I can understand the desire to restrict the format in a more consistent way but I think this is better achieved via a recommendation that repeated end-of-phrase markers SHOULD NOT be used. I'm thinking of two analogies that behave similarly:

• The end-of-phrase markers are kind of similar to line breaks in text. In a text file it is valid to have multiple line breaks one after another. Ignoring empty lines (or empty phrases in this case) is easy for implementations.
• The JSON spec allows duplicate keys in objects. It is noted that this can cause unpredictable behavior but it is technically allowed.

I also think that allowing repeated end-of-phrase markers can simplify (compliant) implementations. Implementations would have to check for empty phrases in both cases: Either to filter them out or to raise an error because the file is invalid. Simply filtering out empty phrases as opposed to rejecting a file seems the more desirable behavior to me.

[RumovZ]

Simply filtering out empty phrases as opposed to rejecting a file seems the more desirable behavior to me.

I don't see why karaoke programs would be forced to reject incompliant input. By definition, this should not be a concern of this spec. Using set notation:
<what editors MAY produce> ⊆ <the spec> ⊆ <what karaoke programs MUST accept

So in this case, repeated markers should be forbidden, such that editors may not produce ambiguous output. However, karaoke programs may still accept invalid input and fix it up, regardless of the spec.

The JSON spec allows duplicate keys in objects.

I'm not aware of any examplaes where UB is an intentional design choice, except to allow for more efficient implementations (like in C) or for legacy support (like in Javascript). In other words, unless there's a very specific reason for it, I would always consider UB a design flaw.