Updated tests and descriptions of duration handling (#620)

* Updated test versions to 8.3.0 and clarified conditions for TAG_INVALID

* Merged NODE_NAME_EMPTY into TAG_INVALID

* Corrected additional NODE_NAME_EMPTY_LINKS

* Lower the schema version on two tests back

* Made hed_error_codes.json only contain codes with json files

* Add hedID validation tests

* Updated some of invalid characters

* Updated the JSON tests to have all fields

* Updated tests to include warning and alt_code in non-schema tests

* Updated the tests with additional alt-codes

* Definition may have no inner group

* Minor update in appendix B errors

* Minor changes to tests

* Corrected a typo in the specification

* Correction to curly brace test

* Modified the test for definition with no group

---------

Co-authored-by: IanCa <[email protected]>

.codespellrc

@@ -1,3 +1,3 @@
-[codespell]
-skip = .git,*.pdf,*.svg,deprecated,*.xml,*.mediawiki,*.omn
-ignore-words-list = covert,hed
+[codespell]
+skip = .git,*.pdf,*.svg,deprecated,*.xml,*.mediawiki,*.omn
+ignore-words-list = covert,hed,recuse

.gitignore

@@ -84,7 +84,6 @@ instance/
 
 # Sphinx documentation
 docs/_build/
-docs/_build/
 
 # PyBuilder
 .pybuilder/

docs/source/03_HED_formats.md

@@ -587,9 +587,8 @@ HED tools are available to map between shortened and long forms as needed.
 The tag must be associated with a schema and must correspond to a path in the schema
 (excluding any extension or value).
 
-See [**NODE_NAME_EMPTY**](./Appendix_B.md#node_name_empty) for errors involving
-forward slashes (`/`) and [**TAG_INVALID**](./Appendix_B.md#tag_invalid) for
-other types of tag syntax errors.
+See [**TAG_INVALID**](./Appendix_B.md#tag_invalid) for errors involving
+forward slashes (`/`), extra white-space, and other types of tag syntax errors.
 
 ### 3.2.3. Tag case-sensitivity
 
@@ -797,9 +796,8 @@ more details on validation errors due to repeated tag expressions.
 #### 3.2.8.1. The `Definition` tag
 
 A HED definition is a tag group consist of a `Definition` tag that takes
-a value representing the definition's name and a tag group defining the concept.
+a value representing the definition's name and an optional tag group defining the concept.
 Each definition is independent and stands alone.
-The definition must contain a non-empty tag group.
 
 The `Definition` tag corresponds to a schema node with the `topLevelTagGroup` attribute,
 assuring that definitions cannot be nested. 
@@ -887,11 +885,32 @@ See [**TEMPORAL_TAG_ERROR**](./Appendix_B.md#temporal_tag_error) and
  [**TAG_GROUP_ERROR**](./Appendix_B.md#tag_group_error) and
 for a listing of specific errors associated with onsets, and offsets, and insets.
 
-[**Chapter 5.3.1 Using Onset and Offset**](./05_Advanced_annotation.md#531-using-onset-and-offset)
+[**Chapter 5.3.1. Using Onset and Offset**](./05_Advanced_annotation.md#531-using-onset-and-offset)
 in Chapter 5 gives examples of usage and additional details.
 
+#### 3.2.8.4. `Duration` and `Delay`
 
-#### 3.2.8.4. The `Event-context` tag
+The `Delay` tag MUST appear in a top-level tag group and MUST take a numerical value.
+Its tag group must contain an inner tag group representing an event whose onset time is delayed by
+the specified amount.
+If the top-level tag group contains a `Duration` tag, then this event has the indicated duration
+otherwise the event is considered a point event.
+
+The `Duration` tag MUST appear in a top-level tag group and MUST take a numerical value indicating
+the duration of the event represented by the other tags in its group.
+The `Duration` top-level tag group may contain a `Delay`, but no other tags with the `topLevelTagGroup attribute`
+(e.g., `Onset`, `Offset`, `Inset`, `Event-context`, `Definition`).
+
+Note that the starting or the ending time of the event group with `Delay` or `Duration`, respectively,
+may not correspond to an actual event-marker appearing in the events file.
+Instead, tools calculate when the scope at analysis time and may insert additional markers if required.
+As with all HED tags and groups, order does not matter.
+
+See [**Chapter 5.3.3. Using Duration**](./05_Advanced_annotation.md#533-using-duration) and
+[**Chapter 5.3.4. Using Delay**](./05_Advanced_annotation.md#534-using-delay) for more information.
+
+
+#### 3.2.8.5. The `Event-context` tag
 
 The `Event-context` tag corresponds to a schema node with both the `topLevelTagGroup` and `unique` attributes.
 This implies that there can be only one `Event-context` group in each assembled event-level HED string.
@@ -1013,6 +1032,9 @@ When a column name appears in curly braces within a HED annotation in
 a JSON sidecar, the corresponding HED annotation for that row is substituted
 for the curly braces and their contents when the HED annotation is assembled.
 
+If a column is used in curly braces in a HED annotation,
+then that column must have a HED annotation or a [**SIDECAR_INVALID**](./Appendix_B.md#sidecar_invalid) error is generated.
+
 
 ``````{admonition} Rules for curly braces notation in sidecars.
 :class: tip

docs/source/05_Advanced_annotation.md

@@ -22,10 +22,12 @@ The following summarizes the syntax of HED definitions.
 ``````{admonition} Syntax summary for HED definitions.
 
 **Short forms:** 
+ ~ *(Definition/xxx)*
  ~ *(Definition/xxx, (definition-content))*
  ~ *(Definition/xxx/#, (definition-content))*
  
 **Long forms:**  
+ ~ *(Property/Organizational-property/<strong>Definition/xxx</strong>)*
  ~ *(Property/Organizational-property/<strong>Definition/xxx</strong>, (definition-content))*
  ~ *(Property/Organizational-property/<strong>Definition/xxx/#</strong>, (definition-content))*
  
@@ -35,6 +37,8 @@ The following summarizes the syntax of HED definitions.
 containing the tags representing the definition’s contents.
 2. If the *xxx/#* form is used, then the *(definition-content)* MUST contain a single `#` 
 representing a value to be substituted for when the definition is used.
+3. A definition without a *(definition-content)* is sometimes use as a general purpose definition for
+anchoring `Onset` groups with varying content.  
 
 ````
 

docs/source/Appendix_A.md

@@ -188,13 +188,13 @@ behavior of certain value classes (for example the `numericClass` value class).
 * - dateTimeClass
   - `digits`,  `colon`,  `hyphen`, `period`, `uppercase`
 * - nameClass
-  - `alphanumeric`, `blank`, `hyphen`, `period`, `underscore`, `nonascii` 
+  - `alphanumeric`, `hyphen`, `underscore`, `nonascii` 
 * - numericClass
-  - `digits`,  `period`,  `hyphen`,  `plus`, 'caret`,  `E`,  `e` `
+  - `digits`,  `period`,  `hyphen`,  `plus`, `caret`,  `E`,  `e` `
 * - posixPath
   -  As yet unspecified.
 * - textClass
-  - `printable` or `nonascii` excluding curly braces.
+  - `printable` or `nonascii` excluding curly braces, commas, and single quotes.
 * - IRIClass
   - Valid International Resource Identifier as standardized by [rfc3987](https://datatracker.ietf.org/doc/html/rfc3987).
 ``````
@@ -215,6 +215,7 @@ A BIDS regular expression for this is:
 5. Values of `numericClass` must be equivalent to a valid floating point value.
 6. Scientific notation is supported with the `numericClass`.
 7. The `textClass` is for descriptions, mainly for use with the `Description` tag or schema element descriptions.
+It is also allowed as the value for other tags such as
 8. The `posixPath` class is as yet unspecified and currently allows any characters except commas.
 9. The IRIClass validity is determined by a library implementing the IETF rfc3987 standard.
 

docs/source/Appendix_B.md

@@ -25,12 +25,12 @@ of errors keyed to the HED specification.
 
 A HED string contains an invalid character.
 
-**a.** A tag extension contains a non `name` character.  
+**a.** A non-printable character (ASCII code < 32 or == 127) appears in a HED string.  
 **b.** Curly braces appear in a HED string not in a sidecar.
 
 
 **Notes:**  
-- HED uses ANSI encoding and does not support UTF-8.  
+- Starting with HED 8.3.0, HED supports UTF-8 encoding.  
 - Different parts of a HED string have different rules for acceptable characters.
 
 See 
@@ -88,7 +88,7 @@ A **definition** is a tag group containing a `Definition` tag and a single tag g
 the definition's contents.  
 
 **a.**  A `Definition` tag does not appear in a tag group at the top level in an annotation.   
-**b.**  A definition's enclosing tag group is missing the inner tag group (.i.e., the definition's contents).    
+**b.**  A definition's enclosing tag group contains an empty group (i.e., an actual inner `()`).    
 **c.**  A definition's enclosing tag group contains more than a `Definition` tag and an inner group.    
 **d.**  A definition's inner tag group contains `Definition`, `Def` or `Def-expand` tags.  
 **e.** A definition uses curly braces.  
@@ -116,13 +116,6 @@ its description is updated to include the reason for deprecation and a suggested
 See [**A.1.4. Schema attributes**](./Appendix_A.md#a14-schema-attributes) for additional information
 about the `deprecatedFrom` schema attribute.
 
-### NODE_NAME_EMPTY
-
-**a.**  A tag has one or more forward slashes (`/`) at beginning or end (ignoring whitespace).  
-**b.**  A tag contains consecutive forward slashes (ignoring whitespace).  
-
-See [**3.2.3 Tag forms**](./03_HED_formats.md#322-tag-forms) for more information.
-
 ### PARENTHESES_MISMATCH
 
 **a.**  A HED string does not have the same number of open and closed parentheses.  
@@ -176,6 +169,7 @@ on the requirements for using sidecars.
 
 **a.**  The `"HED"` key is not a second-level dictionary key.  
 **b.**  An annotation entry is provided for `n/a`.  
+**c.**  A sidecar refers to a column in curly braces, but that column has no HED.  
 
 See [**3.2.9. Sidecars**](./03_HED_formats.md#329-sidecars) for a
 general explanation of sidecar requirements.
@@ -254,9 +248,15 @@ for additional information on the tag extension rules.
 See [**3.2.7.2. Tag group attributes**](./03_HED_formats.md#3272-tag-group-attributes)
 for additional information on the rules for group errors due to schema attributes.
 
+**See also:** [**TEMPORAL_TAG_ERROR**](#temporal_tag_error) for errors involving `Onset`, `Offset`, `Inset`, 
+`Duration` and `Delay`.
+
 ### TAG_INVALID
 
 **a.**  The tag is not valid in the schema it is associated with.  
+**b.**  The tag has extra internal whitespace, including directly before or after forward slashes.  
+**c.**  The tag has a leading, trailing, or consecutive forward slashes.  
+**d.**  A tag or tag extension contains a non `name` character.  
 
 See [**3.2.2. Tag forms**](./03_HED_formats.md#322-tag-forms) for a discussion
 of tag forms and their relationship to the HED schema.
@@ -289,8 +289,8 @@ for an explanation of the `requireChild` attribute.
 Note: For the purpose of `Onset`/`Offset` matching, `Def` or `Def-expand` tags with
 different placeholder substitutions are considered to be different.
 
-**a.**  An `Offset`, `Onset`, `Inset`, `Duration`, or `Delay` tag does not appear in a tag group.  
-**b.**  An `Offset`, `Onset`, `Inset`, `Duration`, or `Delay` tag appears in a nested tag group (not a top-level tag group).   
+**a.**  An `Offset`, `Onset`, `Inset`, `Duration`, or `Delay` tag does not appear in a tag group.   
+**b.**  An `Offset`, `Onset`, `Inset`, `Duration`, or `Delay` tag appears in a nested tag group (not a top-level tag group).    
 **c.**  An `Onset`, `Offset` or `Inset` tag is not grouped with exactly one `Def` tag or `Def-expand` group.   
 **d.** An `Onset`, `Inset`, `Duration`, or `Delay` tag group contains more than one additional tag group.   
 **e.** An `Offset` appears with one or more tags or additional tag groups.   
@@ -303,10 +303,12 @@ that are not in a tag group.
 appears in an event marker with the same time as with another `Onset`, `Inset`, or `Offset`
 that uses the same anchor.  
 **j.** An `Inset` tag is not grouped with a `Def` tag or a `Def-expand` group corresponding to an ongoing `Onset`.  
-**k.** An `Onset`, `Inset`, or `Offset` tag appears in an annotation for a non-time tabular file.
-**l.** A `Duration` or `Delay` tag group contains extra tags or groups, or is missing the required group.
+**k.** An `Onset`, `Inset`, or `Offset` tag appears in an annotation for a non-time tabular file.  
+**l.** A `Duration` or `Delay` tag group contains extra tags or groups, or is missing the required group.  
 **m.** An `Offset`, `Onset`, `Inset`, `Duration`, or `Delay` tag appears with other top level tags, except
-Delay and Duration which can be paired.
+`Delay` and `Duration` which can be paired.  
+
+**See also:** [**TAG_GROUP_ERROR**](#tag_group_error).  
 
 **Note:** if the `Onset` tag group's definition is in expanded form, 
 the `Def-expand` will be an additional internal tag group.