wai-aria: Update results (#133)

* wai-aria: Updated results for AX API

Reflects removal of redundant and irrelevant assertions which had been
present in two tests. (The tests have since been updated.)

* wai-aria: Combine aria-modal="true" subtests into a single test.

The various accessibility APIs handle exposure of aria-modal="true" quite
differently. For instance, in ATK, implementors must add STATE_MODAL to the
modal container. But in AX API, implementors must do nothing to the modal
container, and instead must completely remove all of the other elements from
the accessibility tree.

Differences such as these make it hard to test "implentability" across the
platforms using the current WPT ATTA framework which creates per-element
subtests. By preserving the existing results, but swapping the subtest names
for one of the platforms, we can generate results with wptreport which
accurately show the details of each PASS and FAIL as well as implementability
of the feature.

* wai-aria: Combine results of two subtests for ATK into one subtest

There are two tests which have subtests with assertions only for ATK. The
reason for this is that ATK expects an additional accessibility event as
part of the implementation of the ARIA features in question, and our test
harness creates separate subtests for each element being examined. We can
preserve the detailed results for ATK without otherwise impacting the
cross-platform implementation report by combining the ATK-unique subtest
results with the previous subtest's results.

* wai-aria: Filter out a test for something postponed until ARIA 1.2.

The ARIA spec does not have any normative statements regarding what user
agents should do if an author references a rendered element via the
aria-errormessage attribute with aria-invalid="false". The Working Group
concluded that if the ARIA spec needed to exit and re-enter CR, this
condition should be explicitly addressed in the spec. The ARIA spec did
not exit and re-enter CR, thus the associated test should not be part
of the implementation report for ARIA 1.1

* wai-aria: Remove "placeholder" statement from README.md

Author: joanmarie

wai-aria/FF01.json

@@ -1765,11 +1765,6 @@
           "name": "option selected value changes 1",
           "status": "PASS",
           "message": null
-        },
-        {
-          "name": "option selected value changes 2",
-          "status": "PASS",
-          "message": null
         }
       ],
       "status": "OK",
@@ -2839,11 +2834,6 @@
           "name": "treeitem selected value changes 1",
           "status": "PASS",
           "message": null
-        },
-        {
-          "name": "treeitem selected value changes 2",
-          "status": "PASS",
-          "message": null
         }
       ],
       "status": "OK",

wai-aria/FF02.json

@@ -21,12 +21,12 @@
       "test": "/wai-aria/alertdialog_modal_true-manual.html",
       "subtests": [
         {
-          "name": "alertdialog modal true",
+          "name": "alertdialog modal true 1",
           "status": "FAIL",
-          "message": "assert_true: \"property AXRole is AXGroup\" failed   \n**Actual value:** AXWindow  \n; \"property AXRoleDescription is web alert dialog\" failed   \n**Actual value:** alert dialog  \n;  expected true got false"
+          "message": "assert_false: NOTRUN: No assertion for API AXAPI expected false got true"
         },
         {
-          "name": "alertdialog modal true 1",
+          "name": "alertdialog modal true",
           "status": "FAIL",
           "message": "assert_true: \"property accessible is false\" failed   \n**Actual value:** true  \n;  expected true got false"
         }
@@ -915,12 +915,12 @@
       "test": "/wai-aria/dialog_modal_true-manual.html",
       "subtests": [
         {
-          "name": "dialog modal true",
+          "name": "dialog modal true 1",
           "status": "FAIL",
-          "message": "assert_true: \"property AXRole is AXGroup\" failed   \n**Actual value:** AXWindow  \n; \"property AXRoleDescription is web dialog\" failed   \n**Actual value:** window  \n;  expected true got false"
+          "message": "assert_false: NOTRUN: No assertion for API AXAPI expected false got true"
         },
         {
-          "name": "dialog modal true 1",
+          "name": "dialog modal true",
           "status": "FAIL",
           "message": "assert_true: \"property accessible is false\" failed   \n**Actual value:** true  \n;  expected true got false"
         }

wai-aria/GC02.json

@@ -21,12 +21,12 @@
       "test": "/wai-aria/alertdialog_modal_true-manual.html",
       "subtests": [
         {
-          "name": "alertdialog modal true",
+          "name": "alertdialog modal true 1",
           "status": "FAIL",
-          "message": "assert_true: \"property AXRoleDescription is web alert dialog\" failed   \n**Actual value:** alertdialog  \n;  expected true got false"
+          "message": "assert_false: NOTRUN: No assertion for API AXAPI expected false got true"
         },
         {
-          "name": "alertdialog modal true 1",
+          "name": "alertdialog modal true",
           "status": "FAIL",
           "message": "assert_true: \"property accessible is false\" failed   \n**Actual value:** true  \n;  expected true got false"
         }
@@ -915,12 +915,12 @@
       "test": "/wai-aria/dialog_modal_true-manual.html",
       "subtests": [
         {
-          "name": "dialog modal true",
+          "name": "dialog modal true 1",
           "status": "FAIL",
-          "message": "assert_true: \"property AXRoleDescription is web dialog\" failed   \n**Actual value:** dialog  \n;  expected true got false"
+          "message": "assert_false: NOTRUN: No assertion for API AXAPI expected false got true"
         },
         {
-          "name": "dialog modal true 1",
+          "name": "dialog modal true",
           "status": "FAIL",
           "message": "assert_true: \"property accessible is false\" failed   \n**Actual value:** true  \n;  expected true got false"
         }

wai-aria/README.md

@@ -1,9 +1,6 @@
 WAI ARIA 1.1 Test Results
 =========================
 
-Placeholder for wai-aria test results.  Once the results are consolidated, they will 
-available in:
-
 * <https://w3c.github.io/test-results/wai-aria/all.html>
 * <https://w3c.github.io/test-results/wai-aria/less-than-2.html>
 * <https://w3c.github.io/test-results/wai-aria/complete-fails.html>

wai-aria/WK01.json

@@ -1764,12 +1764,7 @@
         {
           "name": "option selected value changes 1",
           "status": "FAIL",
-          "message": "assert\\_true: \"event type is object:selection-changed\" failed <https://webkit.org/b/171186>  \n;  expected true got false"
-        },
-        {
-          "name": "option selected value changes 2",
-          "status": "FAIL",
-          "message": "assert\\_true: \"event type is object:state-changed:selected\" failed <https://webkit.org/b/171186>  \n;  expected true got false"
+          "message": "assert\\_true: element 'test1' event type is object:selection-changed\" failed\nelement 'test2' event type is object:state-changed:selected\" failed <https://webkit.org/b/171186>  \n;  expected true got false"
         }
       ],
       "status": "OK",
@@ -2838,12 +2833,7 @@
         {
           "name": "treeitem selected value changes 1",
           "status": "FAIL",
-          "message": "assert\\_true: \"event type is object:selection-changed\" failed <https://webkit.org/b/171185>  \n;  expected true got false"
-        },
-        {
-          "name": "treeitem selected value changes 2",
-          "status": "FAIL",
-          "message": "assert\\_true: \"event type is object:state-changed:selected\" failed <https://webkit.org/b/171185>  \n;  expected true got false"
+          "message": "assert\\_true: element 'test1' event type is object:selection-changed\" failed\nelement 'test2' event type is object:state-changed:selected\" failed <https://webkit.org/b/171185>  \n;  expected true got false"
         }
       ],
       "status": "OK",

wai-aria/WK02.json

@@ -21,12 +21,12 @@
       "test": "/wai-aria/alertdialog_modal_true-manual.html",
       "subtests": [
         {
-          "name": "alertdialog modal true",
-          "status": "PASS",
-          "message": null
+          "name": "alertdialog modal true 1",
+          "status": "FAIL",
+          "message": "assert_false: NOTRUN: No assertion for API AXAPI expected false got true"
         },
         {
-          "name": "alertdialog modal true 1",
+          "name": "alertdialog modal true",
           "status": "PASS",
           "message": null
         }
@@ -915,12 +915,12 @@
       "test": "/wai-aria/dialog_modal_true-manual.html",
       "subtests": [
         {
-          "name": "dialog modal true",
-          "status": "PASS",
-          "message": null
+          "name": "dialog modal true 1",
+          "status": "FAIL",
+          "message": "assert_false: NOTRUN: No assertion for API AXAPI expected false got true"
         },
         {
-          "name": "dialog modal true 1",
+          "name": "dialog modal true",
           "status": "PASS",
           "message": null
         }