Make CORS test detect empty responses better (#117)

* Set the origin in cors tests from config so that it uses the same one. * Cast response to string when checking HEAD requests in CORS. * Update test subjects
solid-contrib · Mar 21, 2024 · e56ccf1 · e56ccf1
1 parent a430fc9
commit e56ccf1
Show file tree

Hide file tree

Showing 9 changed files with 49 additions and 40 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,10 @@
 # Solid Specification Conformance Tests
 
+## Release 0.0.19
+* Set the origin in CORS tests from config so that it uses the same one.
+* Cast response to string when checking HEAD requests in CORS.
+* Add retries to authorization tests to allow for servers which propagate access policies asynchronously.
+
 ## Release 0.0.18
 * Update run.sh to support CSS 7.
 * Resolve location headers against base URL before using them in case they are relative.

diff --git a/protocol/cors/acao-vary.feature b/protocol/cors/acao-vary.feature
@@ -7,10 +7,11 @@ Feature: Server returns correct Access-Control-Allow-Origin and Vary headers
   Scenario Outline: Access-Control-Allow-Origin header is set to correct origin for <method> on container
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('<method>', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
+    * string response = response
     And match response <check>
     Examples:
       | method  | statuses   | check |
@@ -21,10 +22,11 @@ Feature: Server returns correct Access-Control-Allow-Origin and Vary headers
   Scenario Outline: Vary header includes Origin for <method> on container
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('<method>', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     When method <method>
     Then match <statuses> contains responseStatus
     And match header Vary contains 'Origin'
+    * string response = response
     And match response <check>
     Examples:
       | method  | statuses   | check |
@@ -35,10 +37,11 @@ Feature: Server returns correct Access-Control-Allow-Origin and Vary headers
   Scenario Outline: Access-Control-Allow-Origin header is set to correct origin for <method> on resource
     Given url resource.url
     And headers clients.alice.getAuthHeaders('<method>', resource.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
+    * string response = response
     And match response <check>
     Examples:
       | method  | statuses   | check      |
@@ -49,10 +52,11 @@ Feature: Server returns correct Access-Control-Allow-Origin and Vary headers
   Scenario Outline: Vary header includes Origin for <method> on resource
     Given url resource.url
     And headers clients.alice.getAuthHeaders('<method>', resource.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     When method <method>
     Then match <statuses> contains responseStatus
     And match header Vary contains 'Origin'
+    * string response = response
     And match response <check>
     Examples:
       | method  | statuses   | check      |

diff --git a/protocol/cors/accept-acah.feature b/protocol/cors/accept-acah.feature
@@ -8,7 +8,7 @@ Feature: Server should explicitly list Accept under Access-Control-Allow-Headers
   Scenario: OPTIONS request doesn't return Accept in Access-Control-Allow-Headers for GET pre-flight if not requested
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('OPTIONS', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'GET'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type'
     When method OPTIONS
@@ -19,7 +19,7 @@ Feature: Server should explicitly list Accept under Access-Control-Allow-Headers
   Scenario: OPTIONS request returns Accept in Access-Control-Allow-Headers for POST pre-flight
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('OPTIONS', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'POST'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type, Accept'
     When method OPTIONS
@@ -30,7 +30,7 @@ Feature: Server should explicitly list Accept under Access-Control-Allow-Headers
   Scenario: OPTIONS request returns Accept in Access-Control-Allow-Headers for GET pre-flight with long Accept
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('OPTIONS', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'GET'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type, Accept'
     When method OPTIONS

diff --git a/protocol/cors/access-control-headers.feature b/protocol/cors/access-control-headers.feature
@@ -9,12 +9,12 @@ Feature: Server must respond to requests sending Origin with the appropriate Acc
 
   Scenario Outline: Simple request: <method> request returns access control headers
     Given url testContainer.url
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And headers <headers>
     * <body>
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     Examples:
       | method | headers!                       | body            | statuses |
       | GET    | {Accept: 'text/turtle'}        | def ignore = 1  | [401]    |
@@ -24,12 +24,12 @@ Feature: Server must respond to requests sending Origin with the appropriate Acc
   Scenario Outline: Requests with credentials: <method> request returns access control headers
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('<method>', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And headers <headers>
     * <body>
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     Examples:
       | method | headers!                       | body            | statuses             |
       | GET    | {Accept: 'text/turtle'}        | def ignore = 1  | [200]                |

diff --git a/protocol/cors/enumerate-headers.feature b/protocol/cors/enumerate-headers.feature
@@ -7,7 +7,7 @@ Feature: Server should enumerate headers in Access-Control-Expose-Headers
   Scenario: Access-Control-Expose-Headers is present but not *
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('GET', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Accept = 'text/turtle'
     When method GET
     Then status 200

diff --git a/protocol/cors/preflight-requests.feature b/protocol/cors/preflight-requests.feature
@@ -6,12 +6,12 @@ Feature: Server must implement the CORS protocol for preflight requests
 
   Scenario Outline: Pre-flight CORS request for <method> request
     Given url testContainer.url
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = '<method>'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type, Accept'
     When method OPTIONS
     Then match [200, 204] contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Allow-Methods contains '<method>'
     And match header Access-Control-Allow-Headers contains 'X-CUSTOM'
     And match header Access-Control-Allow-Headers contains 'Content-Type'
@@ -20,14 +20,14 @@ Feature: Server must implement the CORS protocol for preflight requests
     And match response == ''
 
     Given url testContainer.url
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And headers clients.alice.getAuthHeaders('<method>', testContainer.url)
     # Demonstrates the case where a long Accept header is allowed
     And header Accept = 'text/turtle;q=0.9, application/rdf+xml;q=0.8, application/n-triples;q=0.8, application/n-quads;q=0.8, text/x-nquads;q=0.8, application/trig;q=0.8, text/n3;q=0.8, application/ld+json;q=0.8, application/x-binary-rdf;q=0.8, text/plain;q=0.7'
     * <body>
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Expose-Headers != null
     And match header Access-Control-Expose-Headers != '*'
     # Check Content-Type on GET request only
@@ -44,20 +44,20 @@ Feature: Server must implement the CORS protocol for preflight requests
   Scenario: OPTIONS request returns headers for pre-flight check after redirect from http
     * configure followRedirects = false
     Given url testContainer.url.replace(/^https:/, 'http:')
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'POST'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type'
     When method OPTIONS
     Then match [301, 308] contains responseStatus
     * def location = resolveUri(testContainer.url, karate.response.headerValues('location')[0])
 
     Given url location
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'POST'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type'
     When method OPTIONS
     Then match [200, 204] contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Allow-Methods contains 'POST'
     And match header Access-Control-Allow-Headers contains 'X-CUSTOM'
     And match header Access-Control-Allow-Headers contains 'Content-Type'

diff --git a/protocol/cors/preflight.feature b/protocol/cors/preflight.feature
@@ -5,12 +5,12 @@ Feature: Server must support HTTP OPTIONS for CORS preflight requests
 
   Scenario: OPTIONS request returns headers for pre-flight check
     Given url testContainer.url
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'POST'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type'
     When method OPTIONS
     Then match [200, 204] contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Allow-Methods contains 'POST'
     And match header Access-Control-Allow-Headers contains 'X-CUSTOM'
     And match header Access-Control-Allow-Headers contains 'Content-Type'
@@ -21,20 +21,20 @@ Feature: Server must support HTTP OPTIONS for CORS preflight requests
   @http-redirect
   Scenario: OPTIONS request returns headers for pre-flight check after redirect from http
     Given url testContainer.url.replace(/^https:/, 'http:')
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'POST'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type'
     When method OPTIONS
     Then match [301, 308] contains responseStatus
     * def location = resolveUri(testContainer.url, karate.response.headerValues('location')[0])
 
     Given url location
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And header Access-Control-Request-Method = 'POST'
     And header Access-Control-Request-Headers = 'X-CUSTOM, Content-Type'
     When method OPTIONS
     Then match [200, 204] contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Allow-Methods contains 'POST'
     And match header Access-Control-Allow-Headers contains 'X-CUSTOM'
     And match header Access-Control-Allow-Headers contains 'Content-Type'

diff --git a/protocol/cors/simple-requests.feature b/protocol/cors/simple-requests.feature
@@ -9,12 +9,12 @@ Feature: Server must implement the CORS protocol for simple requests
 
   Scenario Outline: Simple container request: <method> request returns access control headers
     Given url testContainer.url
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And headers <headers>
     * <body>
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Expose-Headers != null
     And match header Access-Control-Expose-Headers != '*'
     Examples:
@@ -25,12 +25,12 @@ Feature: Server must implement the CORS protocol for simple requests
 
   Scenario Outline: Simple resource request: <method> request returns access control headers
     Given url resource.url
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And headers <headers>
     * <body>
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Expose-Headers != null
     And match header Access-Control-Expose-Headers != '*'
     Examples:
@@ -41,12 +41,12 @@ Feature: Server must implement the CORS protocol for simple requests
   Scenario Outline: Requests container with credentials: <method> request returns access control headers
     Given url testContainer.url
     And headers clients.alice.getAuthHeaders('<method>', testContainer.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And headers <headers>
     * <body>
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Expose-Headers != null
     And match header Access-Control-Expose-Headers != '*'
     # Check Vary on GET/HEAD requests only
@@ -60,12 +60,12 @@ Feature: Server must implement the CORS protocol for simple requests
   Scenario Outline: Requests resource with credentials: <method> request returns access control headers
     Given url resource.url
     And headers clients.alice.getAuthHeaders('<method>', resource.url)
-    And header Origin = 'https://tester'
+    And header Origin = config.origin
     And headers <headers>
     * <body>
     When method <method>
     Then match <statuses> contains responseStatus
-    And match header Access-Control-Allow-Origin == 'https://tester'
+    And match header Access-Control-Allow-Origin == config.origin
     And match header Access-Control-Expose-Headers != null
     And match header Access-Control-Expose-Headers != '*'
     And match header Vary contains 'Origin'

diff --git a/test-subjects.ttl b/test-subjects.ttl
@@ -23,8 +23,8 @@
     rdfs:comment "Access for anonymous users is not supported nor required by Solid specifications so tests that depend on this are not run."@en.
 
 <ess#test-subject-release>
-    doap:revision "2.1";
-    doap:created "2023-03-28"^^xsd:date .
+    doap:revision "2.2";
+    doap:created "2024-03-19"^^xsd:date .
 
 <css>
     a earl:Software, earl:TestSubject ;
@@ -37,8 +37,8 @@
     solid-test:skip "acp", "http-redirect" .
 
 <css#test-subject-release>
-    doap:revision "7.0.3" ;
-    doap:created "2024-01-05"^^xsd:date .
+    doap:revision "7.0.4" ;
+    doap:created "2024-02-07"^^xsd:date .
 
 <nss>
     a earl:Software, earl:TestSubject ;
@@ -51,8 +51,8 @@
     solid-test:skip "acp" .
 
 <nss#test-subject-release>
-    doap:revision "5.7.7"@en ;
-    doap:created "2023-03-17"^^xsd:date .
+    doap:revision "5.7.9"@en ;
+    doap:created "2024-03-08"^^xsd:date .
 
 <trinpod>
     a earl:Software, earl:TestSubject ;