CRAN Reported bug caused by changes to Santa Monica dataset examples

[geneorama]

From: Prof Brian Ripley <[email protected]>
Sent: Friday, October 28, 2022 2:16:47 AM (UTC-06:00) Central Time (US & Canada)
To: Developers
Cc: [email protected]
Subject: CRAN package RSocrata

[Warning: External email]

Dear maintainer,

Please see the problems shown on https://cran.r-project.org/web/checks/check_results_RSocrata.html.

Please correct before 2022-11-11 to safely retain your package on CRAN.

NB: 'internal' .Rd files are now checked.

The CRAN Team

The debian version of the error:

Version: 1.7.11-2
Check: tests
Result: ERROR

     Running 'testthat.R' [10s/184s]
    Running the tests in 'tests/testthat.R' failed.
    Complete output:
     > library(testthat)
     > library(RSocrata)
     >
     > test_check("RSocrata")
     2022-11-07 20:06:31.342 getResponse: Error in httr GET: 403 https://data.cityofchicago.org/resource/j8vp-2qpg.json?$order=:id
     2022-11-07 20:08:55.667 getResponse: Error in httr GET: 400 https://soda.demo.socrata.com/resource/4334-bgaj.csv?%24app_token=ew2rEMuESuzWPqMkyPfOSGJgE&$order=:id
     2022-11-07 20:08:57.485 getResponse: Error in httr GET: 400 https://soda.demo.socrata.com/resource/4334-bgaj.csv?%24app_token=ew2rEMuESuzWPqMkyPfOSGJgE&$order=:id
     [ FAIL 2 | WARN 4 | SKIP 4 | PASS 134 ]
    
     == Skipped tests ===============================================================
     * See Issue #174 (4)
    
     == Failed tests ================================================================
     -- Error ('test-all.R:284'): read Socrata JSON with missing fields (issue 19 - binding pages together) --
     Error in `while (nrow(page) > 0 & !limitProvided) {
     query <- paste(validUrl, if (is.null(parsedUrl$query)) {
     "?"
     }
     else {
     "&"
     }, "$limit=50000&$offset=", nrow(result), sep = "")
     response <- getResponse(query, email, password)
     page <- getContentAsDataFrame(response)
     result <- rbind.fill(result, page)
     }`: argument is of length zero
     Backtrace:
     x
     1. \-RSocrata::read.socrata(...) at test-all.R:284:2
     -- Error ('test-all.R:560'): getContentAsDataFrame does not get caught in infinite loop --
     Error in `while (nrow(page) > 0 & !limitProvided) {
     query <- paste(validUrl, if (is.null(parsedUrl$query)) {
     "?"
     }
     else {
     "&"
     }, "$limit=50000&$offset=", nrow(result), sep = "")
     response <- getResponse(query, email, password)
     page <- getContentAsDataFrame(response)
     result <- rbind.fill(result, page)
     }`: argument is of length zero
     Backtrace:
     x
     1. \-RSocrata::read.socrata(u) at test-all.R:560:2
    
     [ FAIL 2 | WARN 4 | SKIP 4 | PASS 134 ]
     Error: Test failures
     Execution halted

[geneorama]

FYI @evejennings @nicklucius I created the issue for this last event, however I did not assign it.

[geneorama]

@nicklucius I assigned this to myself per your email.

The problem appears to be that Santa Monica took down https://data.smgov.net/resource/ia9m-wspt.json, which was an example we used in the test for #19.

From my comments, it looks like we need a json that has missing values for the first 1000 rows (possibly 50,000 now that the paging has increased).

I don't remember how I found the previous example, I propose removing this test.

[geneorama]

@nicklucius I'm still working though the tests and I found two more tests that are not passing.

[geneorama]

@nicklucius

I ran though all the tests and there were two warnings which I mistook for errors: API Conflict and readAPIConflictHumanReadable https://github.com/Chicago/RSocrata/blob/master/tests/testthat/test-all.R#L406

There are two errors both related to Santa Monica examples, read Socrata JSON with missing fields (issue 19 - binding pages together) and getContentAsDataFrame does not get caught in infinite loop

• https://github.com/Chicago/RSocrata/blob/master/tests/testthat/test-all.R#L282 (edit, line 282 not 274!)
• https://github.com/Chicago/RSocrata/blob/master/tests/testthat/test-all.R#L544

I don't remember the second test or know what issue it addresses, and given the complexity of the query I can't imagine how I would invent something similar to perform the same test.

In the short term, I propose that we skip both and reference this issue. I'll open a pull request unless you suggest otherwise.

In the longer term I still support refactoring the code and tests to make the test coverage more clear and organized.