Turns out there's an option to get in-progress responses within the API endpoint we're already using. Appears to be an either/or situation, though--you'd have to get completed and in-progress cases with different runs. )Much like how the web interface shows that on two pages; guessing it's something to do with how the Qualtrics engine works.)

https://api.qualtrics.com/api-reference/reference/responseImportsExports.json/paths/~1surveys~1%7BsurveyId%7D~1export-responses/post (see exportResponsesInProgress)

I've got some other work to add soon, and it would be trivial enough to just add this as a param to fetch_survey(). Happy to do that much at least.

@juliasilge, thoughts on UI goals though? It might be more complicated than just adding a param, since we'd need to add some careful arg checking (e.g., the new param would almost certainly not play nice w/startDate & endDate). We could also do a separate, simplified function, or something else. No strong views myself but I thought I'd solicit thoughts before revisiting later.

What do you think about a separate function? I'm leaning that way, and calling it a "helper function". I'm looking at the args to fetch_survey() and it seems like it would be confusing to users to have those available when they can't be used.

We can use @inheritParams to bring in the args from fetch_survey().

That would be fine, maybe something like fetch_inprogress()?

We'd end up import all the args other than start_date & end_date, yes? Seems like all the others should apply, though it's something that would warrant testing

Both in progress and completed responses have start and end dates on the data end. For reference what I am using right now is below. Essentially re-stacking the in progress and completed responses to set up some reporting. A colleague has an implementation on their package https://github.com/ppssphysics/QualtricsInR .

EOS_add_1 <- QualtricsInR::get_survey_responses(surveys[ i,"id"], exportResponsesInProgress = TRUE,
breakoutSets=FALSE, useLabels = TRUE)
EOS_add_2 <- QualtricsInR::get_survey_responses(surveys[ i,"id"], exportResponsesInProgress = FALSE,
breakoutSets=FALSE, useLabels = TRUE)
EOS <- rbind(EOS_add_1, EOS_add_2)

Ah, looks like you are right:

library(tidyverse)

req_body <- list(useLabels = TRUE,
                 includeDisplayOrder = TRUE,
                 exportResponsesInProgress = TRUE,
                 startDate = "2020-02-01T00:00:00Z",
                 format = "csv") %>%
  map(jsonlite::unbox) %>%
  jsonlite::toJSON()

req_body
#> {"useLabels":true,"includeDisplayOrder":true,"exportResponsesInProgress":true,"startDate":"2020-02-01T00:00:00Z","format":"csv"}

fetch_url <- qualtRics:::create_fetch_url(Sys.getenv("QUALTRICS_BASE_URL"), 
                                          "SV_3gbwq8aJgqPwQDP")
rs <- qualtRics:::qualtrics_api_request("POST", url = fetch_url, body = req_body)
rs$meta
#> $requestId
#> [1] "2ef5f4a5-e845-4762-8e83-b81c1f0f7938"
#> 
#> $httpStatus
#> [1] "200 - OK"

^{Created on 2021-06-10 by the reprex package (v2.0.0)}

So maybe we just yet another 😩 argument to fetch_survey(): in_progress maybe?

Hold on--I thought those two weren't equivalent? (referencing #158.)

I agree you'd expect to see the usual StartDate and EndDate variables in the response download (or at least StartDate? Not sure what would be in EndDate).

BUT per the API docs (and confusingly), arguments startDate and endDate set boundaries for when the survey was RECORDED, not started or ended from the user's perspective. It doesn't seem like that would be meaningful when all the responses you're getting haven't been recorded yet.

@juliasilge I think you'd have to follow through with that request to see if adding startDate actually did anything.

Maybe somewhat more relevant to #158 than here, but while I'm at it--is there an existing tool that can render date-times with the "T" separator, and/or add the timezone adjustments to the end? That would actually make finishing several needed features a lot easier.

Ah, right, I am remembering these issues now with the date time arguments.

I know that in another issue I said it was good to create functions that mirrored the API endpoints, but it's possible that adding in_progress to fetch_survey() will be confusing to users. Will folks think that they will get both in progress and completed responses with in_progress = TRUE? Hopefully we can document it clearly enough. I do see the appeal of a separate function for that reason. Thoughts?

I don't know of a tool in R that creates date time strings that look like what is needed here exactly. It seems pretty non-standard to me. I would probably use glue to create them:

glue::glue("{Sys.Date()}T{format(Sys.time(), '%H:%M:%S')}Z")
#> 2021-06-10T14:56:25Z

^{Created on 2021-06-10 by the reprex package (v2.0.0)}

Huh. I just looked and apparently it's ISO standard 8601: https://www.w3.org/TR/NOTE-datetime.

And lubridate has something for that: https://lubridate.tidyverse.org/reference/format_ISO8601.html

I think the fact that API results are mutually exclusive depending on that switch is a solid case for a separate function.

One counterpoint, though: we could include the in_progress option, but not as a boolean. Instead, it's NULL, "only", and "include", with the latter making two separate calls? The param syntax provides clarity through a boost to functionality.

...come to think of it, though, separate function or not all of this might not naturally play nice with the caching stuff behind force_request. Something to address.

That's an interesting idea of NULL / "only" / "include". 🤔 I am currently thinking the best option is a separate function, though.