Docs overhaul #431

dsweber2 · 2025-01-23T20:59:14Z

Checklist

Please:

Make sure this PR is against "dev", not "main".
Request a review from one of the current epipredict main reviewers:
dajmcdon.
Make sure to bump the version number in DESCRIPTION and NEWS.md.
Always increment the patch version number (the third number), unless you are
making a release PR from dev to main, in which case increment the minor
version number (the second number).
Describe changes made in NEWS.md, making sure breaking changes
(backwards-incompatible changes to the documented interface) are noted.
Collect the changes under the next release number (e.g. if you are on
0.7.2, then write your changes under the 0.8 heading).
Consider pinning the epiprocess version in the DESCRIPTION file if
- You anticipate breaking changes in epiprocess soon
- You want to co-develop features in epipredict and epiprocess

Change explanations for reviewer

Draft ready for review:

Magic GitHub syntax to mark associated Issue(s) as resolved when this is merged into the default branch

Resolves https://github.com/cmu-delphi/forecasting-team-meta/issues/26
Resolves More clearly document symmetrize for residuals #264
Resolves Switch vignettes to using standalone data #261
Resolves Vignette illustrating FluSight Forecaster #256
Resolves Clarify documentation for and sanity-check nafill_buffer usage #320
Resolves Vignettes #232

dajmcdon · 2025-01-23T23:26:47Z

/preview-docs

github-actions · 2025-01-23T23:37:31Z

🚀 Deployed on https://67e478bce7f35616d80fb137--epipredict.netlify.app

dshemetov · 2025-01-23T23:45:49Z

Our setup is generating docs in dev/, so the link is off, this works:
https://6792d4953137ef0ce0547a4f--epipredict.netlify.app/dev/

Also FYI: the bot edits its own comment for links. Each preview is a separate link and the links stick around for like 90 days. You can see the previous links in the comment edit history.

Edit: this has been fixed on main so is no longer necessary

dsweber2 · 2025-01-31T23:50:39Z

/preview-docs

dsweber2 · 2025-01-31T23:51:30Z

So something weird is happening with the plot for flatline_forecaster, not really sure why. Going to dig into that next.

I added an option to replace the data for the autoplot so you can compare with new data instead

dsweber2 · 2025-02-03T18:03:29Z

Draft of the getting started is ready, moving on to a draft of the "guts" page (name a placeholder), which is an overview of creating workflows by hand

dsweber2 · 2025-02-05T23:34:15Z

So something weird is happening with the plot for flatline_forecaster, not really sure why. Going to dig into that next.

After some digging, I don't think there are any bugs, just some edge-case behavior that we may not want:

Thresholding and extrapolation don't interact well. In this case, the quantiles it fits are 0.05 and 0.95, and it correctly rounds the 5% quantile up to zero (b/c of the negative values it is actually negative w/out constraint). But, it also looks to plot the 2.5% and 97.5% quantiles, which extrapolate doesn't know should also be zero. This results in quantiles with negative values.
The other thing is that the interpolated quantiles change quite a bit after thresholding if there's not very many quantiles. For example, the median gets pushed up quite a bit, but the point prediction doesn't reflect that.

My take away: never fit just the 5% and 95% quantiles. At least do the 50%. That fixes most of the jank this uncovers.

dsweber2 · 2025-02-07T20:01:35Z

/preview-docs

dshemetov · 2025-02-07T20:14:31Z

Including 0.5 into the user's selection sounds simple and reasonable to me. They can always filter out what they don't want.

dsweber2 · 2025-02-07T21:50:01Z

/preview-docs

dsweber2 · 2025-02-25T20:24:22Z

/preview-docs

@nmdefries this also updates the backtesting vignette; I'm dropping the Canadian example because it basically had no revisions.

dsweber2 · 2025-02-25T23:08:11Z

/preview-docs

nmdefries · 2025-02-28T21:27:16Z

README.md


 ``` r
-two_week_ahead <- arx_forecaster(
-  covid_case_death_rates,
+four_week_ahead <- arx_forecaster(


issue: This gives me an error

Error in prep.epi_recipe(blueprint$recipe, training = training, fresh = blueprint$fresh, : object 'validate_training_data' not found

I'm using the dev version of epipredict, installed today.

That's internal to {recipes} (unexported). The DESCRIPTION may need to depend on a higher version. News.md here suggests 1.1.0, but worth looking.

(Long term goal is to remove any dependence on internal functions from other packages)

recipes 1.1.1 worked along with an update to hardhat

README.Rmd

nmdefries

Comments/questions so far.

I just need to finish the custom_workflows vignette.

nmdefries · 2025-04-03T19:01:39Z

vignettes/backtesting.Rmd

+As truth data, we'll compare with the `epix_as_of()` to generate a snapshot of
+the archive at the last date[^1].


question: why are we comparing our forecast to the finalized value?

nmdefries · 2025-04-03T21:04:27Z

vignettes/backtesting.Rmd

+    data = percent_cli_data |> filter(geo_value == geo_choose),
    aes(x = time_value, y = percent_cli, color = factor(version)),
    inherit.aes = FALSE, na.rm = TRUE


issue: On the version-faithful plot, the "finalized" line is bolded and pink vs gray on the version un-faithful plot. This makes them confusing to compare.

The version faithful finalized data also doesn't cover the full time period.

nmdefries · 2025-04-03T22:40:06Z

vignettes/custom_epiworkflows.Rmd

+library(epidatr)
+```
+
+To get a better handle on custom `epi_workflow()`s, lets recreate and then


suggestion: please explain what an epi_workflow is and why you'd want to use it.

nmdefries · 2025-04-03T22:42:43Z

vignettes/backtesting.Rmd

+ versions you should assume performance is worse than what the test would
+ otherwise suggest.
+
+[^4]: Until we have a time machine


praise: 😆

nmdefries · 2025-04-03T22:46:12Z

vignettes/backtesting.Rmd

+The version faithful and un-faithful forecasts look moderately similar except for the 1 day horizons  
+(although neither approach produces amazingly accurate forecasts).

-### Example using case data from Canada
+In the version faithful case for California, the March 2021 forecast (turquoise)
+starts at a value just above 10, which is very well lined up with reported values leading up to that forecast.
+The measured and forecasted trends are also concordant (both increasingly moderately fast).

-<details>
+Because the data for this time period was later adjusted down with a decreasing trend, the March 2021 forecast looks quite bad compared to finalized data.

-<summary>Data and forecasts. Similar to the above.</summary>
-
-By leveraging the flexibility of `epiprocess`, we can apply the same techniques
-to data from other sources. Since some collaborators are in British Columbia,
-Canada, we'll do essentially the same thing for Canada as we did above.
-
-The [COVID-19 Canada Open Data Working Group](https://opencovid.ca/) collects
-daily time series data on COVID-19 cases, deaths, recoveries, testing and
-vaccinations at the health region and province levels. Data are collected from
-publicly available sources such as government datasets and news releases.
-Unfortunately, there is no simple versioned source, so we have created our own
-from the Github commit history.
-
-First, we load versioned case rates at the provincial level. After converting
-these to 7-day averages (due to highly variable provincial reporting
-mismatches), we then convert the data to an `epi_archive` object, and extract
-the latest version from it. Finally, we run the same forcasting exercise as for
-the American data, but here we compare the forecasts produced from using simple
-linear regression with those from using boosted regression trees.
-
-```{r get-can-fc, warning = FALSE}
-aheads <- c(7, 14, 21, 28)
-canada_archive <- can_prov_cases
-canada_archive_faux <- epix_as_of(canada_archive, canada_archive$versions_end) %>%
-  mutate(version = time_value) %>%
-  as_epi_archive()
-# This function will add the 7-day average of the case rate to the data
-# before forecasting.
-smooth_cases <- function(epi_df) {
-  epi_df %>%
-    group_by(geo_value) %>%
-    epi_slide_mean("case_rate", .window_size = 7, na.rm = TRUE, .suffix = "_{.n}dav")
-}
-forecast_dates <- seq.Date(
-  from = min(canada_archive$DT$version),
-  to = max(canada_archive$DT$version),
-  by = "1 month"
-)
+The equivalent version un-faithful forecast starts at a value of 5, which is in line with the finalized data but would have been out of place compared to the version data.


comment: I heavily modified this section to focus more on why/how version faithful and un-faithful forecasts differ, and "advertise" backtesting as a useful tool. The previous blurb made it sound like version un-faithful forecasts performed better, which they do on finalized data of course, but is not what we're trying to say here.

Please check over the new version @dsweber2 to see if i left anything out.

nmdefries · 2025-04-04T22:16:19Z

vignettes/custom_epiworkflows.Rmd

+```
+
+So there are 6 steps we will need to recreate.
+One thing to note about the extracted recipe is that it has already been


suggestion: confused about recipe vs workflow (this is probably not the right spot to explain). We should probably link to recipes documentation of recipes so that we don't have to get into a lot of detail here.

nmdefries · 2025-04-04T22:38:05Z

vignettes/custom_epiworkflows.Rmd

+engines (such as `quantile_reg()`) are
+
+- `layer_quantile_distn()`: adds the specified quantiles.
+  If they differ from the ones actually fit, they will be interpolated and/or


suggestion: please clarify: differ in what way?

nmdefries · 2025-04-04T22:58:21Z

vignettes/custom_epiworkflows.Rmd

+
+## Predicting
+
+To do a prediction, we need to first narrow the dataset down to the relevant


suggestion: clarify why they need to be removed. Won't unused obs just be ignored?

nmdefries · 2025-04-04T23:04:00Z

vignettes/custom_epiworkflows.Rmd

+```
+
+The resulting tibble is 800 rows long, however.
+This produces forecasts for not just the actual `forecast_date`, but for every


question: where did we set the forecast_date/how does the workflow know what it is? what if we want to use a different forecast date? do we have to re-define and re-compile the whole workflow?

nmdefries · 2025-04-04T23:05:32Z

vignettes/custom_epiworkflows.Rmd

+This can be useful for cases where `get_test_data()` doesn't pull sufficient
+data.


issue: This sounds suspicious. If get_test_data decided there wasn't sufficient data, it sounds like the predict -> filter approach is doing something wrong (predicting with insufficient data).

I assumed that the predict -> filter approach would return the same predictions for forecast_date, but it sounds like not.

dsweber2 requested a review from dajmcdon as a code owner January 23, 2025 20:59

dsweber2 force-pushed the docsDraft branch from e935edb to fbf578c Compare January 23, 2025 23:36

dsweber2 force-pushed the docsDraft branch 4 times, most recently from 8044b98 to d35363e Compare January 27, 2025 22:50

This was referenced Feb 7, 2025

extrapolate_quantiles and thresholding #434

Closed

Canned median quantile #435

Merged

dsweber2 self-assigned this Feb 7, 2025

dsweber2 force-pushed the docsDraft branch from e58a1ca to d1f3d7d Compare February 7, 2025 21:48

dsweber2 force-pushed the docsDraft branch from d1f3d7d to 13d0655 Compare February 10, 2025 17:40

dsweber2 mentioned this pull request Feb 11, 2025

Hotfix growth rate #437

Merged

5 tasks

dsweber2 force-pushed the docsDraft branch from 359f324 to 14dfcd8 Compare February 14, 2025 23:47

dajmcdon mentioned this pull request Feb 26, 2025

440 epiprocess imports #444

Merged

3 tasks

dsweber2 mentioned this pull request Feb 28, 2025

moving smooth-qr over, updating -> R 4.4.1 cmu-delphi/delphi-tooling-book#18

Draft

nmdefries reviewed Feb 28, 2025

View reviewed changes

nmdefries reviewed Mar 1, 2025

View reviewed changes

README.Rmd Show resolved Hide resolved

dsweber2 force-pushed the docsDraft branch from 0578ace to ce1adfe Compare March 3, 2025 18:50

dsweber2 and others added 25 commits April 1, 2025 11:39

passing check & news

e8854a8

revising custom_epiworkflows

ae332f8

some more editing

682f2db

finished custom_workflows, reviewing backtesting

6622308

backtesting rmd rewrite

07b47f9

dropping CAN backtesting example b/c ~no revisions

17bfd1b

formatting

f49720f

|> in backtesting, dropped a section in get started

c6e55de

landing page wording and get code running

3cd58ac

landing page again but in Rmd

34c42a3

consistent naming, 7dav pull instead of manually

2f14ff8

going back to just using the API call

d91587c

recipes version, include epiprocess in the rmds

015d585

rebuild landing page

1a984e7

first half epipredict.Rmd

7312a38

custom header, dropping arx_classifier smooth-qr

48154eb

follow up on first half of epipredict.Rmd

f075b5d

avoid [link] parsing

7d7bd28

reorganize reference page

fd27a61

postprocessing -> post-processing

8dca98b

lots of reference updates

792a3be

forecast needs ... as a generic

15f83d9

include climate, only calculate necessary days

e0bcba8

Adding short blurb on cdc_flatline

c5a37a9

extra details for symmetrize

d6edae8

dsweber2 force-pushed the docsDraft branch from 3432fd2 to d6edae8 Compare April 1, 2025 16:40

nmdefries added 3 commits April 3, 2025 18:33

epipredict.Rmd

b786750

backtesting.rmd

af932cf

first half custom_epiworkflows.Rmd

d83d9d8

nmdefries reviewed Apr 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs overhaul #431

Docs overhaul #431

dsweber2 commented Jan 23, 2025 •

edited

Loading

dajmcdon commented Jan 23, 2025

github-actions bot commented Jan 23, 2025 •

edited

Loading

dshemetov commented Jan 23, 2025 •

edited by dsweber2

Loading

dsweber2 commented Jan 31, 2025

dsweber2 commented Jan 31, 2025

dsweber2 commented Feb 3, 2025

dsweber2 commented Feb 5, 2025 •

edited

Loading

dsweber2 commented Feb 7, 2025

dshemetov commented Feb 7, 2025

dsweber2 commented Feb 7, 2025

dsweber2 commented Feb 25, 2025

dsweber2 commented Feb 25, 2025

nmdefries Feb 28, 2025

dajmcdon Feb 28, 2025

dajmcdon Feb 28, 2025

nmdefries Mar 3, 2025

nmdefries left a comment

nmdefries Apr 3, 2025

nmdefries Apr 3, 2025

nmdefries Apr 3, 2025

nmdefries Apr 3, 2025

nmdefries Apr 3, 2025

nmdefries Apr 4, 2025

nmdefries Apr 4, 2025

nmdefries Apr 4, 2025

nmdefries Apr 4, 2025

nmdefries Apr 4, 2025

nmdefries Apr 4, 2025

		As truth data, we'll compare with the `epix_as_of()` to generate a snapshot of
		the archive at the last date[^1].


		## Predicting

		To do a prediction, we need to first narrow the dataset down to the relevant

		This can be useful for cases where `get_test_data()` doesn't pull sufficient
		data.

Docs overhaul #431

Are you sure you want to change the base?

Docs overhaul #431

Conversation

dsweber2 commented Jan 23, 2025 • edited Loading

Checklist

Change explanations for reviewer

Magic GitHub syntax to mark associated Issue(s) as resolved when this is merged into the default branch

dajmcdon commented Jan 23, 2025

github-actions bot commented Jan 23, 2025 • edited Loading

dshemetov commented Jan 23, 2025 • edited by dsweber2 Loading

dsweber2 commented Jan 31, 2025

dsweber2 commented Jan 31, 2025

dsweber2 commented Feb 3, 2025

dsweber2 commented Feb 5, 2025 • edited Loading

dsweber2 commented Feb 7, 2025

dshemetov commented Feb 7, 2025

dsweber2 commented Feb 7, 2025

dsweber2 commented Feb 25, 2025

dsweber2 commented Feb 25, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmdefries left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsweber2 commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

dshemetov commented Jan 23, 2025 •

edited by dsweber2

Loading

dsweber2 commented Feb 5, 2025 •

edited

Loading