Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add brms examples for non-linear model #425

Merged
merged 3 commits into from
Feb 18, 2025
Merged

Add brms examples for non-linear model #425

merged 3 commits into from
Feb 18, 2025

Conversation

gowerc
Copy link
Collaborator

@gowerc gowerc commented Jan 30, 2025

Closes #417

Also moved a bunch of older stuff that I don't think is relevant anymore to a "legacy" folder to clean up the design area a bit

Copy link
Contributor

github-actions bot commented Jan 30, 2025

Unit Tests Summary

    1 files    179 suites   6m 10s ⏱️
  157 tests   145 ✅ 12 💤 0 ❌
1 098 runs  1 085 ✅ 13 💤 0 ❌

Results for commit 546eb55.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Jan 30, 2025

Unit Test Performance Difference

Test Suite $Status$ Time on main $±Time$ $±Tests$ $±Skipped$ $±Failures$ $±Errors$
JointModelSamples 💔 $8.26$ $+24.34$ $0$ $-1$ $0$ $0$
compile 💚 $10.41$ $-8.76$ $0$ $0$ $0$ $0$
extract_quantities 💚 $12.26$ $-7.90$ $0$ $0$ $0$ $0$
misc_models 💚 $11.63$ $-7.77$ $0$ $0$ $0$ $0$
model_multi_chain 💚 $52.22$ $-49.61$ $0$ $0$ $0$ $0$
Additional test case details
Test Suite $Status$ Time on main $±Time$ Test Case
Grid 💚 $85.60$ $-83.44$ Grid_objects_work_with_QuantityGenerator_and_QuantityCollapser
GridPrediction 💔 $114.87$ $+13.10$ GridPrediction_works_as_expected_for_Survival_models
JointModelSamples 💔 $8.26$ $+24.34$ print_works_as_expected_for_JointModelSamples
LongitudinalClaretBruno 💚 $1770.08$ $-1763.65$ LongitudinalClaretBruno_works_as_expected_with_default_arguments
LongitudinalGSF 💚 $683.93$ $-676.63$ LongitudinalGSF_works_as_expected_with_default_arguments
LongitudinalQuantiles 💔 $39.08$ $+11.78$ Test_that_LongitudinalQuantities_works_as_expected
LongitudinalRandomSlope 💚 $67.58$ $-41.46$ Print_method_for_LongitudinalRandomSlope_works_as_expected
LongitudinalSteinFojo 💚 $1446.63$ $-1438.86$ LongitudinalSteinFojo_works_as_expected_with_default_arguments
SimSurvival 💔 $3.18$ $+1.11$ SimSurvivalExponential_creates_a_dataset_with_the_correct_parameter
SurvivalExponential 💚 $23.64$ $-21.98$ Can_load_and_compile_SurvivalExponential_model
SurvivalGamma 💚 $193.96$ $-192.45$ Print_method_for_SurvivalGamma_works_as_expected
SurvivalLoglogistic 💚 $40.06$ $-38.57$ Can_load_and_compile_SurvivalLogLogistic_model
SurvivalQuantities 💚 $78.73$ $-48.28$ SurvivalQuantities_and_autoplot.SurvivalQuantities_works_as_expected
SurvivalWeibullPH 💚 $14.69$ $-13.28$ Print_method_for_SurvivalWeibullPH_works_as_expected
brierScore 💚 $17.18$ $-11.29$ brierScore_SurvivalQuantities_returns_same_results_as_survreg
brms_compare 💚 $327.21$ $-327.13$ jmpost_and_brms_get_similar_loo_for_longitudinal_models
compile 💚 $10.41$ $-8.76$ compileStanModel_doesn_t_error_if_the_directory_doesn_t_exist
extract_quantities 💚 $12.26$ $-7.90$ extract_quantities_works_as_expected
misc_models 💚 $11.63$ $-7.77$ Longitudinal_Model_doesn_t_print_sampler_rejection_messages
model_multi_chain 💚 $52.22$ $-49.61$ Can_recover_known_distribution_parameters_from_random_slope_model_when_using_multiple_chains
stan_functions 💚 $38.12$ $-7.68$ GSF_SLD_function_works_as_expected

Results for commit ca96482

♻️ This comment has been updated with latest results.

@gravesti
Copy link
Collaborator

@gowerc For the brms examples I noticed the estimates are not near the diagonals like the jmpost ones are. Is that expected?

jmpost model

image

brms model

image

@gowerc
Copy link
Collaborator Author

gowerc commented Jan 31, 2025

@gravesti - That is not expected ...

Will double check and update accordingly

@gowerc
Copy link
Collaborator Author

gowerc commented Jan 31, 2025

@gravesti - I think the code is fine (ish) I think its just super sensitive to starting values. Most of the time I run the brms code I get the correct output as expected:
image

But every now and again I do get plots similar to what you showed, in these instances the sampler completes in about 10% of the usual time and the diagnostics are completely off. Running with longer warmups seems to correct this.

@gravesti
Copy link
Collaborator

gravesti commented Feb 3, 2025

Thanks @gowerc Maybe worth a note in the script for posterity to say that the models are sensitive to initial values are require a good warmup phase?

@gowerc
Copy link
Collaborator Author

gowerc commented Feb 18, 2025

@gravesti - Added a stronger warning message to highlight this.

Copy link
Contributor

badge

Code Coverage Summary

Filename                           Stmts    Miss  Cover    Missing
-------------------------------  -------  ------  -------  --------------------------------
R/brier_score.R                      167       0  100.00%
R/DataJoint.R                         76       2  97.37%   264, 270
R/DataLongitudinal.R                 119       1  99.16%   244
R/DataSubject.R                       85       1  98.82%   142
R/DataSurvival.R                      98       0  100.00%
R/defaults.R                          10       6  40.00%   18-57, 84
R/generics.R                          33       4  87.88%   52, 382, 402, 457
R/Grid.R                              27       1  96.30%   173
R/GridEven.R                          32       0  100.00%
R/GridEvent.R                         22       0  100.00%
R/GridFixed.R                         30       0  100.00%
R/GridGrouped.R                       52       0  100.00%
R/GridManual.R                        23       3  86.96%   78-80
R/GridObserved.R                      20       0  100.00%
R/GridPopulation.R                    31       4  87.10%   61, 69-71
R/GridPrediction.R                    36       6  83.33%   83, 91-95
R/JointModel.R                       129      10  92.25%   148-153, 203, 207, 249, 295, 301
R/JointModelSamples.R                 66       0  100.00%
R/link_generics.R                     16       4  75.00%   59, 76, 91, 106
R/Link.R                              62       4  93.55%   200-203
R/LinkComponent.R                     19       3  84.21%   88, 120-121
R/LongitudinalClaretBruno.R           90       6  93.33%   168-172, 187
R/LongitudinalGSF.R                   95       0  100.00%
R/LongitudinalModel.R                 19       0  100.00%
R/LongitudinalQuantities.R            71       0  100.00%
R/LongitudinalRandomSlope.R           44       5  88.64%   102-106
R/LongitudinalSteinFojo.R             86       5  94.19%   159-163
R/Parameter.R                         14       0  100.00%
R/ParameterList.R                     42       1  97.62%   184
R/Prior.R                            254       6  97.64%   510, 654-668
R/Promise.R                           25       0  100.00%
R/Quantities.R                        60       0  100.00%
R/QuantityGeneratorPopulation.R       22       0  100.00%
R/QuantityGeneratorPrediction.R       48       0  100.00%
R/QuantityGeneratorSubject.R          19       0  100.00%
R/settings.R                          12      12  0.00%    55-69
R/SimGroup.R                           5       0  100.00%
R/SimJointData.R                      72       1  98.61%   103
R/SimLongitudinal.R                    5       2  60.00%   22, 40
R/SimLongitudinalClaretBruno.R        74       0  100.00%
R/SimLongitudinalGSF.R                80       0  100.00%
R/SimLongitudinalRandomSlope.R        42       0  100.00%
R/SimLongitudinalSteinFojo.R          69       0  100.00%
R/SimSurvival.R                      116       0  100.00%
R/StanModel.R                         15       0  100.00%
R/StanModule.R                       176       6  96.59%   192-193, 235, 246, 383, 411
R/SurvivalExponential.R               10       0  100.00%
R/SurvivalGamma.R                     13       0  100.00%
R/SurvivalLoglogistic.R               11       0  100.00%
R/SurvivalModel.R                     19       0  100.00%
R/SurvivalQuantities.R               170       1  99.41%   125
R/SurvivalWeibullPH.R                 13       0  100.00%
R/utilities.R                        144       2  98.61%   13, 335
R/zzz.R                               29      26  10.34%   4-8, 10-31, 38-47
TOTAL                               3117     122  96.09%

Diff against main

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: 546eb55

Minimum allowed coverage is 80%

♻️ This comment has been updated with latest results

@gowerc gowerc merged commit f055eae into main Feb 18, 2025
23 checks passed
@gowerc gowerc deleted the 417-brms-examples branch February 18, 2025 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add brms example code that uses multiple studies
2 participants