-
Notifications
You must be signed in to change notification settings - Fork 24
/
CHANGELOG
503 lines (392 loc) · 20.2 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
Changelog
=========
## [1.14.0] - 2024-08-26
### Summary
This release adds support for multiple REDCap projects, in case the Optional Modules such as LBD or FTLD are broken out into their own REDCap projects. To pull from multiple REDCap projects into the same `redcap_input.csv` file, create a list of API tokens in your config file. This update also changes NACCulator's event detection to require the presence of both the Z1X and the A1 forms for UDS visits.
### Added
* Add additional unit test (Michael Bentz)
* Add pandas to setup.py (Michael Bentz)
* Add script to combine REDCap project data (Michael Bentz)
### Updated
* Update README with new requirements for ivp and fvp processing (Samantha Emerson)
* Add A1 to required forms for IVP and FVP processing (Samantha Emerson)
* Bump report_handler dependency to 1.3.0 (Michael Bentz)
## [1.13.1] - 2024-04-15
### Summary
This release removes the `.vscode` sub-module from the NACCulator repo, since it points to a private repository and makes NACCulator difficult to install for those who do not have permissions for the .vscode repo.
### Removed
* Remove private submodule (Michael Bentz)
## [1.13.0] - 2024-03-07
### Summary
This release adds logging functionality (report_handler) to NACCulator's output (found in the `logs` folder), along with several minor bug fixes.
### Added
* Added report_handler (Kshitij Sinha)
### Fixed
* Fix filter name in nacculator_cfg.ini.example (Samantha Emerson)
* Add A3NOT field to ivp and fvp builder files (Samantha Emerson)
* Add FTLD fields to Z1X in builder.py for ivp and fvp (Samantha Emerson)
## [1.12.1] - 2023-10-27
### Summary
This release fixes some blanking logic errors in the new Neuropath version 11 module.
### Updated
* Fix blanking rules in NP 11 (Samantha Emerson)
## [1.12.0] - 2023-10-23
### Summary
This release adds processing for Telephone Initial Packets and Neuropath version 11 to NACCulator.
### Added
* Add Neuropath 11 option to processing with complete form and builder files (Samantha Emerson)
* Split neuropath into versions 10 and 11 (Samantha Emerson)
* Add TIP to NACCulator options (Samantha Emerson)
* Add unit tests for tip (Samantha Emerson)
* Add visitnum to error and warning printouts (Samantha Emerson)
* Add vscode config submodule (Michael Bentz)
* Add 'CLS' to CLS warnings for clarity of which form is being talked about (Samantha Emerson)
### Updated
* Add logic to milestone event detection to skip forms marked as Incomplete (0) (Samantha Emerson)
* Add skip to warning for incomplete CLS (Samantha Emerson)
* Update and add unit tests (Samantha Emerson)
## [1.11.0] - 2022-08-31
### Summary
This release updates documentation on the developer end for the functions within NACCulator's code.
### Changed
* Updated every file with new docstrings and comments to better explain how the code works (Samantha Emerson)
* Changed the name of visit number fix filter to be more clear (Samantha Emerson)
## [1.10.0] - 2022-08-31
### Summary
This release introduces minor fixes to the FTLD and LBD Short Version modules.
### Added
* Add missing field for #3 in LBD Short Version FVP form E2L in the builder file (Samantha Emerson)
* Add ability to run filters on a local redcap export csv rather than automatically downloading via redcap api (Samantha Emerson)
### Changed
* Update gitignore (mbentz-uf)
* Update allowable_favlues on FTDRATIO in ftld forms.py files to account for error code (Samantha Emerson)
* Re-adjust Z1X spacing on FVP LBDSV to account for invisible deprecated C1 form (Samantha Emerson)
* Update Z1X positions in forms.py for lbd short version (Samantha Emerson)
## [1.9.0] - 2022-06-24
### Summary
This release changes our REDCap API call from cappy to PyCap, which allows for
a more flexible data export. This new library makes it so a REDCap project of
any size can be automatically exported; the filters will no longer fail when
attempting to export a REDCap data csv larger than 30mb.
### Updated
* Update run_filters.py to use PyCap instead of cappy
## [1.8.2] - 2022-04-26
### Summary
This release fixes another bug in the FTLD blanking logic parsing that caused
an error for rules that contained both a multi-part question and a comma
(i.e. "Question 3b,").
### Updated
* Fix regular expression parsing for blanking rules in FTLD module to account for unusual punctuation in DED
## [1.8.1] - 2021-11-04
### Summary
This release provides one major bug fix in the FTLD follow-up packet, updated
installation instructions, and a few minor fixes for bugs in error logging
etc.
### Updated
* Fix cappy install instructions in README
* Update FTLD FVP Z1X with fixed column spacing
## [1.8.0] - 2021-09-13
### Summary
### Updated
* Add Z1X processing to LBD short version
* Update UDS Z1X to include handling for optional LBD short version fields
* Make C2T optional for telephone follow-ups to reflect NACC's DED
### Added
* Add new CV covid module
## [1.7.1] - 2021-02-03
### Summary
This release updates the telephone follow-up packet (TFP) module to include
all allowable values in the NACC data element dictionary (DED).
### Updated
* C2T and A3 allowable values in TFP version 3.2 forms.py
## [1.7.0] - 2021-01-15
### Summary
This release incorporates the telephone followup packet 3.2 module which
includes the C2T form. Centers will be able to process data from forms like
the T1 and C2T, along with the expanded telehealth regimen in the other
forms provided by NACC due to the COVID-19 pandemic.
### Added
* Unit tests for TFP module functionality
* Re-add skipping logic to TFP form a3 from the updated DED for TFP
* Added the tele_ prefix to drug id filter for form A4
## [1.6.0] - 2020-12-15
### Summary
This change was implemented after getting feedback from OHSU. Some ADRCs do
not have the optional forms like A2 or A3 in their REDCap project at all, since
they will not be used for that center. NACCulator used to run with the
requirement that all forms be present in a REDCap project, whether they were
optional or not. This change is very similar to the one made previously that
made the CLS, Z1, and C1 forms optional for the REDCap csv input.
I also fixed a bug in the FTLD FVP builder that would cause it to look for the
IVP Z1X form.
### added
* Make optional forms for FTLD optional in REDCap project
* Make all optional UDS3 forms optional in REDCap project
### changed
* Fix cruft in FVP builder (Samantha Emerson)
* Update README to account for cappy install complication
* Fix ftld fvp builder to take followup Z1X instead of initial
## [1.5.0] - 2020-12-08
### Summary
This version introduces several smaller maintenance-level updates.
The FTLD builder files now function similar to the UDS3 module builders,
where NACCulator will first scan the Z1X to see which forms are present
in the record rather than automatically trying to process all possible
forms. This will prevent users from having to repeatedly manually delete
empty optional forms that will never be filled out.
NACCulator's event detection has been generalized to be more compatible
with more centers. The requirement for the initial visit packet's event
name to contain "initial_visit" has been changed to "initial", and the
requirement for the followup visit packet's event event name to contain
"followup_visit" has been changed to just "follow" (to account for variants
with either "followup" OR "follow_up").
The README has been updated to include more information about what data
NACCulator needs from REDCap in order to run correctly (such as the filled
Z1X form), and updated documentation on running the filters. I corrected the
example commands for running the individual filters (with an input csv AND an
output csv) so that they work properly now. All filter commands now require an
argument specifying the config file so that the "validate" function in
filters.py works properly. This makes the command structure consistent across
all filters.
### added
* Add note about Z1X to README
* Add optional form logic to fvp builder
* Add optional form logic to ftld ivp builder
### changed
* Fix typo in README
* Edit NACCulator capitalization to be consistent within README
* Update documentation on filters and adjust example config file to make it easier to comment/uncomment filters
* Fixed typo fukid9agd and fusib17pdx to have underscores in uds fvp builder
* Update compatible redcap_event_names to be more general (initial, follow)
## [1.4.0] - 2020-10-21
### Summary
This version adds the LBD short version, updates the cappy dependency, and
quality of life updates for CLS forms optional, expanding the range on ADC
IDs to 99, filters to accept milestone packets, and wording for error logging
to point out the form in error.
### added
* Add documentation, update filters.py, add details to error logging
* Add logic to make the CLS form optional in the input csv
## [1.3.0] - 2020-06-30
### Summary
This version reflects changes to have NACCulator be more compatible with more centers. We removed some hard coded variables for the 1Florida ADRC.
There were changes to how the deprecated Z1 and C1S forms are handled as well qas updates to tests for new functionality in the program.
### Added
* Add Z1 skipping to TFP builder (Samantha Emerson)
* Add tests for new functionality on skip logic and CSV formats (Samantha Emerson)
* Add run_filters.py to setup.py installation (Samantha Emerson)
* Add C1S form skip to uds ivp and fvp builders (Samantha Emerson)
* Add Z1 form skipping to uds3 fvp (Samantha Emerson)
* Add Z1 form skipping to nacculator uds3 ivp (Samantha Emerson)
### Changed
* Complete filter adjustments and repair associated unit tests (Samantha Emerson)
### Removed
* Remove filter that removes all events that are not uds3 initial or followup (Samantha Emerson)
### Updated
* Update and revise README (Samantha Emerson)
* Fix typos in IVP and FVP builder files
* Modify form C1S allowable_values for LOGIPREV (Samantha Emerson)
* Edit filters to accept any AD center's PTID from their config file (Samantha Emerson)
* Update README.md (Taeber Rapczak)
* Move Generating Forms to minimize confusion (Taeber Rapczak)
* Update generator to handle new CSV DED format (Taeber Rapczak)
## [1.2.0] - 2020-04-13
### Summary
Add TFP capability to NACCulator, remove the need for the split_ivp_fvp.sh file.
### Added
* Added internal logic to differentiate visit packets in one csv (Samantha Emerson)
* Added support for Telephone Followup Packets (-tfp flag) (Samantha Emerson and Cooper Martin)
### Changed
* Update ADCID range in milestone forms.py (Samantha Emerson)
* Update README file (Samantha Emerson)
## [1.1.0] - 2020-02-24
### Summary
Change the blanking rule special cases to add specificity to error message and added statement to split function to handle Neuropathology forms.
### Added
* Add allowable_values to lbd forms.py files to reflect nacc's lbd data element dictionary (Samantha Emerson)
* Add type hinting to convert_rule_to_python in each module's blanks file (Samantha Emerson)
* Add if/else logic to blanking rules special cases to single out the violated rule, adjust unit tests to account for new code (Samantha Emerson)
* Add 'Blank if question not answered' regex to ftld blanks file to reduce special cases (Samantha Emerson)
* Add neuropathology statement to split_ivp_fvp.sh file, delint files in np folder (Samantha Emerson)
### Changed
* Change field '54MOMNEUR' to '5a4 MOMNEUR' to reflect NACC's uds3 ivp ded and remove unnecessary special case from uds3 blanks (Samantha Emerson)
* Update copyright date in np forms and builder (Samantha Emerson)
* Update both fvp and ivp DIGFORSL and DIGBACLS fields to include 0 in the allowable_values (Samantha Emerson)
* Update uds3 fvp forms.py to correct an error in TRAILA and TRAILB allowable_values (Samantha Emerson)
* Update uds3 ivp and fvp forms.py to reflect changes in nacc forms since 2015 (Samantha Emerson)
* Update uds3 fvp forms.py to include 1florida id number in range (Samantha Emerson)
### Removed
* Remove unnecessary arguments from uds3 blanks special cases (Samantha Emerson)
## [1.0.1] - 2019-12-05
### Summary
Fixed broken build process for setup.py.
## [1.0.0] - 2019-12-05
### Summary
Updated to Python3 and added CSF, LBD, FTLD modules.
### Added
* Add Python version requirement (Taeber Rapczak)
* Add FTLD module (Samantha Emerson)
* Add CSF module (Samantha Emerson)
* Add LBD module (Samantha Emerson)
* Add Pull Request template (Taeber Rapczak)
* Add character check function for illegal characters (Samantha Emerson)
### Updated
* Rewrite generator for Python 3 by hand (Taeber Rapczak)
* Convert to Python 3 (Cooper Martin)
* Use more nuanced method for C1 vs C2 and Z1 vs Z1X selection (Naomi Braun)
* Adjust z1_z1x_test unit test to account for newly changed headers (Samantha Emerson)
* Fix various module form errors that the new range checker found (Samantha Emerson)
* Fix copyright, version number, tools, and tests organization. (Taeber Rapczak)
* Correct IVP D1 headers (Naomi Braun)
* Correct IVP B8 headers (Naomi Braun)
* Adjust blanking rules to account for 'blank if not answered' statements and add groundwork for some fields with recursive blanking rules (Samantha Emerson)
## [0.7.1] - 2019-09-26
### Summary
Add milestone packets and unit tests.
### Added
* Create filter for good and bad PTIDs
* Add milestone form
* Add unit tests for filters
### Changed
* Modify filters to be usable with I/O independent tests
### Fixed
* Make CLS form addable to followup visits
* Filter out certified visits
## [0.6.1] - 2019-05-19
### Summary
Fix field names and blanks circumstances.
### Fixed
* Incorrect fields used in blanks check on D1 form (Naomi Braun, Nicolas May)
* Field names missing in blanks check on A3 form (Naomi Braun)
* Adjust builder files to use correct field names for Z1X form (Naomi Braun)
## [0.6.0] - 2019-04-17
### Summary
Updated readme file for run instructions, added more tests, corrected assertion handling, add split script
### Added
* Handle assertion errors gracefully (Naomi Braun)
* Add tests for CLS issues that cause Warnings (Naomi Braun)
* Fix errors caused by reformatting (Naomi Braun)
* Restore packet rejection for CLS form on an early packet and remove outdated tests (Naomi Braun)
* Create z1/z1x function and tests (Naomi Braun)
* Add script for ivp/fvp split (Naomi Braun)
### Changed
* Adjust tests for new expected behavior (Naomi Braun)
* Update README with example workflow reflecting our process (Naomi Braun)
* Stop most CLS errors from preventing packet upload (Naomi Braun)
### Fixed
* Fix typo in comment (Naomi Braun)
* Remove error for missing forms on Z1/Z1X and C1S/C2 (Naomi Braun)
## [0.5.0] - 2019-01-19
### Summary
Added automated testing, date checking for certain forms, and bug fixes.
### Added
* Unit tests and Travis CI integration (Taeber Rapczak)
* Several checks for properly including or omitting Form CLS (Taeber Rapczak, Naomi Braun)
* Add date check for C1S/C2 and Z1X/Z1 selection (Naomi Braun)
### Fixed
* Incorrect unknown value '999' for Form A2 INEDUC (Anthony Rodriguez, Nicolas May)
* C1S/C2 FVP checks (Naomi Braun)
* Fixed logic for assuring A5 5f details are blank when arthrit is 0 (Naomi Braun)
## [0.4.1] - 2018-06-14
### Summary
This release corrects some form logic bugs and adds a new filter to normalize visit numbers.
### Added
* add a new filter to correct visit dates (Matthew McConnell)
### Fixed
* fix bug with form logic (Matthew McConnell)
* fix form position for z1x on fvp (Matthew McConnell)
* validate that cls was completed in a particular packet before removing a packet missing the form (Matthew McConnell)
* update fvp field names for cls (Matthew McConnell)
## [0.4.0] - 2018-06-14
### Summary
Nacculator now has logic built in for the Z1X and CLS forms. Also, there
were various bug fixes and code clean up.
### Added
* Add CLS to IVP (Naomi Braun)
* Add CLS to ivp and fvp (Naomi Braun)
* Adding Z1X Form to Nacculator (AjanthaRamineni)
* add z1x to fvp (Matthew McConnell)
### Changed
* Update builder.py (Matthew McConnell)
* Add error for CLS forms appearing too soon (Naomi Braun)
### Fixed
* fix clean_ptid filter to remove only subjects in the nacc current db (Matthew McConnell)
* Account for blank spaces in percentage fields (Naomi Braun)
* Fix typo in form function call (Naomi Braun)
* Fix typo in header (Naomi Braun)
* fix bug with c2 form 12d verification (Matthew McConnell)
* Add extra options in inclusive range for ADCID in header rows (Naomi Braun)
## [0.3.1] - 2018-05-21
### Added
* add csv and txt files to ignore file to avoid adding data to repo (Kevin Hanson)
* Update filters.py (AjanthaRamineni)
* ReadMe to new filter (AjanthaRamineni)
* Adding Lambda functions to filters (AjanthaRamineni)
### Changed
* Adding ReadMe to new filter (AjanthaRamineni)
* Update README.md (AjanthaRamineni)
* Adjusting a Single Ptid file (AjanthaRamineni)
* Extracting a file with Visit Number a type Specified (AjanthaRamineni)
* New filter for Grabbing A Ptid (AjanthaRamineni)
## [0.3.0] - 2017-12-07
### Added
* add a reqs file for using a venv (Matthew McConnell)
* Create new filters.py script to pull data from REDCap and run filters on it (AjanthaRamineni)
* Add a filter to change headers based on config (Matthew McConnell)
* Add fiter to remove unneccessary RedCap Events (AjanthaRamineni)
* Add a script to run all the filters with a config file (Matthew McConnell)
### Changed
* Add a config across all filters with a decorator for validation (Matthew McConnell)
* Change how filter_ptid works - it now checks ptid, visit type, and visit num (AjanthaRamineni)
* Update the aod range for kids and siblings. Changed field from num to char (Matthew McConnell)
* Update changelog and setup (Tarun Gupta Akirala)
* Update README in preperation for release (Matthew McConnell)
* Update notes (Tarun Gupta Akirala)
### Fixed]
* Fix logic for determing existance of C1S and C2 forms (AjanthaRamineni)
* Fix column numbering on FVP B8 form (Naomi Braun)
* Fix bug where 2 questions should be able to hold values regardless of other values (Matthew McConnell)
* fixes #25 and adds more debugging logs (Tarun Gupta Akirala)
## [0.2.4] - 2017-03-27
### Added
* Added few log statements as changes.
* Project handover from takirala to ctsit
## [0.2.3] - 2017-02-14
### Changed
* Refactored c1s form - redcap C1 form to C1S in alz website
### Added
* Added few fields in the fvp form builder and forms file.
## [0.2.2] - 2017-02-02
### Changed
* Header key names for fvp forms.
* Added few fixes for filters to correct fvp forms.
### Added
* Added logic to skip corrupt forms entries and continue processing valid forms
## [0.2.1] - 2016-01-09
### Changed
* While parsing a csv file record by record, if one of the record has both c1s and c2 forms, then that specific record will be discarded and the processing for remaining records will still be continued.
### Added
* Added filters for clean up of data. (Tarun Gupta Akirala)
* Added five filters : {cleanPtid,updateField,replaceDrugId,fillDefault,fixC1S} (Tarun Gupta Akirala)
* Filters output can be piped together to apply multiple filters in a sequence (Tarun Gupta Akirala)
## [0.2.0] - 2016-12-19
### Changed
* Changed header fields for np forms
* Removed redundant fields for np forms.
* Added mutually exclusive flag for fvp vs ivp forms (Tarun Gupta Akirala)
* Restructured, working version for ivp forms. (Tarun Gupta Akirala)
* Adding blank checks to discard empty records (Tarun Gupta Akirala)
### Added
* Added seperate classes for fvp and np forms (Tarun Gupta Akirala)
* Added new builder and class files for fvp, np and ivp forms.
* Added new forms (Tarun Gupta Akirala)
## [0.1.5] - 2016-10-10
### Changed
* Changed README file to reflect that output from NACCulator should be in .txt format (Tarun Gupta Akirala)
* Changed form version value for form C1S to be version 2 (Tarun Gupta Akirala)
### Added
* Added the Neuropathology form definitions and rules (Tarun Gupta Akirala)
* Created error messages where data does not meet form definitions (Tarun Gupta Akirala)
* Added the C1S temporary Spanish form definitions and rules (Tarun Gupta Akirala)
* Added flag to have NACCulator output only the Neuropathology form (Tarun Gupta Akirala)
* Added ability to check for data in C2 or C1S form and make determination on which form to use based on data present or not present (Tarun Gupta Akirala)j