Changelog
=========
## [1.13.1] - 2024-04-15
### Summary
This release removes the `.vscode` sub-module from the NACCulator repo, since it points to a private repository and makes NACCulator difficult to install for those who do not have permissions for the .vscode repo. 

### Removed
* Remove private submodule (Michael Bentz)

## [1.13.0] - 2024-03-07
### Summary
This release adds logging functionality (report_handler) to NACCulator's output (found in the `logs` folder), along with several minor bug fixes. 

### Added
 * Added report_handler (Kshitij Sinha)

### Fixed
 * Fix filter name in nacculator_cfg.ini.example (Samantha Emerson)
 * Add A3NOT field to ivp and fvp builder files (Samantha Emerson)
 * Add FTLD fields to Z1X in builder.py for ivp and fvp (Samantha Emerson)

## [1.12.1] - 2023-10-27
### Summary
This release fixes some blanking logic errors in the new Neuropath version 11 module. 

### Updated
 * Fix blanking rules in NP 11 (Samantha Emerson)

## [1.12.0] - 2023-10-23
### Summary
This release adds processing for Telephone Initial Packets and Neuropath version 11 to NACCulator. 

### Added
 * Add Neuropath 11 option to processing with complete form and builder files (Samantha Emerson)
 * Split neuropath into versions 10 and 11 (Samantha Emerson)
 * Add TIP to NACCulator options (Samantha Emerson)
 * Add unit tests for tip (Samantha Emerson)
 * Add visitnum to error and warning printouts (Samantha Emerson)
 * Add vscode config submodule (Michael Bentz)
 * Add 'CLS' to CLS warnings for clarity of which form is being talked about (Samantha Emerson)

### Updated
 * Add logic to milestone event detection to skip forms marked as Incomplete (0) (Samantha Emerson)
 * Add skip to warning for incomplete CLS (Samantha Emerson)
 * Update and add unit tests (Samantha Emerson)

## [1.11.0] - 2022-08-31
### Summary
This release updates documentation on the developer end for the functions within NACCulator's code.

### Changed
* Updated every file with new docstrings and comments to better explain how the code works (Samantha Emerson)
* Changed the name of visit number fix filter to be more clear (Samantha Emerson)

## [1.10.0] - 2022-08-31
### Summary
This release introduces minor fixes to the FTLD and LBD Short Version modules.

### Added
 * Add missing field for #3 in LBD Short Version FVP form E2L in the builder file (Samantha Emerson)
 * Add ability to run filters on a local redcap export csv rather than automatically downloading via redcap api (Samantha Emerson)

### Changed
 * Update gitignore (mbentz-uf)
 * Update allowable_favlues on FTDRATIO in ftld forms.py files to account for error code (Samantha Emerson)
 * Re-adjust Z1X spacing on FVP LBDSV to account for invisible deprecated C1 form (Samantha Emerson)
 * Update Z1X positions in forms.py for lbd short version (Samantha Emerson)

## [1.9.0] - 2022-06-24

### Summary
This release changes our REDCap API call from cappy to PyCap, which allows for
a more flexible data export. This new library makes it so a REDCap project of
any size can be automatically exported; the filters will no longer fail when
attempting to export a REDCap data csv larger than 30mb.

### Updated
 * Update run_filters.py to use PyCap instead of cappy
## [1.8.2] - 2022-04-26

### Summary
This release fixes another bug in the FTLD blanking logic parsing that caused
an error for rules that contained both a multi-part question and a comma
(i.e. "Question 3b,").

### Updated
 * Fix regular expression parsing for blanking rules in FTLD module to account for unusual punctuation in DED

## [1.8.1] - 2021-11-04
### Summary
This release provides one major bug fix in the FTLD follow-up packet, updated
installation instructions, and a few minor fixes for bugs in error logging
etc.

### Updated
 * Fix cappy install instructions in README
 * Update FTLD FVP Z1X with fixed column spacing


## [1.8.0] - 2021-09-13
### Summary

### Updated
 * Add Z1X processing to LBD short version
 * Update UDS Z1X to include handling for optional LBD short version fields
 * Make C2T optional for telephone follow-ups to reflect NACC's DED

### Added
 * Add new CV covid module


## [1.7.1] - 2021-02-03
### Summary
This release updates the telephone follow-up packet (TFP) module to include
all allowable values in the NACC data element dictionary (DED).

### Updated
* C2T and A3 allowable values in TFP version 3.2 forms.py


## [1.7.0] - 2021-01-15
### Summary
This release incorporates the telephone followup packet 3.2 module which
includes the C2T form. Centers will be able to process data from forms like
the T1 and C2T, along with the expanded telehealth regimen in the other
forms provided by NACC due to the COVID-19 pandemic.

### Added
 * Unit tests for TFP module functionality
 * Re-add skipping logic to TFP form a3 from the updated DED for TFP
 * Added the tele_ prefix to drug id filter for form A4

## [1.6.0] - 2020-12-15
### Summary
This change was implemented after getting feedback from OHSU. Some ADRCs do
not have the optional forms like A2 or A3 in their REDCap project at all, since
they will not be used for that center. NACCulator used to run with the
requirement that all forms be present in a REDCap project, whether they were
optional or not. This change is very similar to the one made previously that
made the CLS, Z1, and C1 forms optional for the REDCap csv input.
I also fixed a bug in the FTLD FVP builder that would cause it to look for the
IVP Z1X form.

### added
 * Make optional forms for FTLD optional in REDCap project
 * Make all optional UDS3 forms optional in REDCap project


### changed
* Fix cruft in FVP builder (Samantha Emerson)
* Update README to account for cappy install complication
* Fix ftld fvp builder to take followup Z1X instead of initial

## [1.5.0] - 2020-12-08
### Summary

This version introduces several smaller maintenance-level updates.
The FTLD builder files now function similar to the UDS3 module builders,
where NACCulator will first scan the Z1X to see which forms are present
in the record rather than automatically trying to process all possible
forms. This will prevent users from having to repeatedly manually delete
empty optional forms that will never be filled out.

NACCulator's event detection has been generalized to be more compatible
with more centers. The requirement for the initial visit packet's event
name to contain "initial_visit" has been changed to "initial", and the
requirement for the followup visit packet's event event name to contain
"followup_visit" has been changed to just "follow" (to account for variants
with either "followup" OR "follow_up").

The README has been updated to include more information about what data
NACCulator needs from REDCap in order to run correctly (such as the filled
Z1X form), and updated documentation on running the filters. I corrected the
example commands for running the individual filters (with an input csv AND an
output csv) so that they work properly now. All filter commands now require an
argument specifying the config file so that the "validate" function in
filters.py works properly. This makes the command structure consistent across
all filters.

### added
 * Add note about Z1X to README
 * Add optional form logic to fvp builder
 * Add optional form logic to ftld ivp builder

### changed
 * Fix typo in README
 * Edit NACCulator capitalization to be consistent within README
 * Update documentation on filters and adjust example config file to make it easier to comment/uncomment filters
 * Fixed typo fukid9agd and fusib17pdx to have underscores in uds fvp builder
 * Update compatible redcap_event_names to be more general (initial, follow)

## [1.4.0] - 2020-10-21
### Summary

This version adds the LBD short version, updates the cappy dependency, and
quality of life updates for CLS forms optional, expanding the range on ADC
IDs to 99, filters to accept milestone packets, and wording for error logging
to point out the form in error.

### added
 * Add documentation, update filters.py, add details to error logging
 * Add logic to make the CLS form optional in the input csv

## [1.3.0] - 2020-06-30
### Summary

This version reflects changes to have NACCulator be more compatible with more centers. We removed some hard coded variables for the 1Florida ADRC.
There were changes to how the deprecated Z1 and C1S forms are handled as well qas updates to tests for new functionality in the program.

### Added
 * Add Z1 skipping to TFP builder (Samantha Emerson)
 * Add tests for new functionality on skip logic and CSV formats (Samantha Emerson)
 * Add run_filters.py to setup.py installation (Samantha Emerson)
 * Add C1S form skip to uds ivp and fvp builders (Samantha Emerson)
 * Add Z1 form skipping to uds3 fvp (Samantha Emerson)
 * Add Z1 form skipping to nacculator uds3 ivp (Samantha Emerson)

 ### Changed
 * Complete filter adjustments and repair associated unit tests (Samantha Emerson)

 ### Removed
 * Remove filter that removes all events that are not uds3 initial or followup (Samantha Emerson)

 ### Updated
 * Update and revise README (Samantha Emerson)
 * Fix typos in IVP and FVP builder files
 * Modify form C1S allowable_values for LOGIPREV (Samantha Emerson)
 * Edit filters to accept any AD center's PTID from their config file (Samantha Emerson)
 * Update README.md (Taeber Rapczak)
 * Move Generating Forms to minimize confusion (Taeber Rapczak)
 * Update generator to handle new CSV DED format (Taeber Rapczak)

## [1.2.0] - 2020-04-13
### Summary

Add TFP capability to NACCulator, remove the need for the split_ivp_fvp.sh file.

### Added
 * Added internal logic to differentiate visit packets in one csv (Samantha Emerson)
 * Added support for Telephone Followup Packets (-tfp flag) (Samantha Emerson and Cooper Martin)

### Changed
* Update ADCID range in milestone forms.py (Samantha Emerson)
* Update README file (Samantha Emerson)

## [1.1.0] - 2020-02-24
### Summary

Change the blanking rule special cases to add specificity to error message and added statement to split function to handle Neuropathology forms.

### Added
 * Add allowable_values to lbd forms.py files to reflect nacc's lbd data element dictionary (Samantha Emerson)
 * Add type hinting to convert_rule_to_python in each module's blanks file (Samantha Emerson)
 * Add if/else logic to blanking rules special cases to single out the violated rule, adjust unit tests to account for new code (Samantha Emerson)
 * Add 'Blank if question not answered' regex to ftld blanks file to reduce special cases (Samantha Emerson)
 * Add neuropathology statement to split_ivp_fvp.sh file, delint files in np folder (Samantha Emerson)

### Changed
 * Change field '54MOMNEUR' to '5a4 MOMNEUR' to reflect NACC's uds3 ivp ded and remove unnecessary special case from uds3 blanks (Samantha Emerson)
 * Update copyright date in np forms and builder (Samantha Emerson)
 * Update both fvp and ivp DIGFORSL and DIGBACLS fields to include 0 in the allowable_values (Samantha Emerson)
 * Update uds3 fvp forms.py to correct an error in TRAILA and TRAILB allowable_values (Samantha Emerson)
 * Update uds3 ivp and fvp forms.py to reflect changes in nacc forms since 2015 (Samantha Emerson)
 * Update uds3 fvp forms.py to include 1florida id number in range (Samantha Emerson)

### Removed
 * Remove unnecessary arguments from uds3 blanks special cases (Samantha Emerson)

## [1.0.1] - 2019-12-05
### Summary

Fixed broken build process for setup.py.


## [1.0.0] - 2019-12-05
### Summary

Updated to Python3 and added CSF, LBD, FTLD modules.

### Added
 * Add Python version requirement (Taeber Rapczak)
 * Add FTLD module (Samantha Emerson)
 * Add CSF module (Samantha Emerson)
 * Add LBD module (Samantha Emerson)
 * Add Pull Request template (Taeber Rapczak)
 * Add character check function for illegal characters (Samantha Emerson)

### Updated
 * Rewrite generator for Python 3 by hand (Taeber Rapczak)
 * Convert to Python 3 (Cooper Martin)
 * Use more nuanced method for C1 vs C2 and Z1 vs Z1X selection (Naomi Braun)
 * Adjust z1_z1x_test unit test to account for newly changed headers (Samantha Emerson)
 * Fix various module form errors that the new range checker found (Samantha Emerson)
 * Fix copyright, version number, tools, and tests organization. (Taeber Rapczak)
 * Correct IVP D1 headers (Naomi Braun)
 * Correct IVP B8 headers (Naomi Braun)
 * Adjust blanking rules to account for 'blank if not answered' statements and add groundwork for some fields with recursive blanking rules (Samantha Emerson)


## [0.7.1] - 2019-09-26
### Summary

Add milestone packets and unit tests.

### Added
 * Create filter for good and bad PTIDs
 * Add milestone form
 * Add unit tests for filters

### Changed
 * Modify filters to be usable with I/O independent tests

### Fixed
 * Make CLS form addable to followup visits
 * Filter out certified visits


## [0.6.1] - 2019-05-19
### Summary

Fix field names and blanks circumstances.

### Fixed
 * Incorrect fields used in blanks check on D1 form (Naomi Braun, Nicolas May)
 * Field names missing in blanks check on A3 form (Naomi Braun)
 * Adjust builder files to use correct field names for Z1X form (Naomi Braun)


## [0.6.0] - 2019-04-17
### Summary

Updated readme file for run instructions, added more tests, corrected assertion handling, add split script

### Added
 * Handle assertion errors gracefully (Naomi Braun)
 * Add tests for CLS issues that cause Warnings (Naomi Braun)
 * Fix errors caused by reformatting (Naomi Braun)
 * Restore packet rejection for CLS form on an early packet and remove outdated tests (Naomi Braun)
 * Create z1/z1x function and tests (Naomi Braun)
 * Add script for ivp/fvp split (Naomi Braun)

### Changed
 * Adjust tests for new expected behavior (Naomi Braun)
 * Update README with example workflow reflecting our process (Naomi Braun)
 * Stop most CLS errors from preventing packet upload (Naomi Braun)

### Fixed
 * Fix typo in comment (Naomi Braun)
 * Remove error for missing forms on Z1/Z1X and C1S/C2 (Naomi Braun)


## [0.5.0] - 2019-01-19
### Summary

Added automated testing, date checking for certain forms, and bug fixes.

### Added
 * Unit tests and Travis CI integration (Taeber Rapczak)
 * Several checks for properly including or omitting Form CLS (Taeber Rapczak, Naomi Braun)
 * Add date check for C1S/C2 and Z1X/Z1 selection (Naomi Braun)

### Fixed
 * Incorrect unknown value '999' for Form A2 INEDUC (Anthony Rodriguez, Nicolas May)
 * C1S/C2 FVP checks (Naomi Braun)
 * Fixed logic for assuring A5 5f details are blank when arthrit is 0 (Naomi Braun)


## [0.4.1] - 2018-06-14
### Summary
 This release corrects some form logic bugs and adds a new filter to normalize visit numbers.
### Added
 * add a new filter to correct visit dates (Matthew McConnell)
### Fixed
 * fix bug with form logic (Matthew McConnell)
 * fix form position for z1x on fvp (Matthew McConnell)
 * validate that cls was completed in a particular packet before removing a packet missing the form (Matthew McConnell)
 * update fvp field names for cls (Matthew McConnell)

## [0.4.0] - 2018-06-14
### Summary
 Nacculator now has logic built in for the Z1X and CLS forms. Also, there
 were various bug fixes and code clean up.
### Added
 * Add CLS to IVP (Naomi Braun)
 * Add CLS to ivp and fvp (Naomi Braun)
 * Adding Z1X Form to Nacculator (AjanthaRamineni)
 * add z1x to fvp (Matthew McConnell)
### Changed
 * Update builder.py (Matthew McConnell)
 * Add error for CLS forms appearing too soon (Naomi Braun)
### Fixed
 * fix clean_ptid filter to remove only subjects in the nacc current db (Matthew McConnell)
 * Account for blank spaces in percentage fields (Naomi Braun)
 * Fix typo in form function call (Naomi Braun)
 * Fix typo in header (Naomi Braun)
 * fix bug with c2 form 12d verification (Matthew McConnell)
 * Add extra options in inclusive range for ADCID in header rows (Naomi Braun)

## [0.3.1] - 2018-05-21
### Added
* add csv and txt files to ignore file to avoid adding data to repo (Kevin Hanson)
* Update filters.py (AjanthaRamineni)
* ReadMe to new filter (AjanthaRamineni)
* Adding Lambda functions to filters (AjanthaRamineni)
### Changed
* Adding ReadMe to new filter (AjanthaRamineni)
* Update README.md (AjanthaRamineni)
* Adjusting a Single Ptid file (AjanthaRamineni)
* Extracting a file with Visit Number a type Specified (AjanthaRamineni)
* New filter for Grabbing A Ptid (AjanthaRamineni)

## [0.3.0] - 2017-12-07
### Added
 * add a reqs file for using a venv (Matthew McConnell)
 * Create new filters.py script to pull data from REDCap and run filters on it (AjanthaRamineni)
 * Add a filter to change headers based on config (Matthew McConnell)
 * Add fiter to remove unneccessary RedCap Events (AjanthaRamineni)
* Add a script to run all the filters with a config file (Matthew McConnell)
### Changed
 * Add a config across all filters with a decorator for validation (Matthew McConnell)
 * Change how filter_ptid works - it now checks ptid, visit type, and visit num (AjanthaRamineni)
 * Update the aod range for kids and siblings. Changed field from num to char (Matthew McConnell)
 * Update changelog and setup (Tarun Gupta Akirala)
 * Update README in preperation for release (Matthew McConnell)
 * Update notes (Tarun Gupta Akirala)
### Fixed]
* Fix logic for determing existance of C1S and C2 forms (AjanthaRamineni)
* Fix column numbering on FVP B8 form (Naomi Braun)
* Fix bug where 2 questions should be able to hold values regardless of other values (Matthew McConnell)
* fixes #25 and adds more debugging logs (Tarun Gupta Akirala)


 ## [0.2.4] - 2017-03-27
 ### Added
 * Added few log statements as changes.
 * Project handover from takirala to ctsit

 ## [0.2.3] - 2017-02-14
 ### Changed
 * Refactored c1s form - redcap C1 form to C1S in alz website

 ### Added
 * Added few fields in the fvp form builder and forms file.


 ## [0.2.2] - 2017-02-02
 ### Changed
 * Header key names for fvp forms.
 * Added few fixes for filters to correct fvp forms.

 ### Added
 * Added logic to skip corrupt forms entries and continue processing valid forms

 ## [0.2.1] - 2016-01-09
 ### Changed
 * While parsing a csv file record by record, if one of the record has both c1s and c2 forms, then that specific record will be discarded and the processing for remaining records will still be continued.

 ### Added
 * Added filters for clean up of data. (Tarun Gupta Akirala)
 * Added five filters : {cleanPtid,updateField,replaceDrugId,fillDefault,fixC1S} (Tarun Gupta Akirala)
 * Filters output can be piped together to apply multiple filters in a sequence (Tarun Gupta Akirala)

 ## [0.2.0] - 2016-12-19
 ### Changed
 * Changed header fields for np forms
 * Removed redundant fields for np forms.
 * Added mutually exclusive flag for fvp vs ivp forms (Tarun Gupta Akirala)
 * Restructured, working version for ivp forms. (Tarun Gupta Akirala)
 * Adding blank checks to discard empty records (Tarun Gupta Akirala)

 ### Added
 * Added seperate classes for fvp and np forms (Tarun Gupta Akirala)
 * Added new builder and class files for fvp, np and ivp forms.
 * Added new forms (Tarun Gupta Akirala)


 ## [0.1.5] - 2016-10-10
 ### Changed

 * Changed README file to reflect that output from NACCulator should be in .txt format (Tarun Gupta Akirala)
 * Changed form version value for form C1S to be version 2 (Tarun Gupta Akirala)


 ### Added
 * Added the Neuropathology form definitions and rules (Tarun Gupta Akirala)
 * Created error messages where data does not meet form definitions (Tarun Gupta Akirala)
 * Added the C1S temporary Spanish form definitions and rules (Tarun Gupta Akirala)
 * Added flag to have NACCulator output only the Neuropathology form (Tarun Gupta Akirala)
 * Added ability to check for data in C2 or C1S form and make determination on which form to use based on data present or not present (Tarun Gupta Akirala)j
