Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add an optional final report to dict_sweep helper function #303

Open
newgene opened this issue Oct 18, 2023 · 0 comments
Open

add an optional final report to dict_sweep helper function #303

newgene opened this issue Oct 18, 2023 · 0 comments
Assignees

Comments

@newgene
Copy link
Member

newgene commented Oct 18, 2023

utils.dataload.dict_sweep is used to:

Remove keys whose values are ".", "-", "", "NA", "none", " "; and remove empty dictionaries

It's quite powerful, but could be potentially dangerous to remove some fields we want to keep. So, it would be nice to have a final report at the end of its run, something like this:

<na_value_1>: 
     - <removed_field_name_1>: cnt_of_removed_values,
        ...
<na_value_2>:
    ...

can add a new report parameter to this function, default can be True.

Also worth checking if setting to True will significantly increase the processing time in a typical data plugin, if that is the case, we might set default to False and use it only if needed for debugging purpose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants