Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tools for DataFrame processing and importing 3W dataset #50

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

zRafaF
Copy link

@zRafaF zRafaF commented Oct 15, 2024

Implements:

  • find_df_transitions: Finds transitions in the dataset that go over a treshold;
  • split_df_percentages: Function to split the data in its percentages.

QoL

  • Added instructions for testing the code with pytest
  • Added testing requierementes
  • Created vscode settings file: This will allow to set standard formatting to the code base, i suggest using black.

@zRafaF
Copy link
Author

zRafaF commented Oct 15, 2024

  • Imported 3w test dataset to the codebase

    • Loading the 3W dataset is possible with bibmon.load_3w() it returns the df and the config file parsed with ConfigParser
  • Added three_w.tools, a file that stores tooling specifically for the 3w dataset

three_w.tools

  • Created load_dataset_ini, a function to load the config file
  • Created split_dataset, a function to prepare the dataset for training, it allows the user to split the data when a transition state happnes, it also allows a subdivision of the dataset in training and validation dfs

QoL

  • Added unity tests to the new functions

@zRafaF zRafaF changed the title Implements two new function tools for DataFrame processing Tools for DataFrame processing and importing 3W dataset Oct 15, 2024
zRafaF and others added 2 commits October 15, 2024 19:17
* fixed relative imports

* fixed relative imports

* removed debbug prints
@zRafaF
Copy link
Author

zRafaF commented Oct 16, 2024

User can now pick another dataset from 3w to download. It will be downloaded and loaded using BytesIO so no file is written to the user's file system,

The download progress is displayed with tqdm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant