Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata.add_column can be slow #2125

Open
amontanez24 opened this issue Jul 12, 2024 · 0 comments
Open

Metadata.add_column can be slow #2125

amontanez24 opened this issue Jul 12, 2024 · 0 comments
Labels
feature:performance Related to time or memory usage feature request Request for a new feature

Comments

@amontanez24
Copy link
Contributor

Problem Description

The metadata.add_column method can sometimes be unreasonably slow (a few seconds). This seems to be because during its validation, we check if it isn't a faker function, which seems to be a slow method in RDT.

def _validate_sdtype(self, sdtype):
if not isinstance(sdtype, str):
raise InvalidMetadataError(
f'Invalid sdtype: {sdtype} is not a string. Please use one of the '
'supported SDV sdtypes.'
)
if sdtype in self._DEFAULT_SDTYPES:
return
if not is_faker_function(sdtype):
raise InvalidMetadataError(
f"Invalid sdtype: '{sdtype}' is not recognized. Please use one of the "
'supported SDV sdtypes.'
)

Expected behavior

  • See if the validation can be sped up. This may require speeding up the underlying function in RDT
@amontanez24 amontanez24 added feature request Request for a new feature feature:performance Related to time or memory usage labels Jul 12, 2024
@R-Palazzo R-Palazzo self-assigned this Jul 31, 2024
@R-Palazzo R-Palazzo added this to the 1.15.1 milestone Jul 31, 2024
@R-Palazzo R-Palazzo removed this from the 1.15.1 milestone Aug 6, 2024
@R-Palazzo R-Palazzo removed their assignment Aug 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature:performance Related to time or memory usage feature request Request for a new feature
Projects
None yet
Development

No branches or pull requests

2 participants