Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve circular import of the internal submodule #860

Open
dodamih opened this issue Jan 23, 2025 · 0 comments
Open

Resolve circular import of the internal submodule #860

dodamih opened this issue Jan 23, 2025 · 0 comments
Labels
bug Something isn't working mazepa solves problems needs investigation

Comments

@dodamih
Copy link
Collaborator

dodamih commented Jan 23, 2025

Our codebase has a circular import where zetta_utils.load_all_modules() loads internal, which loads from zetta_utils. This seems to be the cause of issues with using dill.dumps with recurse=True.

I have investigated using cloudpickle as an alternative, but this seems result in probabilistically inconsistent serialisation across processes that have been started using spawn. It seems to happen very consistently on GitHub CI but not when running locally, and investigating the serialised bytecode shows that the entrypoints are different:

  370: \x8c         SHORT_BINUNICODE '/home/runner/work/zetta_utils/zetta_utils/tests/unit/mazepa/test_id_generation.py'

vs.

  328: \x8c         SHORT_BINUNICODE 'unit.mazepa.test_id_generation'

It's not definite, but the most likely culprit for this is still the circular import as partially initialised modules are known to cause inconsistent behaviour across runs.

@dodamih dodamih added bug Something isn't working mazepa solves problems needs investigation labels Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working mazepa solves problems needs investigation
Projects
None yet
Development

No branches or pull requests

1 participant