Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poor pickling performance on macOS #360

Open
lohedges opened this issue Jul 13, 2022 · 3 comments
Open

Poor pickling performance on macOS #360

lohedges opened this issue Jul 13, 2022 · 3 comments
Labels

Comments

@lohedges
Copy link
Member

lohedges commented Jul 13, 2022

As reported by @msuruzhon, pickling/unpickling large BioSimSpace objects has very poor performance on macOS. Using System.pickle.tar.gz we see:

import BioSimSpace.Sandpit.Exscientia as BSS

import pickle
import time

a = time.time()
s = pickle.load(open("System.pickle", "rb"))
b = time.time()
print(f"Unpickling BioSimSpace system took {b - a} seconds")

a = time.time()
with open("tmp.pickle", "wb") as f:
    pickle.dump(s, f)
b = time.time()
print(f"Pickling BioSimSpace system took {b - a} seconds")

a = time.time()
with open("tmp.pickle", "wb") as f:
    pickle.dump(s._sire_object, f)
b = time.time()
print(f"Pickling Sire system took {b - a} seconds")

Linux:

Unpickling BioSimSpace system took 2.6445674896240234 seconds
Pickling BioSimSpace system took 2.548236131668091 seconds
Pickling Sire system took 1.7895822525024414 seconds

macOS:

Unpickling BioSimSpace system took 3.2096989154815674 seconds
Pickling BioSimSpace system took 125.13923001289368 seconds
Pickling Sire system took 2.2229161262512207 seconds

(The Linux system is newer/faster than the macOS one, but the discrepancy is consistent with different macOS machines.)

While the Sire object pickling takes 1.3 times as long on macOS, the BioSimSpace object takes 50 times longer!

I'll try to figure out what attribute in the Python object is making the pickling so slow.

@lohedges lohedges added the bug label Jul 13, 2022
@lohedges
Copy link
Member Author

Interestingly, if I load the (slow) pickled BioSimSpace back in macOS, then pickle it again, the reported timing is:

Unpickling BioSimSpace system took 3.223018169403076 seconds
Pickling BioSimSpace system took 134.652765750885 seconds
Unpickling BioSimSpace system took 3.700550079345703 seconds
Pickling BioSimSpace system took 78.37889695167542 seconds

Pickling the (supposedly) same object again takes just over half of the original time.

@lohedges
Copy link
Member Author

For reference, this is using Python 3.7 for both OSes. I'll try with a different variant too.

@lohedges
Copy link
Member Author

Performance is also bad on Python 3.9:

Unpickling BioSimSpace sysem took 4.09103798866272 seconds
Pickling BioSimSpace system took 78.69318914413452 seconds
Unpickling BioSimSpace sysem took 4.513953924179077 seconds
Pickling BioSimSpace system took 70.44409394264221 seconds

Interestingly the difference in time between the first/second dump is closer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant