Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use shapely objects to store geometries. #941

Merged
merged 4 commits into from
Nov 5, 2024

Conversation

groutr
Copy link
Contributor

@groutr groutr commented Oct 29, 2024

process_schism_fim uses a lot of memory. I tested with the pacific domain and discovered that loading the shapefile into memory takes almost 3GB of RAM.

Fiona represents geometries as dictionaries of objects. When storing large geometries such as those in the pacific domain, it becomes very inefficient. Fortunately, we can store geometries as shapely objects, which only use a fraction of the memory.

Before/after memory usage of get_mask_geoms_by_group

atlgulf:
increment: 2849.89 MiB   # old
increment: 604.03 MiB   # new

pacific:
increment: 2882.93 MiB   # old
increment: 515.28 MiB    # new

hi:
increment: 0.00 MiB
increment: 0.00 MiB

prvi:
increment: 0.00 MiB
increment: 0.01 MiB

Other misc fixes are removing two ineffective gc calls and an unnecessary enumeration.

Copy link
Collaborator

@shawncrawley shawncrawley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks for doing this!

@shawncrawley shawncrawley merged commit 734ffe9 into NOAA-OWP:ti Nov 5, 2024
1 check passed
@nickchadwick-noaa nickchadwick-noaa added this to the V2.1.8 milestone Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants