Dtree constructor #221

aladinor · 2024-10-27T20:06:36Z

refactored the code to use the DataTree.from_dict constructor to create the DataTree structure instead of manually setting each node during the creation process.

Closes DataTree.from_dict constructor to create the DataTree structure instead of manually setting each node #222
Tests added
Changes are documented in history.md

xradar/io/backends/common.py

kmuehlbauer · 2024-10-28T13:39:19Z

@aladinor I very much agree to your proposal of using the more intuitive from_dict approach. I've incorporated your changes from #220 over in #218. It works!

aladinor · 2024-10-28T14:37:02Z

Thanks, @kmuehlbauer, for implementing this in #223. I was working during the weekend doing some tests, and I had some questions before submitting other changes. However, I think I can add them under the new head node. On the other hand, I realized that some backends might not contain the georeference and calibration groups. I understand they might be optional, but compared to cfradial, which includes all of them, should we include them as empty nodes/groups to make them similar in the data model/structure?

kmuehlbauer · 2024-10-28T14:57:22Z

On the other hand, I realized that some backends might not contain the georeference and calibration groups. I understand they might be optional, but compared to cfradial, which includes all of them, should we include them as empty nodes/groups to make them similar in the data model/structure?

@aladinor Thanks! Yes, I agree. Citing from the FM301 docs for radar_parameters, radar_calibration groups:

This group may be omitted from the file if no radar parameters are to be stored.

So we are on safe ground, to have these subgroups available, even when they are empty.

aladinor · 2024-10-29T16:02:19Z

@kmuehlbauer I did some refactoring for keeping all radar_parameters, georeferencing, and calibration subgroups. I am working to make sure all backends look similar. Let me know what you think about this to "copy" this into the others. For now, Iris and Datamet backends are ready. Should I submit all of them in just one PR?

This is how a cfradial dtree will look like

<xarray.DataTree>
Group: /
│   Dimensions:              (sweep: 9)
│   Dimensions without coordinates: sweep
│   Data variables:
│       volume_number        int32 4B ...
│       platform_type        |S32 32B ...
│       primary_axis         |S32 32B ...
│       status_str           |S1 1B ...
│       instrument_type      |S32 32B ...
│       time_coverage_start  |S32 32B ...
│       time_coverage_end    |S32 32B ...
│       latitude             float64 8B ...
│       longitude            float64 8B ...
│       altitude             float64 8B ...
│       sweep_group_name     (sweep) <U7 252B 'sweep_0' 'sweep_1' ... 'sweep_8'
│       sweep_fixed_angle    (sweep) float32 36B ...
│   Attributes: (12/13)
│       Conventions:         CF/Radial instrument_parameters radar_parameters rad...
│       version:             1.2
│       title:               TIMREX
│       institution:         
│       references:          
│       source:              
│       ...                  ...
│       comment:             
│       instrument_name:     SPOLRVP8
│       site_name:           
│       scan_name:           
│       scan_id:             0
│       platform_is_mobile:  false
├── Group: /sweep_0
│       Dimensions:                    (azimuth: 483, range: 996)
│       Coordinates:
│           time                       (azimuth) datetime64[ns] 4kB 2008-06-04T00:15:...
│           elevation                  (azimuth) float32 2kB ...
│           latitude                   float64 8B ...
│           longitude                  float64 8B ...
│           altitude                   float64 8B ...
│         * azimuth                    (azimuth) float32 2kB 0.0 0.75 ... 358.5 359.2
│         * range                      (range) float32 4kB 150.0 300.0 ... 1.494e+05
│       Data variables: (12/18)
│           sweep_number               int32 4B ...
│           sweep_mode                 <U20 80B 'azimuth_surveillance'
│           prt_mode                   |S32 32B ...
│           follow_mode                |S32 32B ...
│           sweep_fixed_angle          float32 4B ...
│           pulse_width                (azimuth) timedelta64[ns] 4kB ...
│           ...                         ...
│           r_calib_index              (azimuth) int8 483B ...
│           measured_transmit_power_h  (azimuth) float32 2kB ...
│           measured_transmit_power_v  (azimuth) float32 2kB ...
│           scan_rate                  (azimuth) float32 2kB ...
│           DBZ                        (azimuth, range) float32 2MB ...
│           VR                         (azimuth, range) float32 2MB ...
├── Group: /sweep_1
│       Dimensions:                    (azimuth: 483, range: 996)
│       Coordinates:
│           time                       (azimuth) datetime64[ns] 4kB 2008-06-04T00:16:...
│           elevation                  (azimuth) float32 2kB ...
│           latitude                   float64 8B ...
│           longitude                  float64 8B ...
│           altitude                   float64 8B ...
│         * azimuth                    (azimuth) float32 2kB 0.0 0.75 ... 358.5 359.2
│         * range                      (range) float32 4kB 150.0 300.0 ... 1.494e+05
│       Data variables: (12/18)
│           sweep_number               int32 4B ...
│           sweep_mode                 <U20 80B 'azimuth_surveillance'
│           prt_mode                   |S32 32B ...
│           follow_mode                |S32 32B ...
│           sweep_fixed_angle          float32 4B ...
│           pulse_width                (azimuth) timedelta64[ns] 4kB ...
│           ...                         ...
│           r_calib_index              (azimuth) int8 483B ...
│           measured_transmit_power_h  (azimuth) float32 2kB ...
│           measured_transmit_power_v  (azimuth) float32 2kB ...
│           scan_rate                  (azimuth) float32 2kB ...
│           DBZ                        (azimuth, range) float32 2MB ...
│           VR                         (azimuth, range) float32 2MB ...
├── Group: /sweep_2
│       Dimensions:                    (azimuth: 482, range: 996)
│       Coordinates:
│           time                       (azimuth) datetime64[ns] 4kB 2008-06-04T00:17:...
│           elevation                  (azimuth) float32 2kB ...
│           latitude                   float64 8B ...
│           longitude                  float64 8B ...
│           altitude                   float64 8B ...
│         * azimuth                    (azimuth) float32 2kB 0.0 0.75 ... 358.5 359.2
│         * range                      (range) float32 4kB 150.0 300.0 ... 1.494e+05
│       Data variables: (12/18)
│           sweep_number               int32 4B ...
│           sweep_mode                 <U20 80B 'azimuth_surveillance'
│           prt_mode                   |S32 32B ...
│           follow_mode                |S32 32B ...
│           sweep_fixed_angle          float32 4B ...
│           pulse_width                (azimuth) timedelta64[ns] 4kB ...
│           ...                         ...
│           r_calib_index              (azimuth) int8 482B ...
│           measured_transmit_power_h  (azimuth) float32 2kB ...
│           measured_transmit_power_v  (azimuth) float32 2kB ...
│           scan_rate                  (azimuth) float32 2kB ...
│           DBZ                        (azimuth, range) float32 2MB ...
│           VR                         (azimuth, range) float32 2MB ...

├── Group: /radar_parameters
│       Dimensions:                   ()
│       Data variables:
│           radar_beam_width_h        float32 4B ...
│           radar_antenna_gain_v      float32 4B ...
│           radar_antenna_gain_h      float32 4B ...
│           radar_beam_width_v        float32 4B ...
│           radar_receiver_bandwidth  float32 4B ...
├── Group: /georeferencing_correction
│       Dimensions:                            ()
│       Data variables: (12/16)
│           elevation_correction               float32 4B ...
│           heading_correction                 float32 4B ...
│           eastward_ground_speed_correction   float32 4B ...
│           latitude_correction                float32 4B ...
│           pitch_correction                   float32 4B ...
│           longitude_correction               float32 4B ...
│           ...                                 ...
│           vertical_velocity_correction       float32 4B ...
│           tilt_correction                    float32 4B ...
│           northward_ground_speed_correction  float32 4B ...
│           pressure_altitude_correction       float32 4B ...
│           roll_correction                    float32 4B ...
│           radar_altitude_correction          float32 4B ...
└── Group: /radar_calibration
        Dimensions:                   ()
        Data variables: (12/45)
            time                      |S32 32B ...
            pulse_width               timedelta64[ns] 8B ...
            xmit_power_h              float32 4B ...
            xmit_power_v              float32 4B ...
            two_way_waveguide_loss_h  float32 4B ...
            two_way_waveguide_loss_v  float32 4B ...
            ...                        ...
            test_power_h              float32 4B ...
            test_power_v              float32 4B ...
            receiver_slope_hc         float32 4B ...
            receiver_slope_vc         float32 4B ...
            receiver_slope_hx         float32 4B ...
            receiver_slope_vx         float32 4B ...

codecov · 2024-10-29T16:02:34Z

Codecov Report

Attention: Patch coverage is 94.55782% with 8 lines in your changes missing coverage. Please review.

Project coverage is 92.80%. Comparing base (b88218e) to head (9c626c4).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
xradar/io/backends/common.py	92.15%	4 Missing ⚠️
xradar/io/backends/hpl.py	72.72%	3 Missing ⚠️
xradar/io/backends/metek.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #221   +/-   ##
=======================================
  Coverage   92.79%   92.80%           
=======================================
  Files          26       26           
  Lines        4957     5003   +46     
=======================================
+ Hits         4600     4643   +43     
- Misses        357      360    +3

Flag	Coverage Δ
notebooktests	`79.03% <88.43%> (+0.09%)`	⬆️
unittests	`91.22% <91.15%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kmuehlbauer · 2024-10-29T16:07:58Z

@aladinor Yes, make this a bulk refactor. No need to separate I think. But do as you prefer.

The repr looks great!

syedhamidali · 2024-10-29T20:23:25Z

Hi @aladinor, @kmuehlbauer I think we should cover the pytests for the below given functions with this PR in the iris.py module to increase codecov. Currently, it is at 85%.

read_from_record
decode_data
get_sweep
array_from_file

syedhamidali · 2024-10-30T00:40:34Z

I added these above-mentioned pytests in #226

xradar/io/backends/iris.py

kmuehlbauer · 2024-10-30T07:10:54Z

@aladinor I've found the issue with to_zarr. "sweep_group_name" was missing the fixup of the old dtype (int) to the new dtype (str). Not sure what's going on with the GH Actions, though. Maybe a temporal disturbance...

kmuehlbauer · 2024-10-30T07:30:21Z

OK, seems like GH Actions has had an incident (https://www.githubstatus.com/)

kmuehlbauer

This is looking really good. Only the new entries in the radar_calibration subgroup need fixing.

Co-authored-by: Kai Mühlbauer <[email protected]>

xradar/model.py

Co-authored-by: Kai Mühlbauer <[email protected]>

xradar/model.py

kmuehlbauer · 2024-11-02T15:39:59Z

@aladinor Sorry, I've bugged a paranthesis ;-)

kmuehlbauer · 2024-11-02T15:43:17Z

@aladinor Huge thanks to you! I think we can live with the slight inconsistencies in the model. We can handle that later.

Please add entry in history.md, I'm going to merge if you are ready!

syedhamidali · 2024-11-02T15:50:41Z

Great job @aladinor! Thank you very much!

aladinor · 2024-11-02T15:53:27Z

Thank you @kmuehlbauer and @syedhamidali for your help. Excited to have xradar 0.8.0! soon

kmuehlbauer · 2024-11-02T16:13:09Z

😍 @aladinor 💯

* refactoring dtree node apend using _attach_sweep_groups function * refactoring iris datatree to use from_dict constructor * running pre-commit hook * refactoring calibration subgroup * dropping common attributes present in all sweeps * refactoring to use from_dict datatree constructor * refactoring loop by iterating only over sweeps * fixing failing test with attributes * addding comments * Update xradar/io/backends/iris.py * fix lint * fixing typo * refactoring datamet backend to use from_dict datatree construnctor * fixing issue with group test since datamet now has root, georefernce, radar_parameter, and calib groups * adding some calibration variables from Furuno datasets * refactoring furuno backeds to use from_dict contructor * getting ride of unncessary attributes in the root dataset * updating global attrributes * adding gamic radar calibration parameters * refactoring gamic backend to use from_dict datatree constructor * refactoring code. moving _get_required_root_dataset, _get_subgroup, to common.py file * using dtree.match to iterate over sweep nodes * refactoring _get_subgroups and _get_required_root funcition * fixing typo * adding _get_radar_calibration fuction * refactoring and moving _get_requiered_root_Dataset, _get_subgroup, and _get_calibration function to commo.py file * refactoring and moving _get_calibration function to commo.py file * refactoring hpl backends to use from_dict contructor and adding sweeps * using radar_calibration group from common.py in Datamet backend * using radar calibration subgrup fucntion from common.py in iris backend * allowing metek to read lines from gzip file when using context manager * refactoring _get_required_root function to fit with metek backend * refactoring metek backend to use from_dict constructor * fixing nexrad test that support the radar_calibration, georeference, and parameter group * refactoring nexrad backend to use from_dict constructor * reformating odim test to include calibration, georeference, and radar parameter groups in odim backend * refactoring odim backend to use from_dict constructor * refactoring odim export to iterate only over sweeps * refactoring rainbow test to only iterate over sweeps * refactoring rainbow backend to use from_dict constructor * fixin typo * adding test for _get_requered_root_group, _get_subgroup, and _get_radar_calibration functions * Update xradar/model.py * Update history.md

aladinor commented Oct 27, 2024

View reviewed changes

xradar/io/backends/common.py Outdated Show resolved Hide resolved

kmuehlbauer mentioned this pull request Oct 28, 2024

xradar release 0.8.0 #223

Closed

2 tasks

aladinor closed this Oct 28, 2024

aladinor force-pushed the dtree-constructor branch from 14edd9f to 617bd34 Compare October 28, 2024 14:56

aladinor added 4 commits October 29, 2024 08:16

refactoring dtree node apend using _attach_sweep_groups function

a6bfaa0

refactoring iris datatree to use from_dict constructor

e71d046

running pre-commit hook

20d29a3

refactoring calibration subgroup

e4c8c9c

aladinor reopened this Oct 29, 2024

Merge branch 'main' into dtree-constructor

a086389

aladinor added 5 commits October 29, 2024 11:10

dropping common attributes present in all sweeps

8d694db

refactoring to use from_dict datatree constructor

06d37a7

refactoring loop by iterating only over sweeps

e45c673

fixing failing test with attributes

435d281

addding comments

ca659c5

kmuehlbauer reviewed Oct 30, 2024

View reviewed changes

xradar/io/backends/iris.py Outdated Show resolved Hide resolved

kmuehlbauer and others added 2 commits October 30, 2024 07:58

Update xradar/io/backends/iris.py

9cbbca9

fix lint

62e9c62

kmuehlbauer closed this Oct 30, 2024

kmuehlbauer reopened this Oct 30, 2024

kmuehlbauer requested changes Nov 2, 2024

View reviewed changes

Update xradar/model.py

8abc2e7

Co-authored-by: Kai Mühlbauer <[email protected]>

syedhamidali reviewed Nov 2, 2024

View reviewed changes

xradar/model.py Outdated Show resolved Hide resolved

aladinor mentioned this pull request Nov 2, 2024

Empty radar_parameters and calibrations, and georeferencing in several backends #232

Open

aladinor self-assigned this Nov 2, 2024

aladinor added 2 commits November 2, 2024 10:03

Update model.py

9fd48e0

Update model.py

e6be32b

kmuehlbauer reviewed Nov 2, 2024

View reviewed changes

xradar/model.py Outdated Show resolved Hide resolved

kmuehlbauer reviewed Nov 2, 2024

View reviewed changes

xradar/model.py Outdated Show resolved Hide resolved

kmuehlbauer reviewed Nov 2, 2024

View reviewed changes

xradar/model.py Outdated Show resolved Hide resolved

Update model.py

971ea59

kmuehlbauer reviewed Nov 2, 2024

View reviewed changes

xradar/model.py Outdated Show resolved Hide resolved

aladinor and others added 3 commits November 2, 2024 10:34

Update model.py

da470d2

Update xradar/model.py

3cf21cd

Co-authored-by: Kai Mühlbauer <[email protected]>

Update model.py

064662a

kmuehlbauer reviewed Nov 2, 2024

View reviewed changes

xradar/model.py Outdated Show resolved Hide resolved

Update xradar/model.py

468f502

Update history.md

9c626c4

aladinor requested review from kmuehlbauer and syedhamidali November 2, 2024 15:51

kmuehlbauer approved these changes Nov 2, 2024

View reviewed changes

syedhamidali approved these changes Nov 2, 2024

View reviewed changes

kmuehlbauer merged commit 794c606 into openradar:main Nov 2, 2024
12 checks passed

aladinor deleted the dtree-constructor branch November 8, 2024 17:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dtree constructor #221

Dtree constructor #221

aladinor commented Oct 27, 2024 •

edited

Loading

kmuehlbauer commented Oct 28, 2024

aladinor commented Oct 28, 2024 •

edited

Loading

kmuehlbauer commented Oct 28, 2024

aladinor commented Oct 29, 2024 •

edited

Loading

codecov bot commented Oct 29, 2024 •

edited

Loading

kmuehlbauer commented Oct 29, 2024

syedhamidali commented Oct 29, 2024 •

edited

Loading

syedhamidali commented Oct 30, 2024

kmuehlbauer commented Oct 30, 2024

kmuehlbauer commented Oct 30, 2024 •

edited

Loading

kmuehlbauer left a comment

kmuehlbauer commented Nov 2, 2024

kmuehlbauer commented Nov 2, 2024

syedhamidali commented Nov 2, 2024

aladinor commented Nov 2, 2024

kmuehlbauer commented Nov 2, 2024

Dtree constructor #221

Dtree constructor #221

Conversation

aladinor commented Oct 27, 2024 • edited Loading

kmuehlbauer commented Oct 28, 2024

aladinor commented Oct 28, 2024 • edited Loading

kmuehlbauer commented Oct 28, 2024

aladinor commented Oct 29, 2024 • edited Loading

codecov bot commented Oct 29, 2024 • edited Loading

Codecov Report

kmuehlbauer commented Oct 29, 2024

syedhamidali commented Oct 29, 2024 • edited Loading

syedhamidali commented Oct 30, 2024

kmuehlbauer commented Oct 30, 2024

kmuehlbauer commented Oct 30, 2024 • edited Loading

kmuehlbauer left a comment

Choose a reason for hiding this comment

kmuehlbauer commented Nov 2, 2024

kmuehlbauer commented Nov 2, 2024

syedhamidali commented Nov 2, 2024

aladinor commented Nov 2, 2024

kmuehlbauer commented Nov 2, 2024

aladinor commented Oct 27, 2024 •

edited

Loading

aladinor commented Oct 28, 2024 •

edited

Loading

aladinor commented Oct 29, 2024 •

edited

Loading

codecov bot commented Oct 29, 2024 •

edited

Loading

syedhamidali commented Oct 29, 2024 •

edited

Loading

kmuehlbauer commented Oct 30, 2024 •

edited

Loading