Add converter for Luxembourg, Sweden and Denmark - FieldScapes #42

snehalchaudhari98 · 2024-06-05T21:26:30Z

Adding data converter for the sub-sampled data of the following countries.

References :

m-mohr

Thanks. I did only check the dk.py file for now, the changes requested for DK in many cases also apply to lu and se though and should be updated accordingly. I'll re-review fully afterwards.

m-mohr · 2024-06-11T12:17:49Z

fiboa_cli/datasets/dk.py

+URI = "/mnt/c/kerner-lab/fieldscapes/denmark/all_parcels.gpkg" # data seubset for FieldScapes
+
+# Unique identifier for the collection
+ID = "dk"


I think we should prefix the fieldscapes converters with fs_ as done in #43

m-mohr · 2024-06-11T12:18:13Z

fiboa_cli/datasets/dk.py

+# Unique identifier for the collection
+ID = "dk"
+# Title of the collection
+TITLE = "Field boundaries for Denmark"


Suggested change

TITLE = "Field boundaries for Denmark"

TITLE = "Field boundaries for Denmark (FieldScapes)"

m-mohr · 2024-06-11T12:18:35Z

fiboa_cli/datasets/dk.py

+# Supported protcols: HTTP(S), GCS, S3, or the local file system
+
+# data original source : https://landbrugsgeodata.fvm.dk/
+URI = "/mnt/c/kerner-lab/fieldscapes/denmark/all_parcels.gpkg" # data seubset for FieldScapes


Same as in #43: I'm not sure how to proceed with this. It's not really useful for the general public if there's no way to get the source data. Any thoughts from the fieldscapes members?

The core idea here is to get the converter code out asap, but the first goal is to get the resulting files into fiboa on source cooperative. The hope is that @eddiechoi00 will be able to help with adapting the converters to work with the original source data, and then putting the full datasets (instead of the subsets fieldscapes is focused on) in source cooperative.

Yeah, I was thinking the same. We should make it so that the source data can get converted and then there could be an option that people can enable to run on fieldscapes data.

m-mohr · 2024-06-11T12:20:12Z

fiboa_cli/datasets/dk.py

+# Add columns with constant values.
+# The key is the column name, the value is a constant value that's used for all rows.
+ADD_COLUMNS = {
+    "determination_datetime": "2021-01-01T00:00:00Z"


Is this a leftover from the template?

No, so the dataset doesn't have any date attributes for individual parcels but from the metadata, it is possible to identify the year in which the data was collected. So for simplicity, we thought we could choose the template date 01.01.XXXX .

Because in the actual fieldscapes dataset, we have the harvest and plantation season's data from the seasonal crop calendar for the given year.

m-mohr · 2024-06-11T12:20:30Z

fiboa_cli/datasets/dk.py

+    'STOETPROC' : 'support_proc',  # fiboa custom field - translated. Supported Scheme. Currently all values are NaN
+    'GB_FRADRAG' : 'deduction', # fiboa custom field - translated 
+    'MB_TYPE' : 'block_type', #fiboa custom field
+    'GB_AREAL': 'eligible_area', # fiboa core field i.e. eligible area


not a core field

Yep, will fix this.

m-mohr · 2024-06-11T12:21:00Z

fiboa_cli/datasets/dk.py

+    'MARKBLOKNR' : 'id', # fiboa core field , TODO MARKBLOKNR-id conversion is needed.     
+    'GEOMETRISK' : 'area',#fiboa custom field i.e total area = eligible area + ineligible area
+    'TARAAREAL': 'ineligible_area', #fiboa custom field
+    'STOETPROC' : 'support_proc',  # fiboa custom field - translated. Supported Scheme. Currently all values are NaN


If all values are NaN, it whould probably be removed for now?

m-mohr · 2024-06-11T12:21:18Z

fiboa_cli/datasets/dk.py

+# You also need to list any column that you may have added in the MIGRATION function (see below).
+COLUMNS = {    
+    'MARKBLOKNR' : 'id', # fiboa core field , TODO MARKBLOKNR-id conversion is needed.     
+    'GEOMETRISK' : 'area',#fiboa custom field i.e total area = eligible area + ineligible area


That's a core field

Yes, I changed the description in the end thus there was a mixup with the actual area and eligible area.

m-mohr · 2024-06-11T12:21:40Z

fiboa_cli/datasets/dk.py

+# Map original column names to fiboa property names
+# You also need to list any column that you may have added in the MIGRATION function (see below).
+COLUMNS = {    
+    'MARKBLOKNR' : 'id', # fiboa core field , TODO MARKBLOKNR-id conversion is needed.     


There's an open todo here.

m-mohr · 2024-06-11T12:22:31Z

fiboa_cli/datasets/dk.py

+            'type': 'float'
+        },
+        'block_type': {
+            'type': 'string'


This should probably be an enum?

m-mohr · 2024-06-11T12:23:11Z

fiboa_cli/datasets/dk.py

+# Schemas for the fields that are not defined in fiboa
+# Keys must be the values from the COLUMNS dict, not the keys
+MISSING_SCHEMAS = {
+    "properties": {


Should some of the float fields be have a minimum: 0 constraint?

Yes, good suggestion. We'll add the required constraints to the properties.

m-mohr · 2024-06-17T13:46:44Z

Please rebase the PR to the latest CLI version, including renaming the URI constant to SOURCES and renaming the parameter cache_file to cache. I tried to push it upstream, but you didn't permit changes to this PR so you need to do these changes yourself. Thanks.

…and Sweden)

m-mohr self-requested a review June 5, 2024 21:37

snehalchaudhari98 changed the title ~~Add converter for Luxembourg - FieldScapes~~ Add converter for Luxembourg and Sweden - FieldScapes Jun 6, 2024

snehalchaudhari98 changed the title ~~Add converter for Luxembourg and Sweden - FieldScapes~~ Add converter for Luxembourg, Sweden and Denmark - FieldScapes Jun 10, 2024

m-mohr requested changes Jun 11, 2024

View reviewed changes

m-mohr assigned snehalchaudhari98 Jun 11, 2024

snehalchaudhari98 force-pushed the fieldscapes branch from 98ad213 to 5048c3b Compare June 22, 2024 01:01

snehalchaudhari98 requested a review from m-mohr June 22, 2024 01:03

m-mohr marked this pull request as draft July 10, 2024 15:27

snehalchaudhari1 added 4 commits July 14, 2024 23:13

Add converter for Luxembourg - FieldScapes

e9ae732

Add converter for Sweden - FieldScapes

f89750a

Add converter for Denmark

2c77aa7

upating template for the fielscapes data subset (Denmark, Luxembourg …

7ebfd72

…and Sweden)

snehalchaudhari98 force-pushed the fieldscapes branch from d7736c3 to 7ebfd72 Compare July 15, 2024 07:23

updating date/year and custom licenses

e4904ee

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add converter for Luxembourg, Sweden and Denmark - FieldScapes #42

Add converter for Luxembourg, Sweden and Denmark - FieldScapes #42

snehalchaudhari98 commented Jun 5, 2024 •

edited

Loading

m-mohr left a comment

m-mohr Jun 11, 2024

m-mohr Jun 11, 2024

m-mohr Jun 11, 2024

cholmes Jun 19, 2024

m-mohr Jun 19, 2024

m-mohr Jun 11, 2024

snehalchaudhari98 Jun 21, 2024

m-mohr Jun 11, 2024

snehalchaudhari98 Jun 21, 2024

m-mohr Jun 11, 2024

m-mohr Jun 11, 2024

snehalchaudhari98 Jun 21, 2024

m-mohr Jun 11, 2024

m-mohr Jun 11, 2024

m-mohr Jun 11, 2024

snehalchaudhari98 Jun 21, 2024

m-mohr commented Jun 17, 2024

	TITLE = "Field boundaries for Denmark"
	TITLE = "Field boundaries for Denmark (FieldScapes)"

Add converter for Luxembourg, Sweden and Denmark - FieldScapes #42

Are you sure you want to change the base?

Add converter for Luxembourg, Sweden and Denmark - FieldScapes #42

Conversation

snehalchaudhari98 commented Jun 5, 2024 • edited Loading

m-mohr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m-mohr commented Jun 17, 2024

snehalchaudhari98 commented Jun 5, 2024 •

edited

Loading