0.5.0

Full Changelog: 0.4.0...0.5.0

0.4.0

What's Changed

• Remove ONA support by @efcaguab in #13
• Improve README by @efcaguab in #14
• Add ingest timor commit to the right branch by @efcaguab in #17
• Ingest timor metadata by @efcaguab in #18
• Update from main by @efcaguab in #19
• Create a skeleton pipeline by @efcaguab in #20
• Updating branch with main by @efcaguab in #22
• Update validate-imei with feat-airtable by @efcaguab in #27
• Feat Airtable - Closes #24 by @efcaguab in #25
• Update with main branch by @efcaguab in #28
• Adding script to generate link to edit submission by @efcaguab in #29
• Feat legacy by @langbart in #26
• Validate imei (addressing Issue #16) by @efcaguab in #23
• Feat legacy preprocessing.1 by @langbart in #34
• Feat legacy preprocessing.1 (#34) by @efcaguab in #37
• Fix environment variable problem by @efcaguab in #35
• Rebase main by @efcaguab in #38
• Merge landings by @langbart in #36
• Checking server response status before surveys retrieval by @langbart in #39
• Feat pds data by @efcaguab in #40
• Rebase to main by @efcaguab in #41
• Bug kobo limits by @efcaguab in #42
• updating species recoding by @langbart in #45
• Feat/preprocess pds trips by @langbart in #43
• Feat/validation functions by @langbart in #44
• Update branch by @efcaguab in #50
• Feat/test data by @efcaguab in #49
• Feat/merge trips by @efcaguab in #51
• Feat/format export by @efcaguab in #54
• Fixing an error in SFF landings species recoding by @langbart in #57
• Retrieve fish length parameters by @langbart in #48
• Fix bug induced by rfishbase pkg upgrade by @langbart in #66
• Feat/model revenue by @efcaguab in #61
• Adding categorical vars to the validation step by @langbart in #60
• Model overall catch by @efcaguab in #70
• Feat/catch composition by @efcaguab in #72
• Improve outliers detection by @langbart in #73
• Improve pds valid by @langbart in #68
• Fix pds step by @langbart in #74
• Swapping octopus for MOO and reordering given newer calculations by @efcaguab in #76
• Validate number of fishers by @efcaguab in #75
• tracks ingestion has to be called after trips ingestion by @langbart in #77
• Replace MOO with GZP by @langbart in #78
• Improve validations by @langbart in #79
• Feat/data report by @langbart in #71
• Nutritional values by @langbart in #80
• Fix update report by @langbart in #82
• Nutrients export by @langbart in #83
• Nutrients export by @langbart in #84
• Export with dataverse by @langbart in #52
• Validate mesh size and gleaners - tidy up validation step by @langbart in #85
• Convert mesh by @langbart in #86
• Store matched pds landings by @langbart in #88
• Improve catch data validation and flagging by @langbart in #87
• Integrate fishing trips filtering option in map generation by @langbart in #90
• Improve weight estimation by @langbart in #91
• Improve weight estimation by @langbart in #92
• Improve weight estimation by @langbart in #93
• Fix code error in validate_catch_params, some flags were hidden in gr… by @langbart in #94
• Upload data frame splitted by grids to produce leaflet maps by @langbart in #95
• Integrate gear type and time as filtering options for grid map file by @langbart in #96
• Shaping tracks_grid file in order to allow for taxa selection and fil… by @langbart in #97
• fix error in flagging fishers numbers by @langbart in #98
• Upload to dataverse edu by @langbart in #100
• Fix error in fishers number validation by @langbart in #101
• Model municipalities by @langbart in #99
• Update boats number - improve estimates model by @langbart in #102
• Improve raw landings merge by @langbart in #103
• Improve model estimates by @langbart in #104
• Integrate enumerators summary by @langbart in #105
• Fix price weight estimation by @langbart in #106
• Update reports by @langbart in #107
• Improve surveys validation by @langbart in #108
• Fix legacy landings validation by @langbart in #110
• Feat update composition estimates by @langbart in #111
• Generate Kepler.gl maps by @langbart in #112
• Feat integrate new form by @langbart in #113
• Export fuel, conservations and happiness rating by @langbart in #114
• Update validation params by @langbart in #115
• Fix price per weight tresholds by @langbart in #116
• Downweight monthly few-samples estimations by @langbart in #117
• Update nutrients values by @langbart in #118
• Export regional cpue by @langbart in #121

New Contributors

• @langbart made their first contribution in https://githu...

peskas.timor.data.pipeline 0.3.0

New features

• The preprocessing of East Timor landings is implemented in preprocess_landings()
• Added pt_nest_attachments() to group all attachment columns into a nested column containing data frames.
• Added pt_nest_species() to group all attachment columns into a nested column containing data frames.
• Added cloud_object_name() as a complement to add_version() to return the latest or an specified version of an object in an storage location.
• Added download_cloud_file() to download files from cloud storage providers.

Improvements

• Now using cloud_storage_authenticate() to internally authenticate to cloud storage instead of authenticating separately in each cloud functionjj. This simplifies authentication and ensures authentication is not attempted when credentials have been already validated.

peskas.timor.data.pipeline 0.1.0

Adds infrastructure to download survey data and upload it to cloud storage providers and implements the ingestion of East Timor landings.

New features

• The ingestion of East Timor Landings is implemented in ingest_timor_landings().
• The functions download_survey_data() and download_survey_metadata() which download data and metadata for an electronic survey hosted by kobo, kobohr, or ona.
• download_survey() can be used as a wrapper to download data and metadata in a single call.
• upload_cloud_file() can be used to upload a set of files to a cloud storage bucket. Currently only Google Cloud Services (GCS) is supported.
• add_version() is an utility function that appends date-time and sha information to a string and is used to version file names.
• get_host_url() is an utility function that gets the host url of an electronic survey provider API.

Pipeline

The data pipeline is implemented and run in GitHub Actions on a schedule.