Skip to content

Commit

Permalink
Merge pull request #12 from rolfsimoes/main
Browse files Browse the repository at this point in the history 
Fix documentation
  • Loading branch information
@rolfsimoes
rolfsimoes authored Jan 16, 2025
2 parents aee1e1d + fce2bc8 commit 040c1b7
Show file tree
Hide file tree
Showing 25 changed files with 6,617 additions and 522 deletions.
165 changes: 153 additions & 12 deletions .test.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@
"outputs": [],
"source": [
"# TODO: call this before ds.storage_size\n",
"ds._update_metadata()"
"ds.update_metadata()"
]
},
{
Expand Down Expand Up @@ -5099,13 +5099,12 @@
"outputs": [],
"source": [
"from zen import Zenodo\n",
"from zen import LocalFiles\n",
"import os"
"from zen import LocalFiles"
]
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -5114,7 +5113,7 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -5123,38 +5122,180 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 12,
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"<Files: 2 file(s)>"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"local_files = ['examples/file1.csv', 'examples/file2.csv']\n",
"\n",
"dataset_path = 'examples/dataset.json'\n",
"\n",
"ds = LocalFiles(local_files, dataset_path=dataset_path)\n",
"ds.set_deposition(zen, create_if_not_exists=True)\n",
"...\n",
"ds.save()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<Deposition: {'id': 112377, 'title': '', 'state': 'unsubmitted'}>"
"<Deposition: {'id': 112399, 'title': '', 'state': 'unsubmitted'}>"
]
},
"execution_count": 11,
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dep"
"ds.deposition"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'filename': 'file1.csv',\n",
" 'links': {'download': 'examples/file1.csv'},\n",
" 'properties': {},\n",
" 'filesize': 34308,\n",
" 'filedate': '2023-11-17T17:05:10.551491'},\n",
" {'filename': 'file2.csv',\n",
" 'links': {'download': 'examples/file2.csv'},\n",
" 'properties': {},\n",
" 'filesize': 34249,\n",
" 'filedate': '2023-11-17T17:05:10.551491'}]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds.data"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<Files: 2 file(s)>"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"local_files = ['examples/file2.csv', 'examples/file3.csv']\n",
"\n",
"dataset_path = 'examples/dataset.json'\n",
"\n",
"ds = LocalFiles(local_files, dataset_path=dataset_path)\n",
"ds.set_deposition(zen, create_if_not_exists=True)\n",
"ds.save()\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'filename': 'file2.csv',\n",
" 'links': {'download': 'examples/file2.csv'},\n",
" 'properties': {},\n",
" 'filesize': 34249,\n",
" 'filedate': '2023-11-17T17:05:10.551491'},\n",
" {'filename': 'file3.csv',\n",
" 'links': {'download': 'examples/file3.csv'},\n",
" 'properties': {},\n",
" 'filesize': 34182,\n",
" 'filedate': '2023-11-17T17:05:10.551491'}]"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds.data"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[<Deposition: {'id': 112399, 'title': '', 'state': 'unsubmitted'}>,\n",
" <Deposition: {'id': 112397, 'title': '', 'state': 'unsubmitted'}>,\n",
" <Deposition: {'id': 112377, 'title': '', 'state': 'unsubmitted'}>,\n",
" <Deposition: {'id': 43034, 'title': '', 'state': 'unsubmitted'}>,\n",
" <Deposition: {'id': 31944, 'title': '', 'state': 'unsubmitted'}>,\n",
" <Deposition: {'id': 314, 'title': 'Example dataset', 'state': 'unsubmitted'}>,\n",
" <Deposition: {'id': 51, 'title': 'My first dataset', 'state': 'unsubmitted'}>]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"zen.depositions.list()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"ename": "ValueError",
"evalue": "Invalid deposition assignment. The saved deposition (112399) differs from the provided one (112397). Please, consider creating a new dataset.",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[21], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mds\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mset_deposition\u001b[49m\u001b[43m(\u001b[49m\u001b[43mzen\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdeposition\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m112397\u001b[39;49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/gh/zen/zen/dataset.py:890\u001b[0m, in \u001b[0;36mLocalFiles.set_deposition\u001b[0;34m(self, api, metadata, deposition, create_if_not_exists)\u001b[0m\n\u001b[1;32m 888\u001b[0m saved_deposition_id \u001b[38;5;241m=\u001b[39m dataset\u001b[38;5;241m.\u001b[39mzenodo[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mid\u001b[39m\u001b[38;5;124m'\u001b[39m]\n\u001b[1;32m 889\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m saved_deposition_id \u001b[38;5;241m!=\u001b[39m deposition\u001b[38;5;241m.\u001b[39mid:\n\u001b[0;32m--> 890\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mInvalid deposition assignment. The saved deposition \u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;241m+\u001b[39m\n\u001b[1;32m 891\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m(\u001b[39m\u001b[38;5;132;01m{\u001b[39;00msaved_deposition_id\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m) differs from the provided one \u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;241m+\u001b[39m\n\u001b[1;32m 892\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m(\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mdeposition\u001b[38;5;241m.\u001b[39mid\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m). Please, consider creating a new dataset.\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 893\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_deposition \u001b[38;5;241m=\u001b[39m deposition\n\u001b[1;32m 894\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\n",
"\u001b[0;31mValueError\u001b[0m: Invalid deposition assignment. The saved deposition (112399) differs from the provided one (112397). Please, consider creating a new dataset."
]
}
],
"source": [
"ds.set_deposition(zen, deposition=112397)"
]
}
],
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ You can update these files to modify or expand the documentation.
4. **Commit and push the changes**:
Add the changes and commit using the `docs` type in the commit message format:
```bash
git add sphinx
git add docs
git commit -m "docs: update deployed documentation"
git push
Expand Down
2 changes: 1 addition & 1 deletion deploy.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ rm -fr docs/*
cp -r sphinx/_build/html/* docs/
[ ! -f "docs/.nojekyll" ] && touch "docs/.nojekyll"
git add docs
git commit -m "Documentation deployment"
git commit -m "docs: update deployed documentation"
3 changes: 0 additions & 3 deletions docs/_sources/index.rst.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,12 @@ Indices and tables
* :ref:`modindex`
* :ref:`search`


Background & Motivation
-----------------------
The **FAIR** principles stand for improving the **F**\indability, **A**\ccessibility, **I**\nteroperability, and **R**\euse of data by machines with minimal or no human intervention and were introduced by
`Willkinson et al. (2016) <https://doi.org/10.1038/sdata.2016.18>`_ to deal with increasing data volume and complexity. Environmental data, driven by Earth Observation with satellite imagery, has already faced that challenge as many satellite image archives, such as those from NASA Landsat mission and ESA Copernicus missions become open, providing PB-scale data volumes for research and other exploration.

Within the Open-Earth-Monitor Cyberinfrastructure project, a comprehensive survey was conducted where both environmental data users and producers were targeted to understand how they are familiar with FAIR principles in their data management. The survey results revealed that FAIR principles are considered important for geospatial data management but still lack implementation by many geospatial data users and providers. One of the reasons for that is the lack of resources and tools to make data more FAIR. Furthermore, both groups find it most important for data to be online findable and open. As a reflection to this feedback, the `zen` library has been introduced as an open tool to efficiently expose and manage data at the Zenodo repository, making such datasets findable.


Acknowledgements & Funding
--------------------------

Expand Down
Loading

0 comments on commit 040c1b7

Please sign in to comment.