From dd60d63b34be0d1bd6b04e770dff45d47ad5095c Mon Sep 17 00:00:00 2001 From: Kylli Ek Date: Tue, 26 Sep 2023 16:14:56 +0300 Subject: [PATCH] Rearranging --- index.rst | 7 ++- materials/csc.md | 15 +---- materials/data/moving_data.md | 34 +++++++---- materials/supercomputer/jobs.md | 2 +- materials/supercomputer/software.md | 73 ++++++++++++++--------- materials/supercomputer/supercomputing.md | 3 +- 6 files changed, 75 insertions(+), 59 deletions(-) diff --git a/index.rst b/index.rst index cd3a3f84..8887260a 100644 --- a/index.rst +++ b/index.rst @@ -18,14 +18,14 @@ THIS MATERIAL IS WORK IN PROGRESS, do not trust anything ! ;) materials/practicalities.md materials/welcome.md materials/csc.md - materials/support.md + .. toctree:: :maxdepth: 2 :caption: Supercomputer: materials/supercomputer/supercomputing.md - materials/supercomputer/data_storage.md + materials/data/data_storage.md materials/supercomputer/software.md materials/supercomputer/webinterface.md materials/supercomputer/jobs.md @@ -33,7 +33,7 @@ THIS MATERIAL IS WORK IN PROGRESS, do not trust anything ! ;) materials/supercomputer/exercise_gdal.md materials/supercomputer/exercise_python.md materials/supercomputer/exercise_r.md - materials/supercomputer/isntallations.md + materials/supercomputer/installations.md .. toctree:: @@ -49,6 +49,7 @@ THIS MATERIAL IS WORK IN PROGRESS, do not trust anything ! ;) :caption: Wrap-up: materials/own_project.md + materials/support.md materials/wrapup.md materials/where_to_go.md diff --git a/materials/csc.md b/materials/csc.md index c5682c0b..059e89f5 100644 --- a/materials/csc.md +++ b/materials/csc.md @@ -26,17 +26,4 @@ * Teaching materials: [Library of Open Educational Resources](https://aoe.fi/) * Sending data: [Funet FileSender](https://filesender.funet.fi/) -* [CSC service catalog](https://research.csc.fi/en/service-catalog) - - - -Supercomputers -How they differ from normal computer -resources available -OS and software -resource allocation -jobs -Puhti/Mahti/LUMI -differences -Puhti software & data -Skills needed and how to get them \ No newline at end of file +* [CSC service catalog](https://research.csc.fi/en/service-catalog) \ No newline at end of file diff --git a/materials/data/moving_data.md b/materials/data/moving_data.md index a66eb7cd..0a1dd4e5 100644 --- a/materials/data/moving_data.md +++ b/materials/data/moving_data.md @@ -1,35 +1,43 @@ -# [Moving files between a local computer and Puhti](https://docs.csc.fi/data/moving/) +# Moving data -## [Puhti Web Interface](puhti.csc.fi) +## Moving files between a local computer and a supercomputer + +* [Moving files between a local computer and a supercomputer](https://docs.csc.fi/data/moving/) + +### Puhti Web Interface - Very easy, no installations needed. - For smaller amounts of data, < 10 Gb. - Upload, download, moving, creating folders. +- [Puhti Web Interface](https://puhti.csc.fi) +- [Puhti Web Interface in Docs](https://docs.csc.fi/computing/webinterface/) -## [Graphical data transfer tools](https://docs.csc.fi/data/moving/graphical_transfer/) on local computer +### Graphical data transfer tools on local computer - For example: _FileZilla_ and _WinSCP_ - For medium amounts of data, < 1 Tb. - Very easy, but installation required. +- [Graphical data transfer tools in Docs](https://docs.csc.fi/data/moving/graphical_transfer/) -## Command line tools +### Command line tools - For any amount of data, practically required if data size > 1 Tb. -### [`scp`](https://docs.csc.fi/data/moving/scp/) +#### scp - The most usual Linux tool for moving file - `scp` works even in Windows Powershell +- [`scp` in Docs](https://docs.csc.fi/data/moving/scp/) ``` # One file: scp /path/to/a_file cscusername@puhti.csc.fi:/scratch/project_xxxx/data_dir # One folder: -scp -r /path/to/data_directory cscusername@puhti.csc.fi:/scratch/project_#####/data_dir +scp -r /path/to/directory cscusername@puhti.csc.fi:/scratch/project_#####/directory ``` -### rsync -- [`rsync`](https://docs.csc.fi/data/moving/rsync/) +#### rsync + - Does not copy what is already there - Can resume a copy process which disconnected - Efficient for small files @@ -38,12 +46,13 @@ scp -r /path/to/data_directory cscusername@puhti.csc.fi:/scratch/project_#####/d - Available on Linux and Mac and WSL (windows subsystem linux) - Organization firewalls might have forbidden using rsync, especially in Valtori organizations - Windows Powershell does not have `rsync`, _MobaXterm_ has `rsync`, but it removes write permissions of copied files +- [`rsync` in Docs](https://docs.csc.fi/data/moving/rsync/) ``` # One file: rsync --info=progress2 -a /path/to/a_file cscusername@puhti.csc.fi:/scratch/project_xxxx/data_dir # One folder: -rsync --info=progress2 -a /path/to/data_directory cscusername@puhti.csc.fi:/scratch/project_xxxx/data_dir +rsync --info=progress2 -a /path/to/directory cscusername@puhti.csc.fi:/scratch/project_xxxx/directory ``` * `progress2` shows time left and percentage @@ -57,8 +66,9 @@ rsync --info=progress2 -a /path/to/data_directory cscusername@puhti.csc.fi:/scra - wget/curl if HTTP-urls avaialable - OGC APIs, STAC ? -### [wget](https://docs.csc.fi/data/moving/wget/) -- +### wget + +- [`wget` in Docs](https://docs.csc.fi/data/moving/wget/) ``` # One file: @@ -70,4 +80,4 @@ wget -r -nc ftp://ftp.aineistot.metsaan.fi/Metsamaski/Maakunta/ --cut-dirs=2 ## Moving data as part of workflow -> TODO: Example of moving data into local scratch eg from Allas +> ToDo: Example of moving data into local scratch eg from Allas diff --git a/materials/supercomputer/jobs.md b/materials/supercomputer/jobs.md index 1b0c477b..7ec40391 100644 --- a/materials/supercomputer/jobs.md +++ b/materials/supercomputer/jobs.md @@ -1,6 +1,6 @@ -# Puhti +# Jobs ![](../images/puhti_overview.png) diff --git a/materials/supercomputer/software.md b/materials/supercomputer/software.md index 441fe4f8..e55c8198 100644 --- a/materials/supercomputer/software.md +++ b/materials/supercomputer/software.md @@ -1,39 +1,56 @@ -## GIS tools available on Puhti +# GIS tools + +## GIS tools available in Puhti + +* [Ames Stereo Pipeline](https://docs.csc.fi/apps/ames-stereo.md) for processing stereo images +* [ArcGIS Python API](https://docs.csc.fi/apps/arcgis.md) +* [CloudCompare](https://docs.csc.fi/apps/cloudcompare.md) for visualizing, editing and processing poing clouds +* [FORCE](https://docs.csc.fi/apps/force.md) for mass-processing of medium-resolution satellite images +* [GDAL](https://docs.csc.fi/apps/gdal.md) for geospatial data formats +* [Geoconda](https://docs.csc.fi/apps/geoconda.md) - Python spatial analysis libraries +* [GRASS GIS](https://docs.csc.fi/apps/grass.md) General purpose GIS software family for viewing, editing and analysing geospatial data +* [LAStools](https://docs.csc.fi/apps/lastools.md) for LiDAR datasets +* [OpenDroneMap](https://docs.csc.fi/apps/opendronemap.md) for processing aerial drone imagery +* [Orfeo ToolBox](https://docs.csc.fi/apps/otb.md) for remote sensing applications +* [PCL](https://docs.csc.fi/apps/pcl.md) for 2D/3D image and point cloud processing +* [PDAL](https://docs.csc.fi/apps/pdal.md) for point cloud translations and processing +* [QGIS](https://docs.csc.fi/apps/qgis.md) General purpose GIS software family for viewing, editing and analysing geospatial data +* [R for GIS](https://docs.csc.fi/apps/r-env-for-gis.md) R spataial analysis libraries +* [SAGA GIS](https://docs.csc.fi/apps/saga-gis.md) General purpose GIS software family for viewing, editing and analysing geospatial data +* [Sen2Cor](https://docs.csc.fi/apps/sen2cor.md) for atmospheric-, terrain and cirrus correction of the Sentinel-2 products +* [Sen2mosaic](https://docs.csc.fi/apps/sen2mosaic.md) for download, preprocessing and mosaicing of Sentinel-2 products +* [SNAP](https://docs.csc.fi/apps/snap.md) for remote sensing applications +* [WhiteboxTools](https://docs.csc.fi/apps/whiteboxtools.md) an advanced geospatial data analysis platform +* [Zonation](https://docs.csc.fi/apps/zonation.md) Spatial conservation prioritization framework +* [pytorch](https://docs.csc.fi/apps/pytorch.md) for deep learning +* [tensorflow](https://docs.csc.fi/apps/tensorflow.md) for deep learning + + +## GIS tools available in Mahti -* [Ames Stereo Pipeline](ames-stereo.md) for processing stereo images -* [ArcGIS Python API](arcgis.md) -* [CloudCompare](cloudcompare.md) for visualizing, editing and processing poing clouds -* [FORCE](force.md) for mass-processing of medium-resolution satellite images -* [GDAL](gdal.md) for geospatial data formats * [Geoconda](geoconda.md) - Python spatial analysis libraries -* [GRASS GIS](grass.md) General purpose GIS software family for viewing, editing and analysing geospatial data -* [LAStools](lastools.md) for LiDAR datasets -* [OpenDroneMap](opendronemap.md) for processing aerial drone imagery -* [Orfeo ToolBox](otb.md) for remote sensing applications -* [PCL](pcl.md) for 2D/3D image and point cloud processing -* [PDAL](pdal.md) for point cloud translations and processing -* [QGIS](qgis.md) General purpose GIS software family for viewing, editing and analysing geospatial data -* [R for GIS](r-env-for-gis.md) R spataial analysis libraries -* [SAGA GIS](saga-gis.md) General purpose GIS software family for viewing, editing and analysing geospatial data -* [Sen2Cor](sen2cor.md) for atmospheric-, terrain and cirrus correction of the Sentinel-2 products -* [Sen2mosaic](sen2mosaic.md) for download, preprocessing and mosaicing of Sentinel-2 products -* [SNAP](snap.md) for remote sensing applications -* [WhiteboxTools](whiteboxtools.md) an advanced geospatial data analysis platform -* [Zonation](zonation.md) Spatial conservation prioritization framework -* Deep learning: pytorch, tensorflow -* [Puhti softwate pages](https://docs.csc.fi/apps/by_discipline/#geosciences) - -In Mahti: geoconda, PyTorch, TensorFlow - -In LUMI: -* GDAL, QGIS, GRASS GIS, PDAL, SagaGIS, PyTorch, TensorFlow +* [pytorch](https://docs.csc.fi/apps/pytorch.md) for deep learning +* [tensorflow](https://docs.csc.fi/apps/tensorflow.md) for deep learning + +## GIS tools available in LUMI + +* [GDAL](https://docs.csc.fi/apps/gdal.md) for geospatial data formats +* [GRASS GIS](https://docs.csc.fi/apps/grass.md) General purpose GIS software family for viewing, editing and analysing geospatial data +* [PDAL](https://docs.csc.fi/apps/pdal.md) for point cloud translations and processing +* [QGIS](https://docs.csc.fi/apps/qgis.md) General purpose GIS software family for viewing, editing and analysing geospatial data +* [SAGA GIS](https://docs.csc.fi/apps/saga-gis.md) General purpose GIS software family for viewing, editing and analysing geospatial data +* [pytorch](https://docs.csc.fi/apps/pytorch.md) for deep learning +* [tensorflow](https://docs.csc.fi/apps/tensorflow.md) for deep learning + +## Documentation +* [Puhti and Mahti softwate pages](https://docs.csc.fi/apps/by_discipline/#geosciences) * [LUMI software pages](https://docs.lumi-supercomputer.eu/software/) Something missing? Ask us :) servicedesk@csc.fi -## Geospatial software NOT available in Puhti +## Geospatial software NOT available in supercomputers * Servers -> these can be run in cPouta * Web map servers diff --git a/materials/supercomputer/supercomputing.md b/materials/supercomputer/supercomputing.md index e9462826..be32e403 100644 --- a/materials/supercomputer/supercomputing.md +++ b/materials/supercomputer/supercomputing.md @@ -1,4 +1,4 @@ -# Supercomputing +# Recources and possibilies ## Why use a supercomputer? @@ -65,6 +65,7 @@ Puhti - Mahti - LUMI ![](../images/lumi2.jpg) Name | CPUs | GPUs | Pre-installed GIS tools | Finnish spatial data locally | Scope | +--- | --- | --- | --- | --- | --- | Puhti | 28 000 | 240 Nvidia V100 | 20 | Yes | National | Mahti | 90 000 | 96 Nvidia A100 | 1 | No | National | LUMI | 100 000 | 10 000 AMD MI250X | 5 | No | EU |