Organise finetune code and docs (#301)

* Move finetune scripts into respective folders

* Move readme files into docs directory

* Add small introduction and link to code to finetune docs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Move finetuning on embeddings into finetuning section

* Add torchgeo to environment

* Document pythonpath requirement for finetune scripts

* Also add path docs to regression

* Add walkthrough videos to docs index page.

* Run finetune scripts as modules

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

docs/_toc.yml

@@ -26,8 +26,16 @@ parts:
       file: tutorials/visualize-embeddings
     - title: Clay MAE reconstruction
       file: tutorials/reconstruction
+- caption: Finetune examples
+  chapters:
+    - title: Segmentation using Chesapeake
+      file: finetune/segment
+    - title: Classification using Eurosat
+      file: finetune/classify
+    - title: Regression using Biomasters
+      file: finetune/regression
     - title: Finetunig on an embeddings database
-      file: tutorials/finetune-on-embeddings
+      file: finetune/finetune-on-embeddings
 - caption: About Clay
   chapters:
     - title: GitHub

finetune/classify/README.md → docs/finetune/classify.md

@@ -1,22 +1,30 @@
-# Classifier
+# Classification head fine tuning
+
+We have built an example for training a classification head on top of
+the class token embeddings from the frozen Clay encoder.
+
+All the code for this example can be found in the
+[classify finetuning folder](https://github.com/Clay-foundation/model/blob/main/finetune/classify)
+of this repository.
+
+## Classifier
 
 The `Classifier` class is designed for classification tasks, utilizing the Clay Encoder for feature extraction and adding a classification head on top of it.
 
-## Parameters
+### Parameters
 
 - `num_classes (int, optional)`: The number of classes for classification. Defaults to 10.
 - `ckpt_path (str, optional)`: Path to the Clay MAE pretrained model checkpoint. Defaults to None.
 
-## Example
+### Example
 
 In this example, we will use the `Classifier` class to classify images from the [EuroSAT MS dataset](https://github.com/phelber/EuroSAT). The implementation includes data preprocessing, data loading, and model training workflow using [PyTorch Lightning](https://lightning.ai/) & [TorchGeo](https://github.com/microsoft/torchgeo).
 
 In this example we freeze the Clay encoder and only train a very simple 2 layer MLP head for classification. The MLP head recieves as input the Clay class token embedding, which already contains the essence of the image as seen by Clay. The model for classification can then be kept very simple while still guaranteeing high quality results.
 
 Notice that the EuroSAT dataset comes without date stamps or location information. The Clay model requires encoded versions of a date stamp and a latitude and longitude information. These values can be set to zero if they are not available, which is what we are doing in the datamodule script.
-## Dataset
 
-### Citation
+## Dataset citation
 
 If you have used the EuroSAT dataset, please cite the following papers:
 
@@ -95,9 +103,13 @@ The model can be run via LightningCLI using configurations in `finetune/classify
     ```
 
 4. Train the classification head:
-    ```bash
-    python classify.py fit --config configs/classify_eurosat.yaml
-    ```
+
+To ensure that the imports work properly, ensure that the root of
+the repository is in the python path before running the script.
+
+```bash
+python -m finetune.classify.classify fit --config configs/classify_eurosat.yaml
+```
 
 ## Acknowledgments
 

finetune/regression/README.md → docs/finetune/regression.md

@@ -1,4 +1,14 @@
+# Regression head fine tuning
+
+We have built an example for training a regression head on top of
+feature maps extracted from the frozen Clay encoder.
+
+All the code for this example can be found in the
+[regression finetuning folder](https://github.com/Clay-foundation/model/blob/main/finetune/regression)
+of this repository.
+
 ## Download data
+
 The data comes as multifile zip, it can be downloaded from the
 [BioMassters](https://huggingface.co/datasets/nascetti-a/BioMassters/)
 huggingface repository. Grab a coffee, this is about 250GB in size.
@@ -175,3 +185,13 @@ python finetune/regression/preprocess_data.py \
   --sample=1 \
   --overwrite
 ```
+
+### Train the model
+
+To ensure that the imports work properly, ensure that the root of
+the repository is in the python path before running the script.
+
+
+```bash
+python -m finetune.regression.regression fit --config configs/regression_biomasters.yaml
+```

finetune/segment/README.md → docs/finetune/segment.md

@@ -1,24 +1,33 @@
-# Segmentor
+# Segmentation head fine tuning
+
+We have built an example for training a segmentation head on top of
+feature maps extracted from the frozen Clay encoder.
+
+All the code for this example can be found in the
+[segment finetuning folder](https://github.com/Clay-foundation/model/blob/main/finetune/segment)
+of this repository.
+
+## Segmentor
 
 The `Segmentor` class is designed for semantic segmentation tasks, extracting feature maps from intermediate layers of the Clay Encoder and adding a Feature Pyramid Network (FPN) on top of it.
 
-Decoder is inspired by the Segformer paper.
-Todo:
-- Add neck & head for segmentation task from other papers like UperNet, PPANet, etc. to compare with other GeoAI models.
+The decoder in this example is inspired by the Segformer paper.
 
+A possible future improvement will be to add a neck & head for segmentation
+task from other papers like UperNet, PPANet, etc. to compare with other
+GeoAI models.
 
-## Parameters
+
+### Parameters
 
 - `feature_maps (list)`: Indices of intermediate layers of the Clay Encoder used by FPN layers.
 - `ckpt_path (str)`: Path to the Clay model checkpoint.
 
-## Example
+### Example
 
 In this example, we will use the `Segmentor` class to segment Land Use Land Cover (LULC) classes for the Chesapeake Bay CVPR dataset. The implementation includes data preprocessing, data loading, and model training workflow using PyTorch Lightning.
 
-## Dataset
-
-### Citation
+## Dataset citation
 
 If you use this dataset, please cite the associated manuscript:
 
@@ -111,9 +120,13 @@ The model can be run via LightningCLI using configurations in `finetune/segment/
     ```
 
 4. Train the model:
-    ```bash
-    python segment.py fit --config configs/segment_chesapeake.yaml
-    ```
+
+To ensure that the imports work properly, ensure that the root of
+the repository is in the python path before running the script.
+
+```bash
+python -m finetune.segment.segment fit --config configs/segment_chesapeake.yaml
+```
 
 ## Acknowledgments
 

docs/index.md

@@ -6,6 +6,12 @@ Clay is a [foundation model](https://www.adalovelaceinstitute.org/resource/found
 
 Clay’s model takes satellite imagery, along with information about location and time, as an input, and outputs embeddings, which are mathematical representations of a given area at a certain time on Earth’s surface. It uses a Vision Transformer architecture adapted to understand geospatial and temporal relations on Earth Observation data. The model is trained via Self-supervised learning (SSL) using a Masked Autoencoder (MAE) method.
 
+| Zero to Hero technical Guide (For developers) | Non-Technical User Guide (upcoming webapps) |
+|:--:|:--:|
+| [![](https://github.com/Clay-foundation/model/assets/434029/0cbfb109-c391-4f73-99f0-abc8769a7a14)](https://www.youtube.com/live/Zd3rbBj56P0?si=_ts3aWBcso0KEH7K) | [![](https://github.com/Clay-foundation/model/assets/434029/5cd209ec-3571-4ae7-98a1-1cef9a898f25)](https://youtu.be/gFjbrGaAL6w?si=ampWKQi9bArVoJsy) |
+
+## Usage
+
 The Clay model can be used in three main ways:
 - **Generate semantic embeddings for any location and time.** You can use embeddings for a variety of tasks, including to:
   - _Find features:_ Locate objects or features, such as surface mines, aquaculture, or concentrated animal feeding operations.

environment.yml

@@ -27,6 +27,7 @@ dependencies:
   - stackstac~=0.5.0
   - timm~=0.9.16
   - torchdata~=0.7.1
+  - torchgeo~=0.5.2
   - torchvision~=0.16.1
   - transformers~=4.35.2
   - typeshed-client~=2.4.0