Update docs and package metadata

- Add Hydra configuration documentation to mkdocs
- Update pyproject.toml to include YAML config files
- Update changelog for release

CHANGELOG.md

@@ -1,19 +1,26 @@
 # Changelog
 
-## [v1.1.3] - 2024-11-20
+## [v1.1.1] - 2024-11-21
 
 ### Distribution
-- Update package-data handling
+- Moved configuration files into package structure
+- Switched to `importlib.resources` for resource management
+- Removed MANIFEST.in in favor of pyproject.toml configuration
+- Enhanced root directory resolution logic
 
-## [v1.1.2] - 2024-11-20
-
-### Distribution
-- Update project metadata for pypi
+### Infrastructure
+- Improved cluster management with better dry-run and slurm support
 
-## [v1.1.1] - 2024-11-20
+### CLI
+- Enhanced command-line interface error handling
+- Improved argument parsing for multiple commands
+- Added `init_config` for config-based customization of network and training
 
 ### Documentation
-- Remove broken badges from README for now
+- Updated package metadata for PyPI
+- Added project URLs to package configuration
+- Removed broken badges from README
+- Added explanations for hydra-config-based customization of network and training in `CLI Reference`
 
 ## [v1.1.0] - 2024-11-20
 
@@ -27,10 +34,13 @@
 
 ### Documentation
 - Improved docs
+- Updated project metadata for PyPI
+- Removed broken badges from README
 
 ### Infrastructure
 - Removed strict version pins for better compatibility (particularly removed UMAP constraints)
 - Updated GitHub workflows
 - Updated README badges
+- Updated package-data handling
 
 [v1.1.0]: https://github.com/TuragaLab/flyvis/releases/tag/v1.1.0

docs/docs/reference/hydra_default_config.md

@@ -0,0 +1,121 @@
+# Configuration System
+
+`flyvis` uses [Hydra](https://hydra.cc/) for configuration management of network and training. The configuration system is organized hierarchically, allowing for modular configuration of different components.
+
+## Configuration Components
+
+### Configuration Components
+
+#### Network Configuration
+
+* `network/`: Core network architecture and parameters
+* `connectome/`: Neural connectivity specification
+* `dynamics/`: Neural dynamics parameters
+* `edge_config/`: Synapse-related configurations (sign, strength, count)
+* `node_config/`: Neuron-related configurations (bias, time constants)
+* `stimulus_config/`: Input stimulus parameters
+
+#### Training Configuration
+
+* `optim/`: Optimization parameters
+* `penalizer/`: Loss function penalties
+* `scheduler/`: Learning rate scheduling
+* `task/`: Task-specific settings and parameters
+
+#### Top-level Configuration
+* `solver.yaml`: Training loop parameters
+
+
+## Default Configuration Structure
+
+The default configuration is structured as follows:
+
+```bash
+config
+├── network                         # Network configuration
+│   ├── connectome                  # Connectome configurations
+│   │   └── connectome.yaml         # Default connectome configuration
+│   ├── dynamics                    # Dynamics configurations
+│   │   └── dynamics.yaml           # Default dynamics configuration
+│   ├── edge_config                 # Edge configuration
+│   │   ├── edge_config.yaml        # Default edge configuration
+│   │   ├── sign                    # Signaling configurations
+│   │   │   └── sign.yaml           # Default signaling configuration
+│   │   ├── syn_count               # Synaptic count configurations
+│   │   │   └── syn_count.yaml      # Default synaptic count configuration
+│   │   └── syn_strength            # Synaptic strength configurations
+│   │       └── syn_strength.yaml   # Default synaptic strength configuration
+│   ├── network.yaml                # Default network configuration
+│   ├── node_config                 # Node configuration
+│   │   ├── bias                    # Bias configurations
+│   │   │   └── bias.yaml           # Default bias configuration
+│   │   ├── node_config.yaml        # Default node configuration
+│   │   └── time_const              # Time constant configurations
+│   │       └── time_const.yaml     # Default time constant configuration
+│   └── stimulus_config             # Stimulus configuration
+│       └── stimulus_config.yaml    # Default stimulus configuration
+├── optim                           # Optimizer configuration
+│   └── optim.yaml                  # Default optimizer configuration
+├── penalizer                       # Penalizer configuration
+│   └── penalizer.yaml              # Default penalizer configuration
+├── scheduler                       # Scheduler configuration
+│   └── scheduler.yaml              # Default scheduler configuration
+├── solver.yaml                     # Default solver configuration
+└── task                            # Task configuration
+    └── task.yaml                   # Default task configuration
+
+15 directories, 16 files
+```
+
+## Customizing Configurations
+
+### Overriding Default Parameters
+
+For quick parameter changes you can use hydra command-line overrides.
+
+```bash
+# Example: Change the scale of the synaptic strength to 0.5
+flyvis train-single network.edge_config.syn_strength.scale=0.5 ensemble_and_network_id=0042/007 task_name=flow
+
+# Example: Add a custom parameter to an existing config
+flyvis train-single +network.edge_config.syn_strength.custom_param=100 ensemble_and_network_id=0042/007 task_name=flow
+```
+
+More information on hydra command-line overrides can be found [here](https://hydra.cc/docs/advanced/override_grammar/basic/).
+
+### Using Custom Configurations
+
+In many situations you will want to create custom configuration files, to either break
+the existing config structure for new code or to keep track of different sets of parameters.
+
+This particularly applies when installing `flyvis` as a package instead of using the
+developer mode, where one can directly modify the source code configuration.
+
+To create and maintain your own configuration files proceed as follows:
+
+1. Initialize a local config directory:
+
+    ```bash
+    # Copy the default config structure into flyvis_config
+    flyvis init-config
+    ```
+
+2. Create your custom configurations in the generated `flyvis_config` directory.
+
+3. Use your custom configs:
+
+    ```bash
+    # Using the full custom config directory
+    flyvis train-single --config-path $(pwd)/flyvis_config [other options]
+
+    # Using a specific custom config
+    flyvis train-single network/edge_config/syn_strength=custom_syn_strength --config-dir $(pwd)/flyvis_config
+    ```
+
+    Notice that **`config-path`** overrides the entire default config directory, while **`config-dir`** only adds another search path for resolving configs.
+
+    **Important:** It is recommended to use different file names than the defaults.
+    This is because for partial usage with `config-dir` hydra resolves the config names in the default config directory first. I.e., your changes reflected in files of the same name in the custom config directory might not have the intended effect. E.g., name your config file `custom_syn_strength.yaml` instead of `syn_strength.yaml`.
+
+
+More information on hydra configuration can be found [here](https://hydra.cc/docs/advanced/search_path/).

docs/docs/release.md

@@ -0,0 +1,171 @@
+# Release Process
+
+## Prerequisites
+
+1. Ensure all tests pass:
+```bash
+pytest
+```
+
+2. Update and verify the documentation:
+Follow the instructions in the [readme](README.md) to update and verify the documentation.
+The deployment to github can be done last or via workflow.
+
+3. Install required tools:
+```bash
+python -m pip install build twine
+```
+
+## Release Steps
+
+0. **Test PyPi before committing (optional)**
+
+One can create the wheel and upload it to Test PyPi before committing to develop the package.
+This can be useful to test if the installation will work before commiting to a version number and
+push to the remote repository. However, the wheels that are uploaded to Test PyPi cannot
+be deleted so one should probably do this sparingly and not use version numbers one wants to reserve for
+the actual release.
+
+### Upload to Test PyPi
+```bash
+# Clean previous builds
+rm -rf dist/
+
+# Set version temporarily for this session manually (change version number)
+export SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0.dev4
+
+# Now build and test
+python -m build
+
+# Upload to Test PyPI first
+python -m twine upload --repository testpypi dist/*
+```
+
+### Test Installation
+
+In a clean environment, run these sporadic tests to verify the installation:
+```bash
+# Install in clean environment to test (change version number)
+python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ flyvis==0.0.0.dev4
+
+# Test installation
+flyvis download-pretrained
+flyvis train --help
+flyvis train-single --help
+python -c "from flyvis import Network; n = Network()"
+python -c "from flyvis import EnsembleView; e = EnsembleView('flow/0000')"
+
+
+# Test custom configuration
+flyvis init-config
+# Add custom_syn_strength.yaml to flyvis_config/network/edge_config/syn_strength/
+cat > flyvis_config/network/edge_config/syn_strength/custom_syn_strength.yaml << 'EOL'
+defaults:
+  - /network/edge_config/syn_strength/syn_strength@_here_
+  - _self_
+
+scale: 1.0
+EOL
+# Run training and check if custom config is loaded correctly
+flyvis train-single --config-dir $(pwd)/flyvis_config network/edge_config/syn_strength=custom_syn_strength ensemble_and_network_id=0 task_name=flow delete_if_exists=true 2>&1 | while read line; do
+    echo "$line"
+    if [[ "$line" == *"'syn_strength': {'type': 'SynapseCountScaling'"* && "$line" == *"'scale': 1.0"* ]]; then
+        echo "Found custom scale parameter in config!"
+        pkill -P $$
+        break
+    fi
+done
+# Delete custom config
+rm -rf flyvis_config
+
+# Delete installation and downloaded models
+pip uninstall flyvis -y
+
+# When done testing, unset it
+unset SETUPTOOLS_SCM_PRETEND_VERSION
+```
+
+### Commit Changes
+
+Commit all open changes to the repository.
+
+### Update Changelog
+
+- Append entry in `CHANGELOG.md` with new version number
+- Include all notable changes under appropriate sections, e.g.,
+   - Breaking
+   - Features
+   - Documentation
+   - Infrastructure
+   - Distribution
+   - Bug Fixes
+
+```bash
+git add CHANGELOG.md
+git commit -m "docs: add changelog for vX.Y.Z"
+```
+
+### Create and Push Tag
+
+```bash
+# Create annotated tag using changelog
+git tag -a vX.Y.Z -F CHANGELOG.md
+
+# Push to both remotes
+git push origin main
+git push origin vX.Y.Z
+git push public_repo main
+git push public_repo vX.Y.Z
+```
+
+### Build and Upload to PyPI
+```bash
+# Clean previous builds
+rm -rf dist/
+
+# Build package
+python -m build
+
+# Set version temporarily for this session manually
+export SETUPTOOLS_SCM_PRETEND_VERSION=X.Y.Z
+
+# Now build and test
+python -m build
+python -m twine upload --repository testpypi dist/*
+
+# When done testing, unset it
+unset SETUPTOOLS_SCM_PRETEND_VERSION
+
+# Upload to Test PyPI first (recommended)
+python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ flyvis==X.Y.Z
+
+# Upload to PyPI
+python -m twine upload dist/*
+```
+
+### Create GitHub Release
+   - Go to GitHub releases page
+   - Create new release using the tag
+   - Copy changelog entry into release description
+
+## Post-release
+
+1. Verify package can be installed from PyPI:
+```bash
+python -m pip install flyvis
+```
+
+## Check documentation is updated on the documentation website
+
+## Version Numbering
+
+We follow semantic versioning (MAJOR.MINOR.PATCH):
+- MAJOR: Breaking changes
+- MINOR: New features (backwards compatible)
+- PATCH: Bug fixes
+
+## Notes
+
+- Always test on Test PyPI before releasing to PyPI
+- Ideally CI checks pass before releasing
+- Keep both origin and public_repo remotes in sync

docs/mkdocs.yml

@@ -42,6 +42,7 @@ nav:
     - CLI Reference:
       - Command Line Interface: reference/cli.md
       - CLI Entry Point: reference/flyvis_cli/flyvis_cli.md
+      - Hydra Configuration: reference/hydra_default_config.md
       - Training:
         - Run Training for Single Model: reference/flyvis_cli/training/train_single.md
         - Launch Ensemble Training on Compute Cloud: reference/flyvis_cli/training/train.md

pyproject.toml

@@ -15,6 +15,8 @@ flyvis = [
   "analysis/visualization/matplotlibrc",
   "analysis/__main__.ipynb",
   "analysis/__main_per_model__.ipynb",
+  "config/**/*.yaml",
+  "config/**/*.yml",
 ]
 
 [project]