merge upstream develop

.dockerignore

@@ -1,2 +1,7 @@
 .db/
-deploy/
+deploy/
+.mypy_cache/
+.pytest_cache/
+go-logs/
+media/
+go.log

.github/workflows/add-issue-to-backlog.yml

@@ -0,0 +1,16 @@
+name: Add issues to Backlog
+
+on:
+  issues:
+    types:
+      - opened
+
+jobs:
+  add-to-project:
+    name: Add issue to project
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/[email protected]
+        with:
+          project-url: https://github.com/orgs/IFRCGo/projects/12
+          github-token: ${{ secrets.ADD_TO_PROJECT_PAT }}

CHANGELOG.md

@@ -6,6 +6,79 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## Unreleased
 
+## 1.1.471
+
+### Added
+ - Hotfix for DREF permissions
+
+## 1.1.470
+
+### Added
+ - Internal plan files: collecting only PDF-s
+ - Nicer LocalUnit admin lists
+ - ISO3 filtering possibility for districts and appeals
+ - GitHub Actions - add issue to Backlog project
+
+## 1.1.469
+
+### Added
+ - Ingest country plan and internal plan files
+ - Bump up cryptography and django modules
+ - Surge Alert statuses: Open, Closed, Stood down
+
+## 1.1.468
+
+### Added
+ - Fix Surge Alert error 500 when no linked event
+
+## 1.1.467
+
+### Added
+ - Pagination fix (event, personnel)
+
+## 1.1.466
+
+### Added
+ - DREF Final Report finalization + new features
+ - LocalUnits
+ - Event visibility fix
+
+## 1.1.465
+
+### Added
+ - Ingest country plan
+
+## 1.1.464
+
+### Added
+ - Fix timeout in Surgealert: Export All
+ - Bump up some packages
+ - Fix some settings
+
+## 1.1.463
+
+### Added
+ - DREF Final report, v0.1
+ - Ops Update: optimistic lock
+ - Dropping non-used enum Choices
+ - Fix filter in DREF
+ - Admin page: search possibility of Admin2 countries
+
+## 1.1.462
+
+### Added
+ - Country Plans – strategic priorities
+ - Adding IDN, MYS, PHL, POL to Admin2 areas
+
+## 1.1.461
+
+### Added
+ - Introducing COUNTRY PLANs
+ - Only active users to be shown in DREF forms (for sharing)
+ - Add centroid processing for Admin2
+ - DREF Ops Update validation fixes
+ - Update snapshottest to 0.6.0 (and other small modules)
+
 ## 1.1.460
 
 ### Added
@@ -2090,7 +2163,18 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## 0.1.20
 
-[Unreleased]: https://github.com/IFRCGo/go-api/compare/1.1.460...HEAD
+[Unreleased]: https://github.com/IFRCGo/go-api/compare/1.1.471...HEAD
+[1.1.471]: https://github.com/IFRCGo/go-api/compare/1.1.470...1.1.471
+[1.1.470]: https://github.com/IFRCGo/go-api/compare/1.1.469...1.1.470
+[1.1.469]: https://github.com/IFRCGo/go-api/compare/1.1.468...1.1.469
+[1.1.468]: https://github.com/IFRCGo/go-api/compare/1.1.467...1.1.468
+[1.1.467]: https://github.com/IFRCGo/go-api/compare/1.1.466...1.1.467
+[1.1.466]: https://github.com/IFRCGo/go-api/compare/1.1.465...1.1.466
+[1.1.465]: https://github.com/IFRCGo/go-api/compare/1.1.464...1.1.465
+[1.1.464]: https://github.com/IFRCGo/go-api/compare/1.1.463...1.1.464
+[1.1.463]: https://github.com/IFRCGo/go-api/compare/1.1.462...1.1.463
+[1.1.462]: https://github.com/IFRCGo/go-api/compare/1.1.461...1.1.462
+[1.1.461]: https://github.com/IFRCGo/go-api/compare/1.1.460...1.1.461
 [1.1.460]: https://github.com/IFRCGo/go-api/compare/1.1.459...1.1.460
 [1.1.459]: https://github.com/IFRCGo/go-api/compare/1.1.458...1.1.459
 [1.1.458]: https://github.com/IFRCGo/go-api/compare/1.1.457...1.1.458

README.md

@@ -177,13 +177,27 @@ set. Check the script for the specific variables in your environment.
 ## Deployment command
 
 ```(bash)
-docker run -p 80:80 --env-file .env -d -t ifrcgo/go-api:{TAG_NUMBER}
+docker-compose up serve celery
+```
+or (just the base serve command):
+```(bash)
+docker-compose run --rm --service-ports serve
 ```
-
 ## Comment for loading data
 
 In `main/runserver.sh` the line containing the `loaddata` command is only necessary when creating a new database. In other cases it might be causing the conflict, so it is commented. 
 
+## Initializing ElasticSearch
+
+For the initial creation of an index
+```(bash)
+docker-compose exec serve bash python manage.py rebuild_index
+```
+For updating the index
+```(bash)
+docker-compose exec serve bash python manage.py update_index
+```
+
 # Management commands to update and import admin0 and admin1 data
 
 There are two Django management commands that helps to work with ICRC admin0 and admin1 shapefiles. These commands should be used only when you want to update geometries, or import new ones from a shapefile. The structure of the shapefile is not very flexible, but can be adjusted easily in the scripts. 
@@ -225,7 +239,7 @@ The shapefile should have the following mandatory fields:
 * admin1_id (this is the ID of the GO district this admin2 belongs to)
 
 See [this ticket](https://github.com/IFRCGo/go-api/issues/1492#issuecomment-1284120696) for a full workflow of preparing the admin2 shapefiles.
-The above command will generate a list of missing admin2s in the database based on the code (we use pcodes) to a file called `missing-admin2.txt`
+The above command will generate a list of missing admin2-s in the database based on the code (we use pcodes) to a file called `missing-admin2.txt`
 
 ### Options available for the command
 * `--update-geom` -- updates the geometry for all admin2 matched in the shapefile.
@@ -237,13 +251,13 @@ The above command will generate a list of missing admin2s in the database based
 Run `python manage.py update-region-bbox` to update the bbox for each region in the database.
 
 ## Import FDRS codes
-Run `python manage.py import-fdrs iso-fdrs.csv` to update the countries table with FDRS codes. The csv should have `iso,fdrs` structure
+Run `python manage.py import-fdrs iso-fdrs.csv` to update the countries table with FDRS codes. The csv should have `iso, fdrs` structure
 
 ## Update sovereign state and disputed status
-Run ` python manage.py update-sovereign-and-disputed new_fields.csv` to update the countries table with sovereign states and disputed status. The CSV should have the `id,iso,name,sovereign_state,disputed` columns. The matching is based on iso and name. If iso is null, we fallback to name.
+Run ` python manage.py update-sovereign-and-disputed new_fields.csv` to update the countries table with sovereign states and disputed status. The CSV should have the `id,iso,name,sovereign_state,disputed` columns. The matching is based on iso and name. If iso is null, we fall back to name.
 
 ## Update Mapbox Tilesets
-To update GO countries and districts Mapbox tilesets, run the management command `python manage.py update-mapbox-tilesets`. This will export all country and district geometries to a GeoJSON file, and then upload them to Mapbox. The tilesets will take a while to process. The updated status can be viewed on the Mapbox Studio under tilesets. To run this management command, MAPBOX_ACCESS_TOKEN should be set in the environment.
+To update GO countries and districts Mapbox tilesets, run the management command `python manage.py update-mapbox-tilesets`. This will export all country and district geometries to a GeoJSON file, and then upload them to Mapbox. The tilesets will take a while to process. The updated status can be viewed on the Mapbox Studio under tilesets. To run this management command, MAPBOX_ACCESS_TOKEN should be set in the environment. The referred files are in ./mapbox/..., so you should **not** run this command from an arbitrary point of the vm's filesystem (e.g. from the location of shapefiles), but from Django root.
 
 ### Options available for the command
 * `--production` — update production tilesets. If this flag is not set, by default the script will only update staging tiles

api/admin.py

@@ -245,24 +245,6 @@ def field_reports(self, instance):
         return mark_safe('<span class="errors">No related field reports</span>')
     field_reports.short_description = 'Field Reports'
 
-# For multiple document fields inline. TO be FIXED: only the last one is saved. Change also tabular.html (DELETEME)
-#   def save_formset(self, request, form, formset, change):
-#       if hasattr(formset.model, 'document'): # SituationReports (or other similars)
-#           instances = formset.save(commit=False)
-#           for inst in formset.deleted_objects:
-#               inst.delete()
-#           for inst in formset.changed_objects:
-#               inst.save()
-#           for inst in formset.new_objects:
-#               for i,one_document in enumerate(request.FILES.getlist('documents_multiple')):
-#                   if i<30: # not letting tons of documents to be attached
-#                       inst.name     = inst.name if i == 0 else inst.name + '-' + str(i)
-#                       inst.document = one_document
-#                       inst.save()
-#           formset.save_m2m()
-#       else:
-#           formset.save()
-
 
 class GdacsAdmin(CompareVersionAdmin, RegionRestrictedAdmin, TranslationAdmin):
     country_in = 'countries__pk__in'
@@ -526,7 +508,7 @@ class DistrictAdmin(geoadmin.OSMGeoAdmin, CompareVersionAdmin, RegionRestrictedA
 
 class CountryAdmin(geoadmin.OSMGeoAdmin, CompareVersionAdmin, RegionRestrictedAdmin, TranslationAdmin):
     country_in = 'pk__in'
-    list_display = ('__str__', 'record_type')
+    list_display = ('__str__', 'record_type', 'iso3')
     region_in = 'region__pk__in'
     list_editable = ('record_type',)
     search_fields = ('name',)
@@ -550,10 +532,12 @@ class RegionAdmin(geoadmin.OSMGeoAdmin, CompareVersionAdmin, RegionRestrictedAdm
     search_fields = ('name',)
     modifiable = True
 
+
 class Admin2Admin(geoadmin.OSMGeoAdmin, CompareVersionAdmin, RegionRestrictedAdmin):
-    search_fields = ('name',)
+    search_fields = ('name', 'admin1__country__name')
     modifiable = True
 
+
 class UserProfileAdmin(CompareVersionAdmin):
     search_fields = ('user__username', 'user__email', 'country__name',)
     list_filter = (

api/drf_views.py

@@ -11,13 +11,14 @@
 from django_filters import rest_framework as filters
 from django.contrib.auth.models import User
 from django.db import models
-from django.db.models import Prefetch, Count, Q
+from django.db.models import Prefetch, Count, Q, OuterRef
 from django.utils import timezone
 
 from main.utils import is_tableau
 from deployments.models import Personnel
 from databank.serializers import CountryOverviewSerializer
 
+from .utils import is_user_ifrc
 from .event_sources import SOURCES
 from .exceptions import BadRequest
 from .view_filters import ListFilter
@@ -61,6 +62,8 @@
     CountryOfFieldReportToReview,
 )
 
+from country_plan.models import CountryPlan
+
 from .serializers import (
     ActionSerializer,
     DisasterTypeSerializer,
@@ -134,8 +137,8 @@ class DeploymentsByEventViewset(viewsets.ReadOnlyModelViewSet):
                                     'personneldeployment__personnel',
                                     filter=Q(
                                         personneldeployment__personnel__type=Personnel.TypeChoices.RR,
-                                        personneldeployment__personnel__start_date__lte=timezone.now(),
-                                        personneldeployment__personnel__end_date__gte=timezone.now(),
+                                        personneldeployment__personnel__start_date__date__lte=timezone.now(),
+                                        personneldeployment__personnel__end_date__date__gte=timezone.now(),
                                         personneldeployment__personnel__is_active=True
                                     )
                                 )
@@ -167,7 +170,11 @@ class DisasterTypeViewset(viewsets.ReadOnlyModelViewSet):
 
 
 class RegionViewset(viewsets.ReadOnlyModelViewSet):
-    queryset = Region.objects.all()
+    queryset = Region.objects.annotate(
+        country_plan_count=Count(
+            'country__country_plan', filter=Q(country__country_plan__is_publish=True)
+        )
+    )
 
     def get_serializer_class(self):
         if self.action == 'list':
@@ -185,17 +192,20 @@ class Meta:
 
 
 class CountryViewset(viewsets.ReadOnlyModelViewSet):
-    queryset = Country.objects.filter(is_deprecated=False)
+    queryset = Country.objects.filter(is_deprecated=False).annotate(
+        has_country_plan=models.Exists(CountryPlan.objects.filter(country=OuterRef('pk'), is_publish=True))
+    )
     filterset_class = CountryFilter
     search_fields = ('name',)  # for /docs
 
     def get_object(self):
         pk = self.kwargs['pk']
+        qs = self.get_queryset()
         try:
-            return Country.objects.get(pk=int(pk))
+            return qs.get(pk=int(pk))
         except ValueError:
             # NOTE: If pk is not integer try searching for name or iso
-            country = Country.objects.filter(
+            country = qs.filter(
                 models.Q(name__iexact=str(pk)) | models.Q(iso__iexact=str(pk))
             )
             if country.exists():
@@ -328,7 +338,7 @@ def get_serializer_class(self):
 class DistrictFilter(filters.FilterSet):
     class Meta:
         model = District
-        fields = ('country',)
+        fields = ('country', 'country__iso3', 'name',)
 
 
 class DistrictViewset(viewsets.ReadOnlyModelViewSet):
@@ -428,10 +438,20 @@ def retrieve(self, request, pk=None, *args, **kwargs):
         if pk:
             try:
                 if self.request.user.is_authenticated:
-                    if self.request.user.is_superuser:
+                    if is_user_ifrc(self.request.user):
                         instance = Event.objects.get(pk=pk)
                     else:
-                        instance = Event.objects.exclude(visibility=VisibilityChoices.IFRC).exclude(Q(visibility=VisibilityChoices.IFRC_NS) & ~Q(countries__id__in=UserCountry.objects.filter(user=self.request.user.id).values_list('country',flat=True).union(Profile.objects.filter(user=self.request.user.id).values_list('country',flat=True)))).get(pk=pk)
+                        user_countries = UserCountry.objects\
+                            .filter(user=request.user.id).values('country')\
+                            .union(
+                                Profile.objects.filter(user=request.user.id).values('country')
+                            )
+                        instance = Event.objects\
+                            .exclude(
+                                visibility=VisibilityChoices.IFRC)\
+                            .exclude(
+                                Q(visibility=VisibilityChoices.IFRC_NS) & ~Q(countries__id__in=user_countries))\
+                            .get(pk=pk)
                 else:
                     instance = Event.objects.filter(visibility=VisibilityChoices.PUBLIC).get(pk=pk)
                 # instance = Event.get_for(request.user).get(pk=pk)
@@ -553,6 +573,7 @@ class Meta:
             'valid_from': ('exact', 'gt', 'gte', 'lt', 'lte'),
             'valid_to': ('exact', 'gt', 'gte', 'lt', 'lte'),
             'appeal__real_data_update': ('exact', 'gt', 'gte', 'lt', 'lte'),
+            'country__iso3': ('exact',),
         }
 
 
@@ -1074,6 +1095,7 @@ class GoHistoricalViewSet(viewsets.ReadOnlyModelViewSet):
     def get_queryset(self):
         return Event.objects.filter(appeals__isnull=False)
 
+
 class CountryOfFieldReportToReviewViewset(viewsets.ReadOnlyModelViewSet):
     queryset = CountryOfFieldReportToReview.objects.order_by('country')
     serializer_class = CountryOfFieldReportToReviewSerializer