New dashboard #4330

srugano · 2024-10-15T13:07:20Z

AB#219780

codecov · 2024-10-15T13:59:58Z

Codecov Report

Attention: Patch coverage is 77.72021% with 43 lines in your changes missing coverage. Please review.

Project coverage is 55.05%. Comparing base (6ef7a33) to head (1a824d1).

Files with missing lines	Patch %	Lines
src/hct_mis_api/apps/dashboard/celery_tasks.py	53.12%	15 Missing ⚠️
src/hct_mis_api/apps/dashboard/views.py	75.00%	15 Missing ⚠️
src/hct_mis_api/apps/dashboard/services.py	80.00%	11 Missing ⚠️
src/hct_mis_api/config/settings.py	88.88%	1 Missing ⚠️
src/hct_mis_api/middlewares/xframe.py	90.90%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6ef7a33) and HEAD (1a824d1). Click for more details.

HEAD has 2 uploads less than BASE

Flag BASE (6ef7a33) HEAD (1a824d1)

unittests 2 0

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #4330       +/-   ##
============================================
- Coverage    77.59%   55.05%   -22.55%     
============================================
  Files          944      950        +6     
  Lines        44624    44620        -4     
============================================
- Hits         34628    24564    -10064     
- Misses        9996    20056    +10060

Flag	Coverage Δ
e2e	`55.05% <77.72%> (+7.44%)`	⬆️
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

… to allow i-frame.

domdinicola · 2024-10-31T12:54:12Z

@srugano regenerate lock

domdinicola · 2024-10-31T13:09:52Z

@srugano regenerate pdm

mmaciekk · 2024-11-04T09:11:02Z

src/hct_mis_api/apps/dashboard/static/dashboard/js/dashboard.js

+    }
+
+    const url = householdDataUrl;
+    console.log(url);


remove console.log

mmaciekk · 2024-11-04T09:19:13Z

src/hct_mis_api/apps/dashboard/static/dashboard/js/dashboard.js

+          ).value();
+
+          const totalSuccessfulUSD = successfulPaymentsGroupUSD.sum;
+          console.log(totalSuccessfulUSD);


remove console.log

mmaciekk · 2024-11-04T09:42:06Z

src/hct_mis_api/apps/dashboard/static/dashboard/js/dashboard.js

+            (p, v) => p - (v.children_count || 0),
+            () => 0
+          ).value();
+          console.log( successfulPaymentsCountUSD.sum, totalPaymentsCount, pendingPaymentsGroupUSD.count);


remove console log

mmaciekk · 2024-11-04T10:03:04Z

tests/selenium/country_dashboard/test_country_dashboard.py

-        assert "0" in pageCountryDashboard.getTotalNumberOfFeedback().text
-        assert "USD 0.00" in pageCountryDashboard.getTotalAmountTransferred().text
+        pageCountryDashboard.switch_to_dashboard_iframe()
+        # assert pageCountryDashboard.get_total_amount_paid().text == "", "Expected total amount paid to be empty."


Why are these tests commented out?

mmaciekk · 2024-11-04T10:06:07Z

src/hct_mis_api/middlewares/xframe.py

+from typing import List, Optional
+
+from django.http import HttpRequest, HttpResponse
+from django.utils.deprecation import MiddlewareMixin


@johniak, @MarekBiczysko Please check if this middleware is correct. I needed to overwrite the default django setting in order to display dashboard page in an iframe

saxix · 2024-11-03T17:39:00Z

pyproject.toml

@@ -98,6 +98,7 @@ dev = [
    "pytest<8.0.0,>=7.4.4",
    "pytest-django<5.0.0,>=4.5.2",
    "pytest-echo<2.0.0,>=1.7.1",
+    "pytest-mock<4.0.0,>=3.6.0", 


What is the reason for this library? Is it needed?

That's the way I was able to test tasks running in exceptions.

saxix · 2024-11-03T17:40:24Z

tests/unit/apps/dashboard/test_tasks.py

+    """
+    Test that generate_dash_report_task refreshes data for the given business area.
+    """
+    mock_refresh_data = mocker.patch.object(DashboardDataCache, "refresh_data")


This is the wrong way to call patch.object. You are not patching an object here

saxix · 2024-11-03T17:41:38Z

tests/unit/apps/dashboard/test_services.py

+    mock_household_objects.using.return_value.filter.return_value = mock_households
+    mock_serializer.return_value.data = {"id": 1, "name": "test"}
+
+    with patch.object(DashboardDataCache, "store_data") as mock_store_data:


this can be done with monkeypatch and it is more correct in this case

saxix · 2024-11-04T09:47:36Z

tests/selenium/country_dashboard/test_country_dashboard.py

-        assert "0" in pageCountryDashboard.getTotalNumberOfGrievances().text
-        assert "0" in pageCountryDashboard.getTotalNumberOfFeedback().text
-        assert "USD 0.00" in pageCountryDashboard.getTotalAmountTransferred().text
+        pageCountryDashboard.switch_to_dashboard_iframe()


what are these commented out lines?

still updating the figures

tests/unit/apps/dashboard/test_services.py

saxix · 2024-11-04T10:14:48Z

tests/unit/apps/dashboard/test_tasks.py

+@patch.object(DashboardDataCache, "refresh_data")
+@patch("hct_mis_api.apps.core.models.BusinessArea.objects")
+def test_update_dashboard_figures(
+    mock_business_area_objects: MagicMock, mock_refresh_data: MagicMock, afghanistan: BusinessArea


where are these fixtures ?

src/frontend/src/containers/pages/dashboard/NewDashboardPage.tsx

MarekBiczysko · 2024-11-04T12:46:58Z

src/hct_mis_api/apps/dashboard/services.py

+    @classmethod
+    def refresh_data(cls, business_area_slug: str) -> Dict[str, Any]:
+        """
+        Generate and store updated data for a given business area.
+        """
+        households = Household.objects.using("read_only").filter(business_area__slug=business_area_slug)
+        serialized_data = DashboardHouseholdSerializer(households, many=True).data
+
+        cls.store_data(business_area_slug, serialized_data)
+        return serialized_data


If I understand correctly we cache ALL households/payments data for the whole BA and then do the statistics calculations in the browser. This is huge amount of data for some business areas, wouldn't be easier to precalculate statistics on server side and cache only the final results?

I have tested with 500000 payments over 3 years and it can handle. I have precalculated some totals

src/frontend/src/containers/pages/dashboard/NewDashboardPage.tsx

src/hct_mis_api/apps/dashboard/static/dashboard/js/dashboard.js

New dashbiard

0620f24

srugano added 3 commits October 21, 2024 09:11

Merge branch 'develop' into feature/dashboarmodule

9d83f62

Refactored and set redis

eb7f3f0

Lint fixes

129dcf4

srugano marked this pull request as ready for review October 22, 2024 01:06

srugano and others added 8 commits October 22, 2024 06:18

Test api fixes

6708a2c

Deleted migration files

128182c

rewritting tests

cad73f2

Merge branch 'develop' into feature/dashboarmodule

16bf1ee

Fixing tests

e3e850d

Fix

7bd5f62

embed new dashboard in the hope react app, modify django app settings…

d09d687

… to allow i-frame.

Merge branch 'develop' into feature/dashboarmodule

e520d85

mmaciekk assigned srugano and mmaciekk Oct 30, 2024

mmaciekk added frontend Changes in FE code backend Changes in BE code labels Oct 30, 2024

Some other tests

26e5b49

srugano added 9 commits October 31, 2024 14:53

Updated template

c8a6175

Rearanged the files

87c0286

Before adding per year

4d42a3a

Better js fixes

68aeb1e

Merge branch 'develop' into feature/dashboarmodule

8e3395e

Pdm updates

bb3c41f

Env fixes

4d507a0

Updated tests

37025e6

Tests are ok

48abc99

Testing services

df515a0

srugano requested review from johniak, domdinicola and ntrncic November 3, 2024 01:46

updated smoke tests

c82993d

srugano requested a review from mmaciekk November 3, 2024 05:53

Minimalize e2e tests

d4c1a16

domdinicola requested review from saxix, MarekBiczysko and pkujawa November 4, 2024 08:07

mmaciekk reviewed Nov 4, 2024

View reviewed changes

saxix requested changes Nov 4, 2024

View reviewed changes

MarekBiczysko reviewed Nov 4, 2024

View reviewed changes

mmaciekk reviewed Nov 4, 2024

View reviewed changes

src/frontend/src/containers/pages/dashboard/NewDashboardPage.tsx Show resolved Hide resolved

src/hct_mis_api/apps/dashboard/static/dashboard/js/dashboard.js Outdated Show resolved Hide resolved

srugano added 3 commits November 6, 2024 15:34

Updated services

409b19e

Merge branch 'develop' into feature/dashboarmodule

c78fc7f

Merged develop

1a824d1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New dashboard #4330

New dashboard #4330

srugano commented Oct 15, 2024 •

edited by mmaciekk

Loading

codecov bot commented Oct 15, 2024 •

edited

Loading

domdinicola commented Oct 31, 2024

domdinicola commented Oct 31, 2024

mmaciekk Nov 4, 2024

mmaciekk Nov 4, 2024

mmaciekk Nov 4, 2024

mmaciekk Nov 4, 2024

mmaciekk Nov 4, 2024

saxix Nov 3, 2024

srugano Nov 4, 2024 •

edited

Loading

saxix Nov 3, 2024

saxix Nov 3, 2024

saxix Nov 4, 2024

srugano Nov 6, 2024

saxix Nov 4, 2024

MarekBiczysko Nov 4, 2024

srugano Nov 6, 2024

New dashboard #4330

Are you sure you want to change the base?

New dashboard #4330

Conversation

srugano commented Oct 15, 2024 • edited by mmaciekk Loading

codecov bot commented Oct 15, 2024 • edited Loading

Codecov Report

domdinicola commented Oct 31, 2024

domdinicola commented Oct 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srugano Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srugano commented Oct 15, 2024 •

edited by mmaciekk

Loading

codecov bot commented Oct 15, 2024 •

edited

Loading

srugano Nov 4, 2024 •

edited

Loading