Prt 40 periodic full system reconcile process #568

miker985 · 2024-05-16T17:46:39Z

Not a real PR yet - just trying to make sure we don't code the same things in different ways...

Add docstring to Role + MANGAER_ROLE constant blacken models/shared_drive.py Change SharedDrive.drive_id max_length (32 => 25; 19 is largest value) Add SharedDrive.is_deleted Add SharedDriveRecord.get_itbill_key_remote()

…WIP reconciler

miker985 · 2024-05-16T17:47:54Z

endorsement/dao/shared_drive.py



-def load_shared_drives_from_csv(file_path):
+def load_shared_drives(google_drive_states):


Notable change: load_shared_drives management command is now loading from the MSCA endpoint and not a CSV file

miker985 · 2024-05-16T17:48:42Z

endorsement/dao/shared_drive.py

-        shared_drive=shared_drive)
+        shared_drive=shared_drive,
+        defaults={
+            "datetime_created": dt.datetime.now(dt.timezone.utc),


@mikeseibel Please confirm datetime_created is meant to refer to the creation of the SharedDriveRecord object.

miker985 · 2024-05-16T17:54:42Z

endorsement/dao/shared_drive.py

+    try:
+        defaults["quota_limit"] = a.quota_limit
+    except ValueError:
+        "Drive has no quota set. Probably in invalid org unit."


The expectation is this will get "fixed" during reconciliation, but it's a little awkward prior to our cutover when the majority of the share drives are in an OU with no quota

what should the placeholder for those cases be?

I'd have to confer with Ken on that to see if there's a quota currently being enforced.

I'm aware of 3 OUs

the default UW one, where most drives are

one that follows a different naming scheme but appears to be compatible with a new "XXXGB" OU

the one for drives pending deletion

miker985 · 2024-05-16T17:55:41Z

endorsement/dao/shared_drive.py

-    def __init__(self, *args, **kwargs):
-        columns = kwargs.get('columns', [])
-        row = kwargs.get('row', [])
+def get_subscription(sdr: SharedDriveRecord):


@mikeseibel this method prompted the PR as I think it's getting dangerously close to code you're writing and I'd really rather only write it once 😆

feel free to junk whats there. i just added it quick and dirty to import ken's early csv.

miker985 · 2024-05-16T17:56:27Z

endorsement/dao/shared_drive.py

+    def reconcile_existing_drives(
+        self, drives, id_GoogleDriveState, subsidized_quota
+    ):
+        # helper functions that probably need to be moved to a DAO file


Ignore this function - I quickly put some stubs together but it's not even remotely complete

miker985 · 2024-05-16T17:58:05Z

endorsement/models/shared_drive.py

-    org_unit_name = models.CharField(max_length=64, null=True)
+    org_unit_name = models.CharField(
+        max_length=64, null=True
+    )  # TODO: shouldn't this be non-NULL?


OK to make this not-nullable?

miker985 · 2024-05-16T18:00:33Z

endorsement/models/shared_drive.py

-    drive_id = models.SlugField(max_length=32)
+
+    # NOTE: maximum observed is 19 characters (2024-05-16)
+    drive_id = models.SlugField(max_length=25)


Reduced from 32 => 25 so that the new method get_itbill_key_remote() would never return a value <= length rquired by ITBillSubscription

seems fine... i didn't look too hard and felt like some fudge didn't hurt

miker985 · 2024-05-16T18:01:12Z

endorsement/models/shared_drive.py

+        """
+        # if changing be sure this does not exceed the max_length of
+        # ITBillSubscription.key_remote
+        key_remote = f"prt_sd_{self.shared_drive.drive_id}"


I couldn't find any notes where we agreed on a prefix with the ITBill team. Do you have anything?

I don't think they had any suggestion for key_remote aside from uniqueness since it's for our use. i think they suggested subscription name be something like GSD_<drive_id>

update page title with tab name, aria label for selects

itbill form in modal, notify template

remove js smoke

…import

…g subscription state

itbill secrets for test.provision deploy

jshint

…to PRT-40-periodic-full-system-reconcile-process

coveralls · 2024-05-20T15:34:32Z

coverage: 77.811% (+1.7%) from 76.106%
when pulling c4aa5ea on PRT-40-periodic-full-system-reconcile-process
into b8e4e88 on develop.

mikeseibel · 2024-05-16T19:07:03Z

endorsement/models/shared_drive.py

-    org_unit_name = models.CharField(max_length=64, null=True)
+    org_unit_name = models.CharField(
+        max_length=64, null=True
+    )  # TODO: shouldn't this be non-NULL?


mikeseibel · 2024-05-16T19:08:28Z

endorsement/models/shared_drive.py

-    drive_id = models.SlugField(max_length=32)
+
+    # NOTE: maximum observed is 19 characters (2024-05-16)
+    drive_id = models.SlugField(max_length=25)


seems fine... i didn't look too hard and felt like some fudge didn't hurt

mikeseibel · 2024-05-16T19:54:38Z

endorsement/models/shared_drive.py

+        """
+        # if changing be sure this does not exceed the max_length of
+        # ITBillSubscription.key_remote
+        key_remote = f"prt_sd_{self.shared_drive.drive_id}"


I don't think they had any suggestion for key_remote aside from uniqueness since it's for our use. i think they suggested subscription name be something like GSD_<drive_id>

mikeseibel · 2024-05-20T14:59:05Z

endorsement/dao/shared_drive.py



-def load_shared_drives_from_csv(file_path):
+def load_shared_drives(google_drive_states):


mikeseibel · 2024-05-20T15:00:29Z

endorsement/dao/shared_drive.py

-        shared_drive=shared_drive)
+        shared_drive=shared_drive,
+        defaults={
+            "datetime_created": dt.datetime.now(dt.timezone.utc),


mikeseibel · 2024-05-20T15:00:56Z

endorsement/dao/shared_drive.py

+    try:
+        defaults["quota_limit"] = a.quota_limit
+    except ValueError:
+        "Drive has no quota set. Probably in invalid org unit."


what should the placeholder for those cases be?

mikeseibel · 2024-05-20T15:01:58Z

endorsement/dao/shared_drive.py

-    def __init__(self, *args, **kwargs):
-        columns = kwargs.get('columns', [])
-        row = kwargs.get('row', [])
+def get_subscription(sdr: SharedDriveRecord):


feel free to junk whats there. i just added it quick and dirty to import ken's early csv.

miker985 added 6 commits May 16, 2024 09:14

Add some testing + installation notes

9fcddd0

blacken endorsement/dao/itbill.py

cc27c30

Add ITBillProvision.current_quantity_gigabytes; blacken models/itbill.py

686c6f2

Expand SharedDrive and SharedDriveRecord models

8b66f0b

Add docstring to Role + MANGAER_ROLE constant blacken models/shared_drive.py Change SharedDrive.drive_id max_length (32 => 25; 19 is largest value) Add SharedDrive.is_deleted Add SharedDriveRecord.get_itbill_key_remote()

load_shared_drive_record now populates datetime_created on creation

5446977

Shared drive loading is now done from MSCA endpoint and not CSV; Add …

1e754ca

…WIP reconciler

miker985 commented May 16, 2024

View reviewed changes

miker985 and others added 16 commits May 16, 2024 11:13

Add migration for SharedDrive.is_deleted

102c3d8

update page title with tab name, aria label for selects

e8f68a0

Merge pull request #569 from uw-it-aca/work/google-shared-drives-mikes

aaeb4a4

update page title with tab name, aria label for selects

itbill form in modal, notify template

1b663db

Merge pull request #570 from uw-it-aca/work/google-shared-drives-mikes

05fd034

itbill form in modal, notify template

remove js smoke

053f86e

Merge pull request #571 from uw-it-aca/work/google-shared-drives-mikes

4037c78

remove js smoke

Provide explicit key_remote to new ITBillSubscription; remove unused …

3c69b60

…import

BUGFIX: quantity => current_quantity; reword comment; simplify settin…

ff685a3

…g subscription state

Implement most of the test cases for shared_drive

6a0abb2

itbill secrets for test.provision deploy

7b0f7da

Merge pull request #572 from uw-it-aca/work/google-shared-drives-mikes

1371d7c

itbill secrets for test.provision deploy

Merge branch 'develop' into feature/google-shared-drives

00cb780

jshint

50af284

Merge pull request #574 from uw-it-aca/work/google-shared-drives-mikes

041a38a

jshint

Resolve merge conflicts and merge in upstream feature branch

644de4a

miker985 added 3 commits May 20, 2024 08:19

Merge remote-tracking branch 'origin/feature/google-shared-drives' in…

9972a83

…to PRT-40-periodic-full-system-reconcile-process

Merge migration scripts per django CI error message

6d862a8

BUGFIX - resolve rename of "name" to "netid" for Member model

3f26483

miker985 requested a review from mikeseibel May 20, 2024 16:45

mikeseibel approved these changes May 20, 2024

View reviewed changes

miker985 and others added 3 commits May 20, 2024 11:39

Add 5 MSCA-related settings to docker/settings.py; all use env vars

301675e

Remove null-ability of shareddrivequota.org_unit_name

e4d6a5f

vault secrets and vars

c4aa5ea

mikeseibel merged commit 9add29b into develop May 20, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prt 40 periodic full system reconcile process #568

Prt 40 periodic full system reconcile process #568

miker985 commented May 16, 2024

miker985 May 16, 2024

mikeseibel May 20, 2024

miker985 May 16, 2024

mikeseibel May 20, 2024

miker985 May 16, 2024

mikeseibel May 20, 2024

miker985 May 20, 2024

miker985 May 16, 2024

mikeseibel May 20, 2024

miker985 May 16, 2024

miker985 May 16, 2024

mikeseibel May 16, 2024

miker985 May 16, 2024

mikeseibel May 16, 2024

miker985 May 16, 2024

mikeseibel May 16, 2024

coveralls commented May 20, 2024 •

edited

Loading

mikeseibel May 16, 2024

mikeseibel May 16, 2024

mikeseibel May 16, 2024

mikeseibel May 20, 2024

mikeseibel May 20, 2024

mikeseibel May 20, 2024

mikeseibel May 20, 2024



		def load_shared_drives_from_csv(file_path):
		def load_shared_drives(google_drive_states):

Prt 40 periodic full system reconcile process #568

Prt 40 periodic full system reconcile process #568

Conversation

miker985 commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented May 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented May 20, 2024 •

edited

Loading