retry for LF grant_permissions (#1585)

### Feature or Bugfix
- Bugfix


### Detail
- Add retries to grant_permissions boto3 call in lakeformation client
- timeout is increased by 10% every retry

### Relates
- #1572

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

---------

Co-authored-by: Sofia Sazonova <[email protected]>

backend/dataall/modules/s3_datasets_shares/aws/lakeformation_client.py

@@ -3,6 +3,7 @@
 import time
 
 from botocore.exceptions import ClientError
+from retrying import Retrying
 
 from dataall.base.aws.sts import SessionHelper
 
@@ -160,7 +161,14 @@ def _grant_permissions_to_resource(
                     if permissions_with_grant_options:
                         grant_dict['PermissionsWithGrantOption'] = permissions_with_grant_options
 
-                    response = self._client.grant_permissions(**grant_dict)
+                    response = Retrying(
+                        retry_on_exception=lambda ex: isinstance(
+                            ex, self._client.exceptions.ConcurrentModificationException
+                        ),
+                        stop_max_attempt_number=5,
+                        wait_random_min=1000,
+                        wait_random_max=3000,
+                    ).call(self._client.grant_permissions, **grant_dict)
 
                     log.info(
                         f'Successfully granted principal {principal} '

backend/requirements.txt

@@ -11,4 +11,5 @@ PyYAML==6.0
 requests==2.32.2
 requests_aws4auth==1.1.1
 sqlalchemy==1.3.24
-alembic==1.13.1
+alembic==1.13.1
+retrying==1.3.4