v2.6.1

What's Changed

This release is focused on security enhancements

• Added Token Validations (#1682) + small fix in get-parameter CloudfrontDistributionDomainName from us-east-1 (#1687)
• Add warning to untrust data.all account when removing an environment (#1685)
• Add custom domain support for apigw (#1679)
• Lambda Event Logs Handling (#1678)
• Upgrade Spark version to 3.3 (#1675)
• ES Search Query Collect All Response (#1631)
• Extend Tenant Perms Coverage (#1630)
• Limit Response info dataset queries (#1665)
• Add Removal Policy Retain to Bucket Policy IaC (#1660)
• log API handler response only for LOG_LEVEL DEBUG. Set log level INFO for prod deployments (#1662)
• Add permission checks to markNotificationAsRead + deleteNotification (#1654)
• Added error view and unified utility to check tenant user (#1657)
• Userguide signout flow (#1629)

Full Changelog: v2.6.0...v2.6.1

v2.6.0

What's Changed

New features 🆕

• 🩺 ❗ 🩺 ❗ 🩺 ❗ Adding AWS HealthOmics as a Module in "Play" tools by @ironspur5 in #954
• Allow DA admins to view share logs by @SofiaSazonova in #1274
• Maintenance window by @TejasRGitHub in #1236 and documentation in #1333
• Persistent Email Reminders by @anushka-singh in #1354
• Bulk share reapply on dataset by @TejasRGitHub in #1363
• Convert Dataset Lock Mechanism to Generic Resource Lock by @noah-paige in #1338

Refactoring 💻

• Generic dataset module and specific s3_datasets module by @dlpzx ( #1258 , #1276 , #1281 , #1282 , #1292 , #1297
• Generic shares_base module and specific s3_datasets_shares module by @dlpzx ( #1284 , #1294 , #1298 , #1311 , #1312 , #1320, #1340, #1350, #1351 , #1357 , #1359 )
• Refactoring getStack API by @noah-paige in #1182 and #1344
• Gql schema cleanup sdkcli by @noah-paige in #1330
• Move quicksight monitoring to config.json and disable it in FE by @dlpzx in #1328
• Remove global imports in modules by @dlpzx in #1270

Enhancements 🥇

• Add confirmation pop-ups for deletion of team roles and groups by @SofiaSazonova in #1231
• UI improvement of "Request Access" by @SofiaSazonova in #1228
• ShareView remake by @SofiaSazonova in #1277
• Create RDS database snapshot before executing alembic migrations by @dlpzx in #1267
• Set DataSearch fuzziness to 0 -- strict search by @SofiaSazonova in #1279
• Add dependency of SSM to cognito url trigger by @dlpzx in #1395
• Ignore ruff change in blame by @petrkalos in #1372
• Allow descriptions schema by @noah-paige in #1305
• Update safety check ignore list by @petrkalos in #1310
• Misc logging improvements by @petrkalos in #1317
• Update FE dependency and re-create lock files by @noah-paige in #1326
• Updating encryption for lambda env vars - cont by @mourya-33 in #1322
• Organization Group Permissions Add|Edit by @SofiaSazonova in #1306
• Add support for full or partially updating Config params from SSM by @petrkalos in #1318
• Enhance Share Health Status Verify/ReApply by @noah-paige in #1346
• Split cognito urls setup and cognito user creation by @petrkalos in #1366
• Enforce non null on GQL query string if non null defined by @noah-paige in #1362
• Add search (Autocomplete) in dropdowns by @dlpzx (#1368 , #1356 , #1335 , #1347 , #1367 )
• Rename alias for env_vars kms key in cognito lambdas FE and BE by @dlpzx in #1385
• Add check in delete environment for create_failed stacks by @dlpzx in #1386
• Add delete docs not found when re indexing in catalog task by @noah-paige in #1365
• Introduce check for IAM actions in share_verify bucket and access points + reapply with list of allowed actions by @SofiaSazonova in #1407
• Add cognito urls config trigger func frontend by @noah-paige in #1413

Tests 🧪

• Automate bootstrapping of integrations tests by @petrkalos in #1289
• Codebuild integration tests reads cognito-test-users param from environment account by @petrkalos in #1295
• Add environment tests by @petrkalos in #1371, #1334 and Update gql apis + update_environment tests by @petrkalos in #1348
• Add group/consumption_role invite/remove tests by @petrkalos in #1387
• Add Dataset integration tests - Dataset CRUD + actions outside of data.all by @dlpzx in #1379
• Add Worksheet integration tests - all except run sql query by @dlpzx in #1393
• Add Notebook integration testsby @noah-paige in #1400

Fixes 🪲

• Scope down dataset sharing requester IAM role managed IAM policy S3 permissions by @mourya-33 in #1280
• Fix: timeout error when listing Consumption Roles by @SofiaSazonova in #1303
• Fix: upgrade react avoid ip by @dlpzx in #1308
• Fix: Upgrade Github actions/checkout to v4 by @dlpzx in #1307
• Fix positional args generate env access by @noah-paige in #1316
• Fix s3_datasets and s3_datasets_shares tests by @dlpzx in #1325
• Update profiler run status on Refresh by @SofiaSazonova in #1404
• Share UI Submit fix by @SofiaSazonova in #1403
• Share UI fix: revoke items from share in revoked state by @SofiaSazonova in #1394
• Fix: Env Group Option Forms for create Pipelines and Omic Runs by @noah-paige in #1399
• Fix path deequ jar by @noah-paige in #1402
• Fix/remove edit team modals by @noah-paige in #1412
• Fix error while calling get_cognito_groups function by @TejasRGitHub in #1315
• Fix local dev gql request by @noah-paige in #1337
• Fix get author session API QuickSight by @noah-paige in #1383
• Fix Init Share Base by @noah-paige in #1360
• Fix listOrganizationGroupPermissions by @noah-paige in #1369
• Fix migration to not rely on OrganizationService or RequestContext by @noah-paige in #1361
• Fix: glossary status by @noah-paige in #1373
• Fix lambda_env_key out of scope for vpc-facing cognito setup by @dlpzx in #1384
• Script fix by @SofiaSazonova in #1355
• Fix getOrg query by @petrkalos in #1352
• Fix: Add Maintenance Guard Component separate from AuthGuard by @noah-paige in #1321
• Fix: Extend Sagemaker permissions and fix typo by @noah-paige in #1401
• Fix: Alembic sync by @SofiaSazonova in #1336

Dependencies 📦

• Safety checks - Ignore disputed issue on pip by @dlpzx in #1271
• Bump certifi from 2023.7.22 to 2024.7.4 in /deploy/custom_resources/custom_authorizer by @dependabot in #1390
• Upgrade ejs to 3.1.10 in yarn npm by @dlpzx in #1265
• Bump requests from 2.31.0 to 2.32.0 in /backend by @dependabot in #1291
• Bump requests from 2.31.0 to 2.32.0 in /backend/dataall/base/cdkproxy by @dependabot in #1293
• Bump requests from 2.31.0 to 2.32.2 in /deploy/custom_resources/custom_authorizer by @dependabot in #1309
• Upgrade flask packages to satisfy safety check by @petrkalos in #1313
• Fix npm audit findings by @noah-paige in #1341
• Bump urllib3 from 1.26.18 to 1.26.19 in /deploy/custom_resources/custom_authorizer by @dependabot in #1339
• Update version auth at edge to use node v20 by @noah-paige in #1327

New Contributors

• @ironspur5 made their first contribution in #954

Full Changelog: v2.5.0...v2.6.0

v2.5.0

What's Changed

New features 🆕

• Make visibility of auto-approval toggle configurable based on confidentiality by @anushka-singh in #1223

Refactoring 💻

• Uncouple datasets and dataset_sharing modules by @dlpzx in #1184, #1186, #1185, #1187, #1213, #1214 and #1242
• Refactor core - Stacks by @SofiaSazonova in #1194
• Rename datasets as s3_datasets by @dlpzx in #1250

Enhancements 🥇

• Enable encryption for lambda environment variables by @mourya-33 in #1225
• Add integration tests on a real API client and integrate the tests in CICD by @dlpzx in #1219
• Update lambda_api.py to add encryption for lambda env vars by @mourya-33 in #1255

Fixes 🪲

• Fix Profiling job by @SofiaSazonova in #1222
• Fix Notification link routes to a share request page by @SofiaSazonova in #1227
• Fix listValidEnvironments called only once by @noah-paige in #1238
• Fix Alembic Migration: has table checks by @noah-paige in #1240
• Fix EnvironmentGroup can remove other groups by @SofiaSazonova in #1234
• Fix local test groups listing for listGroups query by @noah-paige in #1239
• Fix DATASET_READ_TABLE read permissions by @SofiaSazonova in #1237
• Add order_by for paginated queries by @noah-paige in #1249
• Explicitly specify dataset_client s3 endpoint_url - fix CORS issue in upload files by @petrkalos in #1260
• Fix TABLE/FOLDER READ shared permissions by @SofiaSazonova in #1259

Dependencies 📦

• Bump werkzeug from 3.0.1 to 3.0.3 in /tests_new/integration_tests by @dependabot in #1254
• Bump werkzeug from 3.0.1 to 3.0.3 in /backend/dataall/base/cdkproxy by @dependabot in #1252
• Bump werkzeug from 3.0.1 to 3.0.3 in /tests by @dependabot in #1253

Full Changelog: v2.4.0...2.5.0

v2.4.0

What's Changed

⚠️ ⚠️ Important: Review the warnings in #1064 if you want to use environments in multiple-regions.

New features 🆕

• Allow multiple environments in the same account with cdk-pivot role by @dlpzx in #1064
• Add high throughput SSM on prod_sizing by @noah-paige in #1154
• Add share_reapply ECS task - ON DEMAND for data.all admins by @dlpzx in #1151
• Initialise RDS database data.all permissions once per deployment by @petrkalos in #1145 and small fix in #1170
• Run RDS database migrations in a custom resource by @petrkalos in #1177
• Ruff code auto-format by @SofiaSazonova in #1105, #1112 and in #1129 and by @petrkalos in #1159 and in #1160

Big Refactoring 💻

• Refactor core/groups by @SofiaSazonova in #1113
• Refactor core/permissions by @SofiaSazonova in #1114
• Refactor core/environment and core/stack by @SofiaSazonova in #1164, in #1169, in #1178 and in #1181

Enhancements 🥇

• Remove allowAll bucket policy statement by @dlpzx in #1106
• Adding check to remove any spaces in confidentiality names by @TejasRGitHub in #1126
• Worksheet UI improvements - fix Team and list Environments of Team by @dlpzx in #1111
• WAF rule parameters in cdk.json + Documentation by @SofiaSazonova in #1140
• Update cdkExecPolicy.yaml to cleanup overly excessive permissions by @mourya-33 in #1085
• Add grants to pivot role in verify tables functions by @dlpzx in #1149
• Implement guardrails and mechanisms to deal with deleted IAM roles in share requests by @SofiaSazonova in #1161
• Implement least privilege principle for cloudfront, lambda and db migration stacks by @mourya-33 in #1134
• Implement less restrictive trust policy for local development pivot roles by @dlpzx in #1176

Fixes 🪲

• Fix EnvUri to check GET_ENV permission for worksheet by @noah-paige in #1125
• Grant IAM permissions to read data to environment team IAM roles independently from CREATE_DATASET permissions by @SofiaSazonova in #1137
• Allow ListEnv to get associated organization information by @noah-paige in #1139
• Redirect the user to correct URL after login by @TejasRGitHub in #1094
• Fixes for email notifications not sending share link in the body by @TejasRGitHub in #1143
• Fix folder pagination missing page by @dlpzx in #1158
• Add "/ "to prefix in crawlers if it is not specified in input by @dlpzx in #1156
• Add Athena List permissions to use AWS SDK for Pandas in SageMaker by @dlpzx in #1155
• Add new data.all permissions REMOVE_ORGANIZATION_GROUP, INVITE_ORGANIZATION_GROUP to teams invited to an Organization by @SofiaSazonova in #1162
• Fix missing GET_FOLDER permissions by @dlpzx in #1163
• Fix input parameters for get credentials get environment group by @dlpzx in #1198
• Update CDK exec role Policy name with region in template by @dlpzx in #1197
• Remove creation of log-groups in Lambdas by @dlpzx in #1192
• Fix missing session in resolve_environment by @dlpzx in #1199
• Fix missing $ in CDK custom policy by @dlpzx in #1204
• Fix unnecessary permission check in resolve_stack functions (failure in list datasets when there are shared datasets) by @dlpzx in #1205
• Fix reference to locationUri by @dlpzx in #1209
• Fix sagemaker tagging permissions by @dlpzx in #1211

Documentation 📚

• Documentation in GitHub pages for release 2.4.0 by @dlpzx in #1191
• Documentation in Userguide for release 2.4 by @dlpzx in #1218

Dependencies 📦

• Upgrade follow-redirects and webpack-dev-middleware depedencies in frontend by @dlpzx in #1121
• Upgrade express in frontend by @dlpzx in #1152
• Bump idna from 3.4 to 3.7 in /deploy/custom_resources/custom_authorizer by @dependabot in #1166

v2.3.0

What's Changed

⚠️ ⚠️ Important: After upgrading to v2.3.0 environment stacks need to be updated before executing data sharing requests. If the environment stack is not data sharing will fail. To update the environment stacks there are 3 options:

1. Using cdk.json parameter enable_update_dataall_stacks_in_cicd_pipeline --> automatically updates the environments and dataset stacks in the CICD pipeline
2. Waiting for overnight update stack task --> same as the above, but it runs at a daily schedule.
3. Updating environments in Environment > Stack tab > click on Update button --> manual update

New features 🆕

• Introduce dataset lock for data sharing, increasing robustness of parallel data sharing by @anushka-singh in #1072
• Add verification of data sharing and reapplying if "unhealthy" by @noah-paige in #1062
• Enable Central Catalog Glue databases import by @TejasRGitHub in #1021 and list them in worksheets in #1079
• Replace IAM inline policies by configurable Managed Policies for folder and bucket sharing by @SofiaSazonova and @dlpzx in #1068
• Simplify LakeFormation Glue database shares - single shared_db and single resource link table by @dlpzx in #1016 and add sharing guardrails drop permissions in #1055 and update Worksheet database names in UI in #1063
• Add data sharing auto-approval option for datasets by @SofiaSazonova in #988
• Introduce feature flags for topics and confidentiality and custom confidentiality list by @TejasRGitHub in #1049

Enhancements 🥇

• Enable key rotation for KMS in CodePipeline by @mourya-33 in #923
• Add support for custom environment linking text with sanitization by @zsaltys in #934
• Add KMS encryption for Aurora DB secrets by @mourya-33 in #935
• Implement Docker user directives by @mourya-33 in #895 and by @noah-paige in #968
• Add checkov GitHub actions by @dlpzx in #962
• Add word-wrap in strings in share lists by @dlpzx in #972
• Add logic to serialize bytes and bytearray datatypes to string by @awskaran in #973
• Add network information to listValidEnvironments by @dlpzx in #986
• Introduce data.all version parameter by @SofiaSazonova in #991
• Add WAF ACL to Cognito User Pool by @noah-paige in #976 and in #1097
• Add checkov baseline by @noah-paige in #1019
• Add dataset Description on shares UI page by @TejasRGitHub in #1026
• Allow update consumption role ownership by @petrkalos in #1020
• Add validation of AWS account and region in environment creation by @dlpzx in #1043
• Remove policies-updater ECS task by @dlpzx in #1046
• Remove git_release functionality by @dlpzx in #1042
• Clean-up auto create pivot permissions by @mourya-33 in #1075
• Add email notification metadata by @TejasRGitHub in #1082
• Add guardrails to alembic sync upgrade/downgrade by @noah-paige in #1084

Fixes 🪲

• Fix reAuth re-renders glitch by @noah-paige in #918
• Fix s3 bucket sharing for federated roles by @zsaltys in #920
• Fix Disappearing Env Value Request Access Modal by @noah-paige in #919
• Fix Frontend Config Role Issue while switching from Cognito Idp to Custom Auth by @TejasRGitHub in #938
• Investigate why some shares did not go to failed state (issue 932), but remained stuck or in-progress by @anushka-singh in #933
• Fix when migrating from Manually Created Pivot Role to Auto Create Pivot Role by @TejasRGitHub in #948
• Validate consumer roles by @SofiaSazonova in #951
• Fix local dev environment is broken after recent changes by @TejasRGitHub in #967
• Bugfix 956 by @anushka-singh in #961
• Add lakeformation in trust policy of dataset role by @dlpzx in #970
• Add else if condition to get tables into InSync state by @TejasRGitHub in #980
• Fix consumption role filtering by @TejasRGitHub in #975
• Replace dataall prefix by resourcePrefix in data pipeline creation by @dlpzx in #985
• Remove AWS Manged Lake Formation Service Linked Role from Pivot Role Nested Stack by @TejasRGitHub in #999
• Fix created dataset naming convention by @noah-paige in #1002
• Add CloudFormation permission to PivotRoleNestedStack by @TejasRGitHub in #1040
• Fix userguide dockerfile by @dlpzx in #1089
• Create DatasetLock for new datasets by @noah-paige in #1090
• Fix verify share table items and access point share no bucket policy by @noah-paige in #1095
• Add check and reapply for attaching S3 IAM policy by @dlpzx in #1096
• Fix counter on paged responses by @petrkalos in #1091
• Handle Error on clean up share and not get stuck in IN_PROGRESS status by @noah-paige in #1099
• Fix issue in SageMaker Create permissions by @dlpzx in #1102

Refactoring 💻

• Refactor Core/Organization to follow api/services/db layers by @dbalintx in #989
• Refactor Core/Vpc refactoring to follow api/services/db layers by @dlpzx in #1044
• Refactor Enums by @SofiaSazonova in #978

Documentation 📚

• Update Userguide documentation for v2.3 updates by @noah-paige in #1100
• Add alembic documentation by @SofiaSazonova in #1033

Dependencies 📦

• Upgrade Aurora postgreSQL engine 11 --> 13 by @noah-paige in #963
• Upgrade axios package to resolve follow-redirect vulnerability by @noah-paige in #952
• Remove unused packages: jinja2, deprecated by @dlpzx in #969
• Upgrade npm packages: axios, css-tools by @dlpzx in #1052
• Upgrade postcss and add yarn resolutions by @dlpzx in #1059
• Applyboto3==1.34.35 in DeployFrontend action by @anandsumit2000 in #1054
• Upgrade starlette version and dependecies to avoid ReDoS by @dlpzx in #1038
• Upgrade ip package in frontend for yarn and npm by @dlpzx in #1070

New Contributors 👨‍💻 👩‍💻

• @SofiaSazonova made their first contribution in #951
• @awskaran made their first contribution in #973
• @petrkalos made their first contribution in #1020
• @anandsumit2000 made their first contribution in #1054

Full Changelog: v2.2.0...v2.3.0

v2.2.0

What's Changed

This time there are no warnings.

New features 🆕

• Enabling S3 bucket share by @anushka-singh in #848
• Support For External IDP and External User Pool Provider by @TejasRGitHub in #897
• Added support for GitHub using AWS CodeStarSourceConnection by @asifma in #834
• BYO VPC in MLStudio by @noah-paige in #894
• New share views by @dlpzx in #885

Enhancements 🥇

• Design better module dependency handling by @maryamkhidir and @dlpzx in #852
• Move feature toggle checker to base by @dbalintx in #833
• Add callback and dependency matrix useclient by @noah-paige in #855
• Update EnvironmentCreateForm.js to combine commands for policy creation and bootstrapping by @mourya-33 in #868
• Add Quicksight Validation on Submit Share by @noah-paige in #873
• Add additional checks for dataset importing by @nikpodsh and @dlpzx in #883
• Add SCP error handling in Quicksight identity region checks by @dlpzx in #896
• Update CodeBuild images to Linux2 standard5.0 (node16 to node18) + Update Docker images to use AmazonLinux:2023 (node18 and Python3.9) by @dlpzx in #889 by @noah-paige in #907
• Changed name button + title to be consistent in UI by @grashopper42 in #888
• Upgrade DDK and Resolve Data.all Pipelines by @noah-paige in #866
• KMS explosion fix (policy optimization) by @anushka-singh in #882

Fixes 🪲

• Add the cloudformation:ContinueUpdateRollback permission to the pivotRole, for administration of linked environment accounts. by @rbernotas in #850
• Fix Module Enabled Pipelines by @noah-paige in #874
• Add Athena:UpdateWorkGroup permissions to CDK Exec Policy by @noah-paige in #892
• Add Pagination to Return Full List Cognito Groups by @noah-paige in #891
• Remove unnecessary MANAGE_ORGANIZATIONS check by @dlpzx in #887
• Fix S3DatasetClient upload data by @noah-paige in #909
• Fix Migration Script for New Deployment by @noah-paige in #908
• Create frontend config role regardless of custom auth or not in backend by @noah-paige in #913
• Fix permissions on share workflows by @dlpzx in #914

Documentation 📚

• Documentation for Setting up External Idp by @TejasRGitHub in #903

Dependencies

• Upgrade Athena engine version to v3 by @dlpzx in #886
• Bump axios from 0.26.1 to 1.6.0 in /frontend by @dependabot in #867
• Bump certifi from 2022.12.7 to 2023.7.22 in /deploy/custom_resources/custom_authorizer by @dependabot in #910
• Bump urllib3 from 1.26.15 to 1.26.18 in /deploy/custom_resources/custom_authorizer by @dependabot in #911
• Bump requests from 2.29.0 to 2.31.0 in /deploy/custom_resources/custom_authorizer by @dependabot in #912

New Contributors 👨‍💻 👩‍💻

• @grashopper42 made their first contribution in #888

Full Changelog: v2.1.0...v2.2.0

v2.1.0

What's Changed

⚠️ Important: After upgrading to v2.1.0 environment stacks need to be updated before creating or editing datasets. If the environment stack is not updated Dataset creation and other functionalities will fail. To update the environment stacks there are 3 options:

1. Using cdk.json parameter enable_update_dataall_stacks_in_cicd_pipeline --> automatically updates the environments and dataset stacks in the CICD pipeline
2. Waiting for overnight update stack task --> same as the above, but it runs at a daily schedule.
3. Updating environments in Environment > Stack tab > click on Update button --> manual update

Governance 🏛️

• Update to Governance Model by @NickCorbett in #736
• Update CONTRIBUTING.md by @NickCorbett in #838
• Create .gitvote.yml by @NickCorbett in #836

New features 🆕

• Re-authorization workflows by @noah-paige in #787
• Email Notification on Share Requests by @TejasRGitHub in #818
• Handle pre-filtering of tables for multiple buckets databases by @anushka-singh in #811
• Limit pivot role S3 permissions by @dlpzx in #780
• Limit pivot role KMS permissions by @dlpzx in #830

Enhancements 🥇

• Fix shell=true semgrep issues by @dlpzx in #760
• Add global flag to replace and avoid scanning issues on incomplete-sanitization by @dlpzx in #762
• Allow to submit a share when you are both an approver and a requester by @zsaltys in #793
• Redirect upon creating a share request by @zsaltys in #799
• Add frontend and backend feature flags by @zsaltys in #817
• Make hosted_zone_id optional by @lorchda in #812
• Add configurable session timeout to Cognito by @manjulaK in #786
• Modularization of notifications, refactor from core to modules by @dlpzx in #822
• Add Additional Error Messages for KMS Key lookup on imported dataset by @noah-paige in #748
• Handle Environment Import of IAM service roles by @noah-paige in #749
• Add condition when there are no public subnets by @lorchda in #794
• Check other share exists before clean up by @noah-paige in #769
• Configure Pytests on Feature Flags by @noah-paige in #764

Fixes 🪲

• Update Lambda runtime from node14 to node16 or node18 and from python3.7 to python3.8 by @nikpodsh in #782
• Build Compliant Names for Opensearch Resources by @noah-paige and made it generic by @dlpzx in #750
• Fix Git branch name length, truncate to 100 chars by @dlpzx in #775
• Fix CodeBuild policy length by @noah-paige in #774
• Fix naming of MLSTUDIO module by @noah-paige in #756
• Fix cdk exec policy for bootstraping linked accounts (#763) by @noah-paige in #768
• Fix external forks for CDK nag by @dlpzx in #767 and in #758
• Fix path of patch_ssm() for pytest fixutre by @noah-paige in #772
• Add Update Permissions to Lambda by @noah-paige in #835

Dependencies

• Add resolutions for yarn.lock pinned packages by @dlpzx in #757
• Upgrade babel to non-vulnerable version 7.23.2 by @dlpzx in #816
• Bump werkzeug from 2.2.3 to 3.0.1 in /tests by @dependabot in #831
• Bump werkzeug from 2.3.3 to 3.0.1 in /backend/dataall/base/cdkproxy by @dependabot in #832
• Bump react-devtools-core from 4.28.0 to 4.28.4 in /frontend by @dependabot in #824

Documentation 📚

• Update architecture diagrams, add region info in deployment pre-requisites and new features of 2.1.0 by @dlpzx, @TejasRGitHub, @lorchda and @noah-paige in #821

New Contributors 👨‍💻 👩‍💻

Special thanks to the new contributors!

• @manjulaK made their first contribution in #786
• @zsaltys made their first contribution in #793
• @anushka-singh made their first contribution in #811
• @TejasRGitHub made their first contribution in #818
• @lorchda made their first contribution in #794

Full Changelog: v2.0.0...v2.1.0

v2.0.0

What's Changed

Major version upgrade ☀️

Data.all v2 is a modular version of data.all that allows customers to easily configure and customize data.all to their needs. In a single config file, the different modules can be configured, enabled or disabled. New features and customizations to the modules can now be added to the source code, as well as complete new modules.

In this release we have carried out a deep refactoring of the backend and frontend packages and the resulting code shows significant differences with the v1.6.2 structure. Refer to the following PRs and issues for more details on the design changes.

• Generic description and motivation
• Backend layout and plug-in architecture
• Frontend layout
• Frontend plug-in architecture

⚠️ Breaking changes?
Upgrading from v1.6.2 to v2 does NOT include any breaking changes. Despite the magnitude of the code changes, there are no changes to the architecture diagram or to existing resources. Pre-existing datasets, environments, shares or any other resources are not affected by the upgrade.

Enhancements and fixes 🪲

• Update auth-at-edge semantic version to latest 2.1.7 by @wolfit in #710
• Update PR template to add security questions by @jorgeig-space in #673
• Add and refine Explicit CDK Execution Policy - Linking Envs by @noah-paige in #667 and in #648
• Fix Dataset Profiling Glue Job by @noah-paige in #649 and in #701
• Fix migration script for v1.2 upgrade by @dlpzx in #651
• Fix delete environment validation on Consumption roles by @noah-paige in #693
• Fix dataset pagination by @noah-paige in #700
• Fix canary user password creation by @dbalintx in #718
• Fix npm version in VPC facing architecture by @dlpzx in #724

Documentation 📚

• Updated GitHub pages in #654 by @dlpzx and @maryamkhidir

Contributors

• data.all V2 contributors: @nikpodsh, @dbalintx, @dlpzx, @itsmo-amzn , @maryamkhidir, @noah-paige and @AmrSaber
• @jorgeig-space made their first contribution in #673
• @wolfit made their first contribution in #710

Full Changelog: v1.6.2...v2.0.0

v1.6.2

What's Changed

⚠️ This is a patch for V1.6.1. If you are upgrading from a previous version of data.all, please have a look at the "Manual actions required" section. Fresh deployments are unaffected.

• Add missing KMS keys for canaries by @dlpzx in #619
• Allow restricted nacls backend VPC by @noah-paige in #626
• Fix cloudfront stack in case custom domain is given by @dbalintx in #607
• resolve unnecessary dependency in git_release role by @dlpzx in #623
• get prefix list ids for dbmigration for infra region by @dlpzx in #624
• Handle External ID SSM v1.6.1> by @noah-paige in #630

Upgrading from <v1.6.0 to v1.6.2

The externalID used to secure the pivotRole(s) in linked environments will be moved from AWS Secrets Manager to AWS Systems Manger Parameter Store as part of this upgrade.

⚠️ NOTE: If you have deployed data.all with enable_pivot_role_auto_create set to true in your cdk.json then you will not have to perform the manual steps listed below and can simply upgrade to v1.6.2. If not please continue with the manual steps below:

In order to retain the same externalID and not have to update the pivotRole(s) of each linked environment, follow the below steps:

1. In your data.all deployment account, Navigate to AWS Secrets Manager and retrieve the secret value of the external ID (named dataall-externalId-{envname}) --> keep this value somewhere for later reference
   

2. Upgrade code from existing version to v1.6.2 and commit latest code changes to deploy via CodePipeline

3. Once the CodePipeline execution is complete, Navigate to SSM Parameter Store in Deployment Account and find externalID Parameter (named /dataall/{envname}/pivotRole/externalId) --> edit the existing value with the one retained from Step 1
   

Full Changelog: v1.6.1...v1.6.2

v2.0.0-beta1

Beta pre-release of version 2.0.0, focused on the refactor to modularize data.all. This version includes a modularized backend but not yet a modularized front-end, which will be published with the final release.

⚠️ We recommend installing this release from scratch instead of upgrading an existing system, since this is a pre-production release.

⚠️ WARNING If upgrading, do so from version 1.6.2.

Known issues affecting deployment

In the deployment guide, run step 8 before step 5, then continue from step 5. This is needed because data.all uses the cdk look up roles in CDK synth, which requires bootstrapping the accounts before running cdk synth locally. Documentation will be updated for the final release.

Known issues

• #556 Request for share is being sent for invalid environment (CREATE_FAILED)
• #540 OpenSearch stack failed during backend deploy due to length of policy name
• #534 Catalog Search along with filters
• #533 Profille Job run fails
• #428 Prefix crawling is crawling complete bucket instead of specific folder
• #374 Error in Monitoring tab in Admin Settings
• #338 Import of Dashboard / Dataset - Environment selection drop-down list is limited to 5 environments
• #288 Can't Paginate to view all Folders
• #625 CDK execution role (custom template) throws S3 access denied error for pivotRole auto-created nested stack
• Denied share requests show the wrong message to the asking user: approved instead of denied (no effect on actual sharing)
• Logging of approvals for sharing shows AWSResourceNotFound for some approvals
• There is an issue when user creates a dataset he/she can’t upload the data using UPLOAD button. We are facing CORS error which disappears after some time
• After creating a dataset, a user may temporarily be unable to upload data using the UPLOAD button

What's Changed

• sync modularization main (frontend) with main by @AmrSaber in #395
• modularization: backend Pluginarchitecture by @nikpodsh in #359
• frontend: simplify dev dockerfile by @AmrSaber in #396
• frontend: make styling consistent and remove dead code by @AmrSaber in #394
• ignore styling commits in git blame by @AmrSaber in #405
• modularization: Dataset Modularization pt.1 by @nikpodsh in #413
• modularization: Datasets modularization pt.2 by @nikpodsh in #432
• modularization: Datasets modularization pt.3 by @nikpodsh in #440
• modularization: Datasets modularization pt.4 by @nikpodsh in #441
• modularization: Datasets modularization pt.5 by @nikpodsh in #442
• modularization: Module dependencies by @nikpodsh in #447
• modularization: Worksheets modularization by @dbalintx in #449
• modularization: Merge from main V1.5.2 to the modularization branch by @nikpodsh in #463
• modularization: MLstudio modularization by @dlpzx in #486
• modularization: testing extensions by @dlpzx in #518
• modularization: Disable and skip module test directories for modules that are inactive by @nikpodsh in #522
• modularization: Dataset Sharing modularization by @nikpodsh in #488
• modularization: Datapipelines modularization by @dbalintx in #457
• Dashboard modularization by @nikpodsh in #537
• feat: Redshift removal by @dbalintx in #551
• Fix methods without permissions by @nikpodsh in #547
• feat:increase memory limit for local frontend container by @dbalintx in #568
• Change the pipeline for the modularization by @nikpodsh in #545
• Add generic way to toggle data.all features by @blitzmohit in #538
• Refactoring of aws calls by @nikpodsh in #550
• Migrate to a new permission checker by @nikpodsh in #569
• Core modularization by @nikpodsh in #592
• Merge main into modularization-main branch by @nikpodsh in #595
• Resolve modularization inconsistencies by @dbalintx in #605
• Modularization-main bugfixes by @dbalintx in #604
• Frontend - Module Enablement by @itsmo-amzn in #602
• Fixing of linting error by @nikpodsh in #627
• Fix assume role for the fresh account by @nikpodsh in #628

New Contributors

• @blitzmohit made their first contribution in #538

Full Changelog: v1.6.1...v2.0.0-beta1