Merge pull request #1526 from gchq/1522-release-0.20

Issue 1522 - Release 0.20

Author: patchwork01

CHANGELOG.md

@@ -4,6 +4,71 @@ Releases
 This page documents the releases of Sleeper. Performance figures for each release
 are available [here](docs/13-system-tests.md#performance-benchmarks)
 
+## Version 0.20.0
+
+*Note: this release contains breaking changes. It is not possible to upgrade from a previous version of Sleeper
+to version 0.20.0*
+
+This contains the following improvements:
+
+Tables:
+
+- Tables are now internally referenced by a unique ID assigned upon creation. This is in preparation for
+  adding the ability to rename tables in the future.
+- Improved support for lots of tables in compaction and ingest status stores by updating the hash key of
+  the DynamoDB tables.
+- Table related infrastructure is now shared between all tables. The following resources are now only deployed once:
+    - Table data bucket.
+    - Table metrics lambda.
+    - State store.
+- Table initialisation is no longer performed by CDK.
+    - A new client class `AddTable` is now responsible for initialising tables.
+- Added configurable timeout property for `TablePropertiesProvider`.
+
+State store:
+
+- The default state store has been updated to the `S3StateStore`.
+- The `minRowKey`, `maxRowKey`, and `rowKeyTypes` fields have been removed from the `FileInfo` class.
+
+Ingest:
+
+- Added instance property to allow setting the S3 upload block size.
+
+Bulk import:
+
+- Added support for overriding spark configuration and platform specification in EMR serverless jobs.
+- Added support for setting the initial capacity of the EMR serverless application.
+- Added support for enabling EMR Studio by using the optional stack `EmrStudioStack`.
+
+Clients:
+
+- The admin client now respects the `EDITOR` environment variable when updating properties.
+- Adding an optional stack in the admin client now uploads docker images if the new stack requires one.
+
+Query:
+
+- Validation failures for queries are now recorded in the `DynamoDBQueryTracker`.
+- Added client to view status of query tracker in `scripts/utility/queryTrackerReport.sh`.
+- Removed inheritance relationship between `Query` and `LeafPartitionQuery`.
+
+Tests:
+
+- Added system tests for using the `S3StateStore`.
+- System tests now purge relevant SQS queues if a test fails.
+- Improved performance of `ingest-runner` module tests.
+- Added system tests with many tables in one instance.
+
+Bugfixes:
+
+- Fixed an issue where the python API would not generate unique IDs for each query.
+- Fixed an issue where the instance ID length was not being validated correctly.
+- Fixed an issue where trying to bulk import using EMR serverless to a table using `S3StateStore` would
+  throw a `NullPointerException`.
+- Fixed an issue where sending an ingest job with a null file would not report the job as invalid.
+- Fixed an issue where the role assumed by tasks in the system test data generation cluster exceeded the maximum size.
+- Fixed an issue where the CDK deployment would fail if an ingest source bucket was not set.
+- Fixed a conflict between temporary directory paths used by the CLI.
+
 ## Version 0.19.0
 
 This contains the following improvements:

docs/13-system-tests.md

@@ -212,3 +212,4 @@ otherwise not have been noticed.
 | 0.17.0         | 09/06/2023 | 308000                      | 163000                           |
 | 0.18.0         | 09/08/2023 | 326000                      | 147000                           |
 | 0.19.0         | 19/09/2023 | 326700                      | 143500                           |
+| 0.20.0         | 20/11/2023 | 318902                      | 137402                           |

java/athena/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <modelVersion>4.0.0</modelVersion>

java/build/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/bulk-import/bulk-import-common/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>bulk-import</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <modelVersion>4.0.0</modelVersion>

java/bulk-import/bulk-import-runner/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <groupId>sleeper</groupId>
         <artifactId>bulk-import</artifactId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/bulk-import/bulk-import-starter/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <groupId>sleeper</groupId>
         <artifactId>bulk-import</artifactId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/bulk-import/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <packaging>pom</packaging>

java/cdk-custom-resources/pom.xml

@@ -21,7 +21,7 @@
     <parent>
         <groupId>sleeper</groupId>
         <artifactId>aws</artifactId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <artifactId>cdk-custom-resources</artifactId>

java/cdk-environment/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/cdk/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/clients/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/common-job/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/compaction/compaction-core/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>compaction</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/compaction/compaction-job-creation/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>compaction</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/compaction/compaction-job-execution/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>compaction</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/compaction/compaction-status-store/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>compaction</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/compaction/compaction-task-creation/pom.xml

@@ -22,7 +22,7 @@
     <parent>
         <groupId>sleeper</groupId>
         <artifactId>compaction</artifactId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <artifactId>compaction-task-creation</artifactId>

java/compaction/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/configuration/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/core/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/distribution/pom.xml

@@ -20,7 +20,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/dynamodb-tools/pom.xml

@@ -21,7 +21,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <modelVersion>4.0.0</modelVersion>

java/garbage-collector/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/ingest/ingest-batcher-core/pom.xml

@@ -21,7 +21,7 @@
     <parent>
         <artifactId>ingest</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/ingest/ingest-batcher-job-creator/pom.xml

@@ -22,7 +22,7 @@
     <parent>
         <groupId>sleeper</groupId>
         <artifactId>ingest</artifactId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <artifactId>ingest-batcher-job-creator</artifactId>

java/ingest/ingest-batcher-store/pom.xml

@@ -22,7 +22,7 @@
     <parent>
         <groupId>sleeper</groupId>
         <artifactId>ingest</artifactId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <artifactId>ingest-batcher-store</artifactId>

java/ingest/ingest-batcher-submitter/pom.xml

@@ -22,7 +22,7 @@
     <parent>
         <groupId>sleeper</groupId>
         <artifactId>ingest</artifactId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
 
     <artifactId>ingest-batcher-submitter</artifactId>

java/ingest/ingest-core/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>ingest</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/ingest/ingest-runner/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>ingest</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/ingest/ingest-starter/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>ingest</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/ingest/ingest-status-store/pom.xml

@@ -21,7 +21,7 @@
     <parent>
         <artifactId>ingest</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/ingest/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/metrics/pom.xml

@@ -20,7 +20,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/parquet/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

java/pom.xml

@@ -21,7 +21,7 @@
     <groupId>sleeper</groupId>
     <artifactId>aws</artifactId>
     <packaging>pom</packaging>
-    <version>0.20.0-SNAPSHOT</version>
+    <version>0.20.0</version>
 
     <modules>
         <module>core</module>

java/query/pom.xml

@@ -19,7 +19,7 @@
     <parent>
         <artifactId>aws</artifactId>
         <groupId>sleeper</groupId>
-        <version>0.20.0-SNAPSHOT</version>
+        <version>0.20.0</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>