Merge from main-3 #5955

corneil · 2024-09-27T13:33:24Z

Resolved compile issues, test and integration-tests.
JUnit 5, Hamcrest -> AssertJ
Spring Boot 3.x best practice.

NOTE: Security Integration test fails since PR wasn't merged.

* Jakarta updates * Updated for Logging Date Format * Replace spring.factories with autoconfig imports * Constructor Bindings removed * Readd the yaml files that caused the migrator to fail * Update SCDF to Boot 3.2.2 as parent * Updated dependencies where needed * Added versions to some dependencies where versions are no longer in the bom * Updated hibernate usages where class names changed * Got to spring-cloud-common-security-config-web. ** The OauthSecurityCOnfiguration has @ConditionalOnMissingBean for the WebSecurityAdapter. ** Which is no longer available. We need to determine what should go here. * Update workflows to use Java 17

@type

* Replaced SocketUtils with TestSocketUtils ** It was moved to the test package. * Migrated httpclient to httpclient5 ** Removed use of httpclient 4.x dependencies added yesterday * Updated Types in AuditRecord Entity to use JdbcTypeCode * Security Modules need to be compiled with Boot3 and securty 6 * Update SCDF to use deployer 3.0.x * Update SCDF pom files to create Java 17 jars * Exclude javax.annotation from deployer artifacts. * Add jakarta annotation dependencies * Update code from javax to jakarta * Replaced Entity @type with @JavaTypeCode * Update JobParam to batch 5. * Update httpclient package to httpclient5 package

@cbono

Added dataflow specific PageQueryProvider for missing functions Batch removed some methods that are still required by dataflow. Created dataflow version of those classes so that we can implement those methods Added hibernate version Checkpoint for server core and server Updated Skipper components Update classic docs to be restful doc 3.0 compliant Update code to resolve review comments Thank you @cbono!!!! One last polish

This commit removes the following dependency overrides that were put in place to override earlier dependencies in Boot 2.x: - logback - jackson - snakeyaml Also, the joda-time version is updated to `2.12.7`

…mon/**` Update Awaitility and Semver libs This commit removes the dependency management for `com.jayway.awaitility` in favor of the Spring Boot managed version at the new `org.awaitility` coordinates. Also, the Semver lib is updated to 0.10.2

Dependency management of `testcontainers-bom` is no longer needed as Spring Boot 3.x provides the version for the `testcontainers-bom`.

A previous commit removed the dependency overrides for snakeyaml in `spring-cloud-dataflow-parent` (which ~50% of the modules extend from). This commit removes the dependency overrides for snakeyaml in `spring-cloud-dataflow-build-dependencies` (the remaining 50% of modules extend from this).

Dependency management of `junit-jupiter-bom` is no longer needed as Spring Boot 3.x provides the version for the `junit-jupiter-bom`.

The version was previously overridden to `2.4.11` due to CVE. This is no longer needed as Spring Boot provides the version of `2.5.0`.

Dependency management of `postgresql` is not needed as Spring Boot 3.x provides the version for `postgresql`.

Also had to account for some changes in Mockito w/ varargs methods. Suggested changes from PR review

@configuration

Fix from PR review * Add back missing nested @configuration annotations

The new HTTP client requires access to the host:port. As such, we now use @SpringBootTest to launch web server to allow test to proceed. Also include H2 test dependency to deal w/ test startup failure.

This commit excludes DataSourceAutoConfiguration from the tests.

This commit updates to Open API 2.x which is the version that supports Boot 3. The SpringDocs migration guide was followed: https://springdoc.org/#migrating-from-springdoc-v1

Add recommendation from vscode itself not that we're a monorepo lsp needs more memory.

Prior to this commit the sink used a PollableMessageSource with a trigger to source its inbound messages. This commit replaces that mechanism w/ a simple SCSt consumer with a @StreamRetryTemplate.

@configuration

We had an issue where some of the auto configurations were not firing. This is because when a user specifies the resource tag in maven, the boot plugin expects the user to fill the resources manually vs. Its default behavior Also removed the version from the H2 dependency so that it can be managed by the bom Removed unnecessary @configuration annotaions where a @autoConfiguration annotation is present

Currently SCDF fails to start with errors around metrics. These will need to be re-added when metric migration begins Remove debug settings from previous commit

…oud#5679) * Stub out DataflowPagingProviders to allow Dataflow to start * Provide PlatformTransactionManager to JobExplorer since Batch 5 no longer provides it * Compile with `-parameters` option

For each database type the following migration was implemented: Prefix the TASK V2 and BATCH V4 tables with V2_ . This allows user to determine what they wish to do with this data Remove BOOT3_prefix for TASK V2 and BATCH V5 tables Make sure that these migrations were supported by flyway Make sure that the Migration SQL scripts were added to associated yaml files. What was affected The following databases were migrated: H2 MariaDB Mysql Postgres Oracle SQLServer DB2 Types of migrations There were 3 types of migrations that occurred. Default - These types of migrations were typically alter tables and alter sequences. The following databases belong to this group: a. MariaDB b. Mysql c. Postgres d. Oracle (with some exceptions) e. Sql Server (commands different than others, but principles remained). In-Memory - For this case the Task V2 and Batch V4 tables/sequences DDL was removed and BOOT3_ prefix was removed. The only in-memory we support is H2. DB2 - Gets its own category To rename the tables db2 requires all primary and foreign keys to be dropped. In this case we decided to create the new tables and copy the contents of the original tables to the new tables. Then remove the original. This was to avoid errors when recreating the keys How to eat this Elephant This PR is a bit large so let's discuss how we can handle this review. Let's review this by sampling one database from each type of migration. This is so that we can make sure the general pattern works for folks. Default type, look at Mariadb In Memory type look at H2 DB2 Type look at (well... ) DB2 :-D After we finish this sampling review, those changes will be applied to the other databases and then a full review can be made.

…pring-cloud#5680) This commit does not cover all the cases where updated dependencies broke existing tests. The main ones resolved here are some basics on Hibernate dialect versions, batch 5 updates, and some basic removal of BOOT3/BOOT2 additions in 2.11.x There are other updates that are causing alot of failures. Here are the issues and a brief description: * Currently Hibernate 6 does not create sequences out of the box. More can be read here: https://github.com/hibernate/hibernate-orm/blob/6.0/migration-guide.adoc#implicit-identifier-sequence-and-table-name One test that shows this error:TabOnTapCompletionProviderTests * Hibernate Dialect. Some tests require the hibernate dialects to be updated to hibernate 6. I did resolve the ones I found. But may have missed others. * Many tests are failing trying with the following exception: `No bean named 'mvcConversionService' available` For example: SpringDocAutoConfigurationTests.enabledWithCustomSpringDocSettings * The TODO excluding wavefront are causing some errors for example: Error creating bean with name 'management.wavefront-org.springframework.boot.actuate.autoconfigure.wavefront.WavefrontProperties': Lookup method resolution failed An Example can be found here: SpringDocIntegrationTests.disabledByDefault

Hibernate 6.0 now creates a sequence per entity hierarchy instead of a single hibernate_sequence. This commit sets the `hibernate.id.db_structure_naming_strategy` property to `single` to preserve the previous behavior of using a single hibernate_sequence.

This commit adds a mock "mvcConversionService" bean to the app context for the SpringDocAutoConfigurationTests as SpringDoc 2.x expects this bean to be available. This commit also updates the use of Mockito verify for varargs in the SpringDocAutoConfigurationTests.

Align wavefront version so that we don't have misaligned versions coming out from other parts of a metric system. Short story is that boot doesn't manage wavefront but there is an explicit dependency to wavefront sdk libs in metric system. Rename metric options within management for influx, prometheus and wavefront to align changes in boot itself. For rsocket proxy keep old management.metrics.export.prometheus.rsocket.enabled as that is going to get moved under micrometer spesific namespace when they release boot3 support. This commit was supposed to get skipper server to start but there is a new issue about missing bean org.springframework.statemachine.data.jpa.JpaStateMachineRepository which will need to get fixed in a separate commit. Fixes spring-cloud#5675

Step 1. Make sure to remove the version from the docker compose. It is no longer needed and causes older versions of docker to fail Step 2. Update compose files to use the latest version of SCDF 3.x instead of 2.11.x Step 3. Update build image script so that uses java 17 when creating containers Update the DataFlowIT and the Abstract classes it is built on so that multipleComposedTaskWithArguments test passes. Notice that JobParameterJacksonDeserializer and JobParametersJacksonMixIn have been updated. These changes mirror those in spring-cloud#5850. These were required for the test to pass. At the time this PR is merged we can merge accepting those from spring-cloud#5850. Provide docs on how SCDF images are created and pushed Also update the DEFAULT_JDK to Java 17 Update PR based on code review comments * Added log message in case a JobParameter Type is invalid * cleaned up workflow.adoc

* Update stream apps version refs to 5.0.0 * Update task apps version refs to 3.0.0 See spring-cloud#5897

This commit removes temporary auto-config exclusions that were in place to allow compiling against Boot 3.x prior to migrating to the newer Observation API.

…ng-cloud-deployer#465 [skip ci] Update paragraph for multiple Init Containers. Fix mariadb mount to be /var/lib/mysql (spring-cloud#5875) Fixes spring-cloud#5877 Add spaces after columns for clarity. This is done during merge

Currently the migration fails with a validation error. The cause of the validation error was that some commits from 2.11.x that contained flyway migration scripts were not ported to the main-3 branch. The following commits that contained flyway migrations were migrated to main-3 from the main branch. * 62ea6c5 * 6f97589 The goal of this PR is to resolve the validation error so DB migrations will work properly. A subsequent PR will be submitted will add the feature code for commit 6f97589.

* Added ThinTaskExecution to store ctrStatus. Update Controller and Service to populate the ctr status. Fixes spring-cloud#5907 * Updated for comments.

This commit re-enables the `MariadbSharedDbIT` and disables the Java 21 test variant as Java 21 container images are not yet supported.

* Update Prometheus Rsocket property name This property rename was missed during the initial update to Prometheus Rsocket 2.0.0-M1 (b61c3c9). * Update tracing props in Docker Compose files This updates the Wavefront and Zipkin docker compose templates with the properly named tracing related properties to enable traces exported for each.

This commit adds the global tracing property 'management.tracing.enabled' to the Zipkin and Wavefront docker compose files. Also, the Wavefront docker compose file excludes the Zipkin tracing auto-configuration and the Zipkin docker compose file excludes the Wavefront tracing auto-configuration.

cppwfs

@corneil Thanks for all the hard work here! Most of the comments are nitpicks.

.github/workflows/build-snapshot-controller.yml

build-containers.sh

spring-cloud-common-security-config/pom.xml

cppwfs · 2024-09-30T19:35:46Z

spring-cloud-common-security-config/spring-cloud-common-security-config-web/pom.xml

 	</properties>
 	<dependencies>
 		<dependency>
 			<groupId>org.springframework.cloud</groupId>
 			<artifactId>spring-cloud-common-security-config-core</artifactId>
-			<version>${project.version}</version>
+			<version>3.0.0-SNAPSHOT</version>


We can switch this back to project version.

...-web/src/main/java/org/springframework/cloud/common/security/OAuthSecurityConfiguration.java

...est/java/org/springframework/cloud/dataflow/server/rest/documentation/BaseDocumentation.java

...rg/springframework/cloud/dataflow/server/rest/documentation/TaskExecutionsDocumentation.java

spring-cloud-dataflow-common/spring-cloud-dataflow-common-test-docker-junit5/pom.xml

...pringframework/cloud/dataflow/common/test/docker/compose/execution/DefaultDockerCompose.java

...ava/org/springframework/cloud/dataflow/common/test/docker/compose/connection/PortsTests.java

.github/workflows/build-snapshot-controller.yml

.github/workflows/ci-carvel.yml

.../springframework/cloud/dataflow/common/test/docker/compose/execution/DockerComposeTests.java

cppwfs · 2024-10-01T11:53:39Z

.../springframework/cloud/dataflow/common/test/docker/compose/execution/DockerComposeTests.java

-                toInputStream("id"),
-                toInputStream("docker-compose version 1.5.6, build 1ad8866"),
-                toInputStream("logs"));
+                IOUtils.toInputStream("id"),


You maybe able to import IOUtils.toInputStream as a static and not have to prefix these with IOUtils..

...framework/cloud/dataflow/common/test/docker/compose/execution/DockerComposeVersionTests.java

cppwfs · 2024-10-01T12:08:12Z

.../org/springframework/cloud/dataflow/common/test/docker/compose/logging/LogDirectoryTest.java

    }

+//    @Test


This commented out tests and imports can be deleted.

cppwfs · 2024-10-01T13:49:52Z

...n/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports

@@ -0,0 +1,2 @@
+org.springframework.cloud.dataflow.server.config.kubernetes.KubernetesSchedulerAutoConfiguration


We can remove the spring.factories file that this imports replaced. I think its removal was missed in main-3.

cppwfs · 2024-10-01T13:55:55Z

...main/java/org/springframework/cloud/dataflow/registry/service/DefaultAppRegistryService.java

@@ -404,13 +401,14 @@ private AppRegistration createAppRegistrations(Map<String, AppRegistration> regi
 		if (!registrations.containsKey(key) && registrations.containsKey(type + name + "latest")) {
 			key = type + name + "latest";
 		}
+		// Allow bootVersion in descriptor file (already in 5.0.x stream app descriptor)
 		if("bootVersion".equals(extra)) {


Just curious do we still need this segment?

cppwfs · 2024-10-01T13:56:54Z

...java/org/springframework/cloud/dataflow/registry/service/DefaultAppRegistryServiceTests.java

@@ -16,14 +16,22 @@

 package org.springframework.cloud.dataflow.registry.service;

+import static org.assertj.core.api.Assertions.assertThat;


We can probably move the static imports below the other imports.

cppwfs · 2024-10-01T14:21:28Z

...ient/src/test/java/org/springframework/cloud/dataflow/rest/client/DataflowTemplateTests.java

@@ -15,16 +15,21 @@
 */
 package org.springframework.cloud.dataflow.rest.client;

+import static org.assertj.core.api.Assertions.assertThat;


We can move these static imports to below the other imports.

cppwfs · 2024-10-01T14:22:31Z

.../java/org/springframework/cloud/dataflow/rest/client/ExecutionContextSerializationTests.java

 import static org.assertj.core.api.Assertions.assertThat;

+import org.springframework.batch.item.ExecutionContext;


We can move these static imports to below the other imports.

Added TODO on comments.

onobc and others added 30 commits January 24, 2024 13:47

Update build files for main-3 [skip-ci]

ca48ed3

Remove dependency overrides from 2.11.x (Boot 2.x)

5182aec

This commit removes the following dependency overrides that were put in place to override earlier dependencies in Boot 2.x: - logback - jackson - snakeyaml Also, the joda-time version is updated to `2.12.7`

Remove <dependencyManagement> of testcontainers-bom

836042c

Dependency management of `testcontainers-bom` is no longer needed as Spring Boot 3.x provides the version for the `testcontainers-bom`.

Remove <dependencyManagement> of junit-jupiter-bom

ee6e456

Dependency management of `junit-jupiter-bom` is no longer needed as Spring Boot 3.x provides the version for the `junit-jupiter-bom`.

Remove dependency override of json-smart.

0dd0b5c

The version was previously overridden to `2.4.11` due to CVE. This is no longer needed as Spring Boot provides the version of `2.5.0`.

Remove <dependencyManagement> of postgresql

13a8854

Dependency management of `postgresql` is not needed as Spring Boot 3.x provides the version for `postgresql`.

Remove unused spring.version property

093105b

Update DockerCompose tests to Junit5

39c1bef

Also had to account for some changes in Mockito w/ varargs methods. Suggested changes from PR review

Use AutoConfiguration.imports for all auto-configs

27d9941

Fix from PR review * Add back missing nested @configuration annotations

Fix startup and EPP ordering (spring-cloud#5670)

de58247

Fix tests for rest-resource (SCDF3 migration)

04224c4

The new HTTP client requires access to the host:port. As such, we now use @SpringBootTest to launch web server to allow test to proceed. Also include H2 test dependency to deal w/ test startup failure.

Fix tests for rest-client (SCDF3 migration)

6bfe953

This commit excludes DataSourceAutoConfiguration from the tests.

Update SpringDoc OpenAPI to latest 2.x

e12c62c

This commit updates to Open API 2.x which is the version that supports Boot 3. The SpringDocs migration guide was followed: https://springdoc.org/#migrating-from-springdoc-v1

Change vscode settings

317a4b8

Add recommendation from vscode itself not that we're a monorepo lsp needs more memory.

Use config legacy processing in Skipper (spring-cloud#5673)

a37cf87

Refactor TaskLauncherSink to SCSt Consumer

9919888

Prior to this commit the sink used a PollableMessageSource with a trigger to source its inbound messages. This commit replaces that mechanism w/ a simple SCSt consumer with a @StreamRetryTemplate.

Add metrics exclusions till metric migration is complete

72e9bdb

Currently SCDF fails to start with errors around metrics. These will need to be re-added when metric migration begins Remove debug settings from previous commit

Stub out DataflowPagingProvider to allow Dataflow to start (spring-cl…

5bf0033

…oud#5679) * Stub out DataflowPagingProviders to allow Dataflow to start * Provide PlatformTransactionManager to JobExplorer since Batch 5 no longer provides it * Compile with `-parameters` option

Add skipper to vscode launch config

22df7b4

cppwfs and others added 15 commits August 20, 2024 13:52

Remove .jdk8 files. (spring-cloud#5900)

791ff60

Update apps version refs to 5.0.0 / 3.0.0 (spring-cloud#5896)

b1028e6

* Update stream apps version refs to 5.0.0 * Update task apps version refs to 3.0.0 See spring-cloud#5897

Update with merge from main-3

d9f39ea

Update with merge from main-3

2a08167

Remove metrics auto-config exclusions (spring-cloud#5902)

b515672

This commit removes temporary auto-config exclusions that were in place to allow compiling against Boot 3.x prior to migrating to the newer Observation API.

Forward port ctr status on Thin Controller. (spring-cloud#5906)

65989eb

* Added ThinTaskExecution to store ctrStatus. Update Controller and Service to populate the ctr status. Fixes spring-cloud#5907 * Updated for comments.

Enable MariadbSharedDbIT (spring-cloud#5908)

9264222

This commit re-enables the `MariadbSharedDbIT` and disables the Java 21 test variant as Java 21 container images are not yet supported.

Fix substitution in github-release.yml [skip ci]

b983044

Merging and resolved errors and test failures.

646f1e7

corneil requested review from ilayaperumalg, cppwfs and onobc September 27, 2024 13:33

corneil added 2 commits September 30, 2024 09:53

Updates for JUnit5 and AssertJ

7a8f337

CI requires Java 17.

5343bc1

corneil force-pushed the main-target branch from 3079439 to 5343bc1 Compare September 30, 2024 15:56

cppwfs requested changes Sep 30, 2024

View reviewed changes

ilayaperumalg reviewed Oct 1, 2024

View reviewed changes

.github/workflows/build-snapshot-controller.yml Outdated Show resolved Hide resolved

ilayaperumalg reviewed Oct 1, 2024

View reviewed changes

.github/workflows/build-snapshot-controller.yml Outdated Show resolved Hide resolved

ilayaperumalg reviewed Oct 1, 2024

View reviewed changes

.github/workflows/ci-carvel.yml Outdated Show resolved Hide resolved

Updated based on comments.

6951b68

cppwfs approved these changes Oct 1, 2024

View reviewed changes

corneil added 2 commits October 1, 2024 17:46

Updated based on comments.

5e0bd9e

Added TODO on comments.

Fixed imports

54e9b81

corneil merged commit b982eed into spring-cloud:main Oct 2, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge from main-3 #5955

Merge from main-3 #5955

corneil commented Sep 27, 2024 •

edited

Loading

cppwfs left a comment

cppwfs Sep 30, 2024

cppwfs Oct 1, 2024

cppwfs Oct 1, 2024

cppwfs Oct 1, 2024

cppwfs Oct 1, 2024

cppwfs Oct 1, 2024

cppwfs Oct 1, 2024

cppwfs Oct 1, 2024

		@@ -0,0 +1,2 @@
		org.springframework.cloud.dataflow.server.config.kubernetes.KubernetesSchedulerAutoConfiguration

		@@ -16,14 +16,22 @@

		package org.springframework.cloud.dataflow.registry.service;

		import static org.assertj.core.api.Assertions.assertThat;

		import static org.assertj.core.api.Assertions.assertThat;

		import org.springframework.batch.item.ExecutionContext;

Merge from main-3 #5955

Merge from main-3 #5955

Conversation

corneil commented Sep 27, 2024 • edited Loading

cppwfs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corneil commented Sep 27, 2024 •

edited

Loading