Skip to content

Commit 997b483

Browse files
MartijnVisserMartijnVisser
authored
[FLINK-36181] Use Java 17 by default (#25898)
* [FLINK-36181] Set CI to use Java 17 by default and drop Java 8 * [FLINK-36181] Update main POM to use Java 17 * [FLINK-36181] Bump Scala to 2.12.20 This is a breaking change for Flink 2.0, and allows us to next remove the specific Scala versions for both Java 17 and 21 * [FLINK-36181] Build Dockerfiles with Java 17 by default * [FLINK-36181] Remove Scala version bump for Java11 and Java17 profiles Since we've bumped Scala all together, we no longer need to bump it for these profiles * [FLINK-36181] Add Java11 target profile * [FLINK-36181] Set CI to have a dedicated Java11 cron * [FLINK-36181] Update README * [FLINK-36181] Make sure that JDK11 APIs can be used * [FLINK-36181] Improve README * [FLINK-36181] Address review comment on configurable Java version * [FLINK-36181] Remove no longer required specific Scala parameters
1 parent 8af0259 commit 997b483

File tree

12 files changed

+100
-68
lines changed

12 files changed

+100
-68
lines changed

‎.github/workflows/ci.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,11 @@ jobs:
2929
name: "Pre-compile Checks"
3030
uses: ./.github/workflows/template.pre-compile-checks.yml
3131
ci:
32-
name: "Default (Java 11)"
32+
name: "Default (Java 17)"
3333
uses: ./.github/workflows/template.flink-ci.yml
3434
with:
35-
environment: 'PROFILE="-Dinclude_hadoop_aws"'
36-
jdk_version: 11
35+
environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk17 -Pjava17-target"'
36+
jdk_version: 17
3737
secrets:
3838
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
3939
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}

‎.github/workflows/nightly.yml

Lines changed: 5 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -39,23 +39,12 @@ jobs:
3939
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
4040
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
4141
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
42-
java17:
43-
name: "Java 17"
44-
uses: ./.github/workflows/template.flink-ci.yml
45-
with:
46-
workflow-caller-id: java17
47-
environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Pjava17-target"'
48-
jdk_version: 17
49-
secrets:
50-
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
51-
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
52-
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
5342
java21:
5443
name: "Java 21"
5544
uses: ./.github/workflows/template.flink-ci.yml
5645
with:
5746
workflow-caller-id: java21
58-
environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Djdk21 -Pjava21-target"'
47+
environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk21 -Pjava21-target"'
5948
jdk_version: 21
6049
secrets:
6150
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
@@ -66,8 +55,8 @@ jobs:
6655
uses: ./.github/workflows/template.flink-ci.yml
6756
with:
6857
workflow-caller-id: hadoop313
69-
environment: 'PROFILE="-Dflink.hadoop.version=3.2.3 -Phadoop3-tests,hive3"'
70-
jdk_version: 11
58+
environment: 'PROFILE="-Dflink.hadoop.version=3.2.3 -Phadoop3-tests,hive3 -Djdk17 -Pjava17-target"'
59+
jdk_version: 17
7160
secrets:
7261
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
7362
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
@@ -77,8 +66,8 @@ jobs:
7766
uses: ./.github/workflows/template.flink-ci.yml
7867
with:
7968
workflow-caller-id: adaptive-scheduler
80-
environment: 'PROFILE="-Penable-adaptive-scheduler"'
81-
jdk_version: 11
69+
environment: 'PROFILE="-Penable-adaptive-scheduler -Djdk17 -Pjava17-target"'
70+
jdk_version: 17
8271
secrets:
8372
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
8473
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}

‎.github/workflows/template.flink-ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ on:
3030
type: string
3131
jdk_version:
3232
description: "The Java version to use."
33-
default: 11
33+
default: 17
3434
type: number
3535
secrets:
3636
s3_bucket:

‎.github/workflows/template.pre-compile-checks.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,15 +23,15 @@ on:
2323
inputs:
2424
jdk_version:
2525
description: "The JDK version that shall be used as a default within the Flink CI Docker container."
26-
default: "11"
26+
default: "17"
2727
type: choice
2828
options: ["11", "17", "21"]
2929

3030
workflow_call:
3131
inputs:
3232
jdk_version:
3333
description: "The JDK version that shall be used as a default within the Flink CI Docker container."
34-
default: 11
34+
default: 17
3535
type: number
3636

3737
permissions: read-all

‎README.md

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -104,15 +104,46 @@ Prerequisites for building Flink:
104104
* Unix-like environment (we use Linux, Mac OS X, Cygwin, WSL)
105105
* Git
106106
* Maven (we require version 3.8.6)
107-
* Java 11
107+
* Java (version 11, 17, or 21)
108+
109+
### Basic Build Instructions
110+
111+
First, clone the repository:
108112

109113
```
110114
git clone https://github.com/apache/flink.git
111115
cd flink
112-
./mvnw clean package -DskipTests # this will take up to 10 minutes
113116
```
114117

115-
Flink is now installed in `build-target`.
118+
Then, choose one of the following commands based on your preferred Java version:
119+
120+
**For Java 11**
121+
122+
```
123+
./mvnw clean package -DskipTests -Djdk11 -Pjava11-target
124+
```
125+
126+
**For Java 17 (Default)**
127+
128+
```
129+
./mvnw clean package -DskipTests -Djdk17 -Pjava17-target
130+
```
131+
132+
**For Java 21**
133+
134+
```
135+
./mvnw clean package -DskipTests -Djdk21 -Pjava21-target
136+
```
137+
138+
The build process will take approximately 10 minutes to complete.
139+
Flink will be installed in `build-target`.
140+
141+
### Notes
142+
143+
* Make sure your JAVA_HOME environment variable points to the correct JDK version
144+
* The build command uses Maven wrapper (mvnw) which ensures the correct Maven version is used
145+
* The -DskipTests flag skips running tests to speed up the build process
146+
* Each Java version requires its corresponding profile (-Pjava<version>-target) and JDK flag (-Djdk<version>)
116147

117148
## Developing Flink
118149

‎azure-pipelines.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,10 +76,10 @@ stages:
7676
vmImage: 'ubuntu-22.04'
7777
e2e_pool_definition:
7878
vmImage: 'ubuntu-22.04'
79-
environment: PROFILE="-Dflink.hadoop.version=2.10.2"
79+
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk17 -Pjava17-target"
8080
run_end_to_end: false
8181
container: flink-build-container
82-
jdk: 11
82+
jdk: 17
8383
- job: docs_404_check # run on a MSFT provided machine
8484
pool:
8585
vmImage: 'ubuntu-22.04'

‎flink-dist-scala/src/main/resources/META-INF/NOTICE

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ This project bundles the following dependencies under the Apache Software Licens
1010

1111
The following dependencies all share the same BSD license which you find under licenses/LICENSE.scala.
1212

13-
- org.scala-lang:scala-compiler:2.12.7
14-
- org.scala-lang:scala-library:2.12.7
15-
- org.scala-lang:scala-reflect:2.12.7
16-
- org.scala-lang.modules:scala-xml_2.12:1.0.6
13+
- org.scala-lang:scala-compiler:2.12.20
14+
- org.scala-lang:scala-library:2.12.20
15+
- org.scala-lang:scala-reflect:2.12.20
16+
- org.scala-lang.modules:scala-xml_2.12:2.3.0

‎flink-end-to-end-tests/test-scripts/common_docker.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,10 +46,10 @@ function build_image() {
4646
start_file_server
4747
local server_pid=$!
4848

49-
echo "Preparing Dockeriles"
49+
echo "Preparing Dockerfiles"
5050
retry_times_with_exponential_backoff 5 git clone https://github.com/apache/flink-docker.git --branch dev-master --single-branch
5151

52-
local java_version=11
52+
local java_version=17
5353
if [[ ${PROFILE} == *"jdk17"* ]]; then
5454
java_version=17
5555
fi

‎flink-rpc/flink-rpc-akka/pom.xml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,6 @@ under the License.
3838

3939
<properties>
4040
<pekko.version>1.1.2</pekko.version>
41-
<scala.binary.version>2.12</scala.binary.version>
42-
<scala.version>2.12.16</scala.version>
4341
</properties>
4442

4543
<dependencies>

‎flink-table/flink-table-planner-loader-bundle/src/main/resources/META-INF/NOTICE

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,6 @@ The Apache Software Foundation (http://www.apache.org/).
66

77
The following dependencies all share the same BSD license which you find under licenses/LICENSE.scala.
88

9-
- org.scala-lang:scala-compiler:2.12.7
10-
- org.scala-lang:scala-library:2.12.7
11-
- org.scala-lang:scala-reflect:2.12.7
9+
- org.scala-lang:scala-compiler:2.12.20
10+
- org.scala-lang:scala-library:2.12.20
11+
- org.scala-lang:scala-reflect:2.12.20

‎pom.xml

Lines changed: 29 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,8 @@ under the License.
124124
<flink.shaded.jackson.version>2.15.3</flink.shaded.jackson.version>
125125
<flink.shaded.jsonpath.version>2.7.0</flink.shaded.jsonpath.version>
126126
<flink.markBundledAsOptional>true</flink.markBundledAsOptional>
127-
<target.java.version>11</target.java.version>
127+
<source.java.version>11</source.java.version>
128+
<target.java.version>17</target.java.version>
128129
<slf4j.version>1.7.36</slf4j.version>
129130
<log4j.version>2.24.1</log4j.version>
130131
<!-- Overwrite default values from parent pom.
@@ -133,9 +134,7 @@ under the License.
133134
<maven.compiler.source>${target.java.version}</maven.compiler.source>
134135
<maven.compiler.target>${target.java.version}</maven.compiler.target>
135136
<scala.macros.version>2.1.1</scala.macros.version>
136-
<!-- Default scala versions, must be overwritten by build profiles, so we set something
137-
invalid here -->
138-
<scala.version>2.12.7</scala.version>
137+
<scala.version>2.12.20</scala.version>
139138
<scala.binary.version>2.12</scala.binary.version>
140139
<chill.version>0.7.6</chill.version>
141140
<!-- keep FlinkTestcontainersConfigurator.configureZookeeperContainer in sync -->
@@ -944,7 +943,7 @@ under the License.
944943
<profile>
945944
<id>scala-2.12</id>
946945
<properties>
947-
<scala.version>2.12.7</scala.version>
946+
<scala.version>2.12.20</scala.version>
948947
<scala.binary.version>2.12</scala.binary.version>
949948
</properties>
950949
<activation>
@@ -1064,15 +1063,34 @@ under the License.
10641063
</build>
10651064
</profile>
10661065

1066+
<profile>
1067+
<id>java11-target</id>
1068+
1069+
<properties>
1070+
<target.java.version>11</target.java.version>
1071+
</properties>
1072+
1073+
<build>
1074+
<plugins>
1075+
<plugin>
1076+
<groupId>org.apache.maven.plugins</groupId>
1077+
<artifactId>maven-compiler-plugin</artifactId>
1078+
<configuration>
1079+
<source>11</source>
1080+
<target>11</target>
1081+
</configuration>
1082+
</plugin>
1083+
</plugins>
1084+
</build>
1085+
</profile>
1086+
10671087
<profile>
10681088
<id>java17</id>
10691089
<activation>
10701090
<jdk>[17,)</jdk>
10711091
</activation>
10721092

10731093
<properties>
1074-
<!-- Bump Scala because 2.12.7 doesn't compile on Java 17. -->
1075-
<scala.version>2.12.15</scala.version>
10761094
<surefire.excludedGroups.jdk>org.apache.flink.testutils.junit.FailsOnJava17</surefire.excludedGroups.jdk>
10771095
</properties>
10781096

@@ -1128,11 +1146,6 @@ under the License.
11281146
<jdk>[21,)</jdk>
11291147
</activation>
11301148

1131-
<properties>
1132-
<!-- Bump Scala because before 2.12.18 doesn't compile on Java 21. -->
1133-
<scala.version>2.12.18</scala.version>
1134-
</properties>
1135-
11361149
<build>
11371150
<pluginManagement>
11381151
<plugins>
@@ -1386,7 +1399,7 @@ under the License.
13861399
</property>
13871400
</activation>
13881401
<properties>
1389-
<target.java.version>11</target.java.version>
1402+
<target.java.version>17</target.java.version>
13901403
</properties>
13911404
<build>
13921405
<plugins>
@@ -1418,7 +1431,7 @@ under the License.
14181431
<!-- versions for certain build tools are enforced to match the CI setup -->
14191432
<!-- the rules below should stay in sync with Flink Release wiki documentation and the CI scripts -->
14201433
<requireJavaVersion>
1421-
<version>[11.0.0,11.1.0)</version>
1434+
<version>[${target.java.version}.*)</version>
14221435
</requireJavaVersion>
14231436
</rules>
14241437
</configuration>
@@ -2075,7 +2088,8 @@ under the License.
20752088
<artifactId>maven-compiler-plugin</artifactId>
20762089
<version>3.8.0</version>
20772090
<configuration>
2078-
<source>${target.java.version}</source>
2091+
<!-- Make sure that we only use Java 11 compatible APIs -->
2092+
<source>${source.java.version}</source>
20792093
<target>${target.java.version}</target>
20802094
<!-- The semantics of this option are reversed, see MCOMPILER-209. -->
20812095
<useIncrementalCompilation>false</useIncrementalCompilation>

‎tools/azure-pipelines/build-apache-repo.yml

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -69,10 +69,10 @@ stages:
6969
name: Default
7070
e2e_pool_definition:
7171
vmImage: 'ubuntu-22.04'
72-
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk11 -Pjava11-target"
72+
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk17 -Pjava17-target"
7373
run_end_to_end: false
7474
container: flink-build-container
75-
jdk: 11
75+
jdk: 17
7676
- job: docs_404_check # run on a MSFT provided machine
7777
pool:
7878
vmImage: 'ubuntu-22.04'
@@ -103,50 +103,50 @@ stages:
103103
- template: build-nightly-dist.yml
104104
parameters:
105105
stage_name: cron_snapshot_deployment
106-
environment: PROFILE="-Djdk11 -Pjava11-target"
106+
environment: PROFILE="-Djdk17 -Pjava17-target"
107107
container: flink-build-container
108-
jdk: 11
108+
jdk: 17
109109
- template: jobs-template.yml
110110
parameters:
111111
stage_name: cron_azure
112112
test_pool_definition:
113113
vmImage: 'ubuntu-22.04'
114114
e2e_pool_definition:
115115
vmImage: 'ubuntu-22.04'
116-
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk11 -Pjava11-target"
116+
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk17 -Pjava17-target"
117117
run_end_to_end: true
118118
container: flink-build-container
119-
jdk: 11
119+
jdk: 17
120120
- template: jobs-template.yml
121121
parameters:
122122
stage_name: cron_hadoop313
123123
test_pool_definition:
124124
name: Default
125125
e2e_pool_definition:
126126
vmImage: 'ubuntu-22.04'
127-
environment: PROFILE="-Dflink.hadoop.version=3.2.3 -Phadoop3-tests,hive3 -Djdk11 -Pjava11-target"
127+
environment: PROFILE="-Dflink.hadoop.version=3.2.3 -Phadoop3-tests,hive3 -Djdk17 -Pjava17-target"
128128
run_end_to_end: true
129129
container: flink-build-container
130-
jdk: 11
130+
jdk: 17
131131
- template: jobs-template.yml
132132
parameters:
133-
stage_name: cron_jdk17
133+
stage_name: cron_jdk11
134134
test_pool_definition:
135135
name: Default
136136
e2e_pool_definition:
137137
vmImage: 'ubuntu-22.04'
138-
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk11 -Djdk17 -Pjava17-target"
138+
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk11 -Pjava11-target"
139139
run_end_to_end: true
140140
container: flink-build-container
141-
jdk: 17
141+
jdk: 11
142142
- template: jobs-template.yml
143143
parameters:
144144
stage_name: cron_jdk21
145145
test_pool_definition:
146146
name: Default
147147
e2e_pool_definition:
148148
vmImage: 'ubuntu-22.04'
149-
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk11 -Djdk17 -Djdk21 -Pjava21-target"
149+
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk21 -Pjava21-target"
150150
run_end_to_end: true
151151
container: flink-build-container
152152
jdk: 21
@@ -157,10 +157,10 @@ stages:
157157
name: Default
158158
e2e_pool_definition:
159159
vmImage: 'ubuntu-22.04'
160-
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Penable-adaptive-scheduler -Djdk11 -Pjava11-target"
160+
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Penable-adaptive-scheduler -Djdk17 -Pjava17-target"
161161
run_end_to_end: true
162162
container: flink-build-container
163-
jdk: 11
163+
jdk: 17
164164
- job: docs_404_check # run on a MSFT provided machine
165165
pool:
166166
vmImage: 'ubuntu-22.04'
@@ -172,5 +172,5 @@ stages:
172172
- template: build-python-wheels.yml
173173
parameters:
174174
stage_name: cron_python_wheels
175-
environment: PROFILE="-Dflink.hadoop.version=2.10.2"
175+
environment: PROFILE="-Dflink.hadoop.version=2.10.2 -Djdk17 -Pjava17-target"
176176
container: flink-build-container

0 commit comments

Comments
 (0)