diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 701f4d2dfd..f861f3eaf7 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -16,6 +16,7 @@ * xref:microprofile.adoc[MicroProfile overview] * xref:jakarta-ee.adoc[Jakarta EE overview] * xref:container-images.adoc[Container images] + ** xref:verify-signatures-for-container-images-in-open-liberty.adoc[Verify signatures for container images in Open Liberty] * xref:rest-microservices.adoc[RESTful services] ** xref:sync-async-rest-clients.adoc[Sync and async REST clients] ** xref:rest-clients.adoc[REST clients] @@ -93,6 +94,9 @@ * xref:deployment-openshift.adoc[Deployment on OpenShift] * xref:open-liberty-operator.adoc[Open Liberty Operator] * xref:container-images.adoc[Container images] +* xref:instanton.adoc[Fast startup with InstantOn] + ** xref:instanton-sycalls-json.adoc[InstantOn system calls] + ** xref:instanton-limitations.adoc[InstantOn limitations and known issues] * xref:runnable-jar-files.adoc[Runnable JAR files] * xref:class-loader-library-config.adoc[Class loader configuration] * xref:virtual-hosts.adoc[Virtual hosts] diff --git a/modules/ROOT/pages/container-images.adoc b/modules/ROOT/pages/container-images.adoc index af528757e5..fd028470e5 100644 --- a/modules/ROOT/pages/container-images.adoc +++ b/modules/ROOT/pages/container-images.adoc @@ -31,7 +31,6 @@ Select from the following container image tags for the latest Open Liberty relea * `full-java11-openj9-ubi` * `full-java17-openj9-ubi` * `beta` -* `beta-instanton` * `latest` These tags pull the latest version because they do not specify a version number. You can specify a version other than the latest version in the image tag. If you do not specify a tag value, `latest` is used by default. For more information about these tag values, see the <<#tags,Container image tags naming conventions>> section. @@ -59,7 +58,7 @@ FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi COPY --chown=1001:0 server.xml /config/ # This script adds the requested XML snippets to enable Liberty features and grow the image to be fit-for-purpose. -# This option is available only in the 'kernel-slim' image type. The 'full', 'beta' and 'beta-instanton' tags already include all features. +# This option is available only in the 'kernel-slim' image type. The 'full' and 'beta' tags already include all features. RUN features.sh # Add interim fixes (optional) @@ -74,6 +73,9 @@ RUN configure.sh A template like this example results in a container image that has your application and configuration preinstalled, which means you can create new fully configured containers at any time. For more information about the different arguments and environment variables that you can specify to customize your application image, see the link:https://github.com/OpenLiberty/ci.docker#readme[Open Liberty container image GitHub documentation]. + +You can also use xref:instanton.adoc[Open Liberty InstantOn] to improve the startup time of your application container image. + [#tags] == Container image tags naming conventions @@ -88,7 +90,7 @@ The following tag values determine the resources that are included in a containe Fix pack version:: This value is optional. If you do not specify a fix pack version, your container image includes the latest release of Open Liberty. In addition to the latest release, the two most recent quarterly fix pack releases are always maintained for the `kernel-slim` and `full` image types. + -Open Liberty quarterly release versions end in `.3`, `.6`, `.9`, or `.12`. For example, if the latest Open Liberty release is `22.0.0.10`, then the `22.0.0.9` and `22.0.0.6` releases are also maintained in ICR. To pull one of these release versions instead of the latest, add your chosen release version to the beginning of any of the `kernel-slim` or `full` image tags in the previous list. For example, if `22.0.0.9` is a recent quarterly release, you can pull an image that includes the `22.0.0.9` release with an OpenJ9 distribution of Java 8 by specifying the `22.0.0.9-full-java8-openj9-ubi` tag. The `beta` and `beta-instanton` images are available only for the latest beta release. +Open Liberty quarterly release versions end in `.3`, `.6`, `.9`, or `.12`. For example, if the latest Open Liberty release is `22.0.0.10`, then the `22.0.0.9` and `22.0.0.6` releases are also maintained in ICR. To pull one of these release versions instead of the latest, add your chosen release version to the beginning of any of the `kernel-slim` or `full` image tags in the previous list. For example, if `22.0.0.9` is a recent quarterly release, you can pull an image that includes the `22.0.0.9` release with an OpenJ9 distribution of Java 8 by specifying the `22.0.0.9-full-java8-openj9-ubi` tag. The `beta` images are available only for the latest beta release. Liberty image type:: Two different image types are available: `kernel-slim` or `full`. The `full` type provides a server with all Open Liberty features and packages installed. @@ -105,10 +107,5 @@ In addition to the tags that follow this naming convention, Open Liberty offers * `latest`: This tag simplifies pulling the full latest Open Liberty release with Java 8. It is an alias for the `full-java8-openj9-ubi` tag. If you do not specify a tag value, `latest` is used by default. * `beta`: This tag provides the most recent beta release, which includes all the features and capabilities from the most recent release, plus new and updated features that are currently in development. -* `beta-instanton`: This tag is the same as the `beta` tag, except it also includes prerequisites for the Open Liberty https://openliberty.io/blog/2022/09/29/instant-on-beta.html[InstantOn beta feature]. -Both the beta images are UBI 8-based images with IBM Semeru 17 for x86-64/amd64 architectures. - - - diff --git a/modules/ROOT/pages/develop-liberty-tools.adoc b/modules/ROOT/pages/develop-liberty-tools.adoc index efeaf384b7..b881aa1df1 100644 --- a/modules/ROOT/pages/develop-liberty-tools.adoc +++ b/modules/ROOT/pages/develop-liberty-tools.adoc @@ -13,15 +13,15 @@ Liberty Tools are intuitive developer tools for the Eclipse IDE, Visual Studio Code, and IntelliJ IDEA development environments. These tools enable fast iterative development with Liberty dev mode and Maven or Gradle. -Liberty Tools automatically detect any projects that specify the Liberty Maven plug-in or the Liberty Gradle plug-in in the project `pom.xml` or `build.gradle` file. These projects are added to a special Liberty dashboard view in Eclipse IDE or Visual Studio Code, or to the Liberty tool window in IntelliJ IDEA. From there, you can access a command menu to manage your projects. +Liberty Tools automatically detect Liberty Maven or Gradle projects. These projects are added to a special Liberty dashboard view in Eclipse IDE or Visual Studio Code, or to the Liberty tool window in IntelliJ IDEA. From there, you can access a command menu to manage your projects. Liberty Tools also provide helpful language-support features such as code completion, hover descriptions, and configuration diagnostics for MicroProfile API, Jakarta EE API, and Liberty configuration and Java files. == Minimum requirements -Liberty Tools are currently available as an early release on all supported environments. +To use Liberty Tools with dev mode, your project must specify the https://github.com/OpenLiberty/ci.maven[Liberty Maven plug-in] or the https://github.com/OpenLiberty/ci.gradle[Liberty Gradle plug-in] in the project `pom.xml` or `build.gradle` file. For the best experience, always use the latest available version of either plug-in. -To use Liberty Tools with dev mode, your project must specify the https://github.com/OpenLiberty/ci.maven[Liberty Maven plug-in] or the https://github.com/OpenLiberty/ci.gradle[Liberty Gradle plug-in] in the project `pom.xml` or `build.gradle` file. A minimum level of `3.7.1` for the Liberty Maven plug-in or `3.5.1` for the Liberty Gradle plug-in is recommended. +Although you can run your Liberty applications on any xref:java-se.adoc[supported Java SE version], Java 17 is required to run Liberty Tools in your IDE. == Develop, test, and debug in your editor @@ -49,7 +49,7 @@ These features are available for the following Liberty configuration files. === MicroProfile APIs -Language support for MicroProfile 3.0 and 4.0 APIs includes the following features. +Language support for MicroProfile 3.0 and later APIs includes the following features. * Completion, hover, validation, and quick fix support for MicroProfile properties in `microprofile-config.properties` files and for MicroProfile code in Java files * Definition and format support for MicroProfile properties in `microprofile-config.properties` files @@ -58,7 +58,7 @@ Language support for MicroProfile 3.0 and 4.0 APIs includes the following featur === Jakarta EE APIs -Language support for the Jakarta EE API 9.x includes the following features for select Jakarta EE APIs that are found in Java files on a project's class path. +Language support for the Jakarta EE API 9.x and 10.0 APIs includes the following features for select Jakarta EE APIs that are found in Java files on a project's class path. * Code completion * Diagnostics @@ -66,7 +66,7 @@ Language support for the Jakarta EE API 9.x includes the following features for == Try it out -You can get an early release of Liberty Tools for your development environment directly from your vendor's marketplace. +You can get Liberty Tools for your development environment directly from your vendor's marketplace. - https://plugins.jetbrains.com/plugin/14856-liberty-tools[Liberty Tools for IntelliJ IDEA] in the Jet Brains Marketplace - https://marketplace.visualstudio.com/items?itemName=Open-Liberty.liberty-dev-vscode-ext[Liberty Tools for Visual Studio Code] in the Visual Studio Code Marketplace diff --git a/modules/ROOT/pages/instanton-limitations.adoc b/modules/ROOT/pages/instanton-limitations.adoc new file mode 100644 index 0000000000..1566497449 --- /dev/null +++ b/modules/ROOT/pages/instanton-limitations.adoc @@ -0,0 +1,297 @@ +// Copyright (c) 2022 IBM Corporation and others. +// Licensed under Creative Commons Attribution-NoDerivatives +// 4.0 International (CC BY-ND 4.0) +// https://creativecommons.org/licenses/by-nd/4.0/ +// +// Contributors: +// IBM Corporation +// +:page-description: OThe following sections describe the limitations and known issues with using Open Liberty InstantOn. +:seo-title: Open Liberty InstantOn +:seo-description: The following sections describe the limitations and known issues with using Open Liberty InstantOn. +:page-layout: general-reference +:page-type: general += InstantOn limitations and known issues + +In addition to the general InstantOn prerequisites, Open Liberty InstantOn is subject to certain limitations. For example, applications that must run early startup code or that rely on certain Liberty features might require modification to use InstantOn. + +For more information about InstantOn prerequisites, see xref:instanton.adoc#prereq[Runtime and host build system prerequisites]. + +The following sections describe the limitations and known issues with using Open Liberty InstantOn. + +- <<#transaction,Jakarta Transactions configuration limitations>> +- <<#trans-before, Jakarta Transaction before checkpoint>> +- <<#mp-config, Accessing MicroProfile Configuration too early>> +- <<#datasource, Injecting a DataSource too early>> +- <<#features, Using product extensions, user features, or features that are not supported by InstantOn>> +- <<#boot,Updating configuration with a bootstrap.properties file>> +- <<#securitymanager, Java SecurityManager is not supported>> +- <<#jvm, Updating JVM options>> +- <<#se, SELinux limitations>> +- <<#yama, Yama Linux Security Module limitations>> +- <<#linux-calls,Access to Linux system calls>> +- <<#linux, Running without the necessary Linux capabilities>> +- <<#processors, Supported processors>> + +[#transaction] +== Jakarta Transactions configuration limitations +Open Liberty Transaction Manager support for InstantOn has limitations around configuration updates when the application process is restored. The configuration attributes for the config:transaction[display=Transaction Manager] must remain constant between the InstantOn checkpoint and restore. This limitation is true only for the configuration attributes that are specified directly with the `transaction` server configuration element, for example, `recoveryGroup` and `recoverIdentity`. The values for these configuration attributes must not change between checkpoint and restore. + +This limitation implies that xref:transaction-service#cloud[transaction recovery in a cloud environment] cannot work as designed because the `recoverIdentity` cannot be parameterized by something like the following `transaction` configuration example. This example gives a unique `recoverIdentity` for each instance of the application: + +[source,xml] +---- + +---- + +[#trans-before] +== Jakarta Transaction before checkpoint +Open Liberty InstantOn does not allow transactions to begin before a checkpoint is performed for the application process. This scenario is possible if the application has xref:instanton.adoc#beforeAppStart[early startup code] that attempts to start a transaction. Consider the following `Servlet`: + +[source,java] +---- +@WebServlet(urlPatterns = { "/StartupServlet" }, loadOnStartup = 1) +public class StartupServlet extends HttpServlet { + @Override + public void init() { + UserTransaction ut = UserTransactionFactory.getUserTransaction(); + try { + ut.begin(); + ... + ut.commit(); + } catch (Exception e) { + // something went wrong + } + } + +} +---- + +This `Servlet` example uses the `loadOnStartup = 1` attribute. When you use this attribute with the `afterAppStart` option, the servlet initializes before the checkpoint. The runtime detects this conflict and logs the following message: + +[source,text] +---- +[WARNING ] WTRN0155W: An application began or required a transaction during the server checkpoint request. The following stack trace for this thread was captured when the transaction was created: +---- + +This warning is followed by a stacktrace that helps identify the application code that is attempting to begin a transaction. The server then fails to checkpoint and the following error is logged: + +[source,text] +---- +WTRN0154E: The server checkpoint request failed because the transaction service is unable to begin a transaction. +---- + +You can avoid this failure by using the `beforeAppStart` option or by modifying the component not to use early startup code. In this example, that modification is to remove the `loadOnStartup = 1` attribute. + +[#mp-config] +== Accessing MicroProfile Config too early +If an application has early startup code and you are using the `afterAppStart` option, it might get injected with a configuration value from xref:external-configuration.adoc[MicroProfile Config] before a checkpoint is performed for the application process. If such a configuration value changes at the time the application image container runs, the application might use the stale value that was set when the application process checkpoint was performed. + +The Open Liberty runtime detects this situation and logs a warning message when the application container image is run that indicates that a configuration value is changed. The following example uses an `example_config` configuration key with a default value set to `theDefault`. When the checkpoint occurs, the environment configuration source is not available to populate MicroProfile configuration values. If this `@Inject` annotation of the configuration is contained in a CDI bean that is created and used before the checkpoint is performed, the value of `theDefault` is injected. + +[source,java] +---- + @Inject + @ConfigProperty(name = "example_config", defaultValue = "theDefault") + String exampleConfig; +---- + +When the InstantOn application container image is run, the environment variable `EXAMPLE_CONFIG` can be used to provide an updated value. The runtime detects this value and logs the following message: + +[source,text] +---- +[WARNING ] CWWKC0651W: The MicroProfile configuration value for the key example_config has changed since the checkpoint action completed on the server. If the value of the key changes after the checkpoint action, the application might not use the updated value. + +---- + +In this situation, use the `beforeAppStart` checkpoint option. Another option is to use a link:https://download.eclipse.org/microprofile/microprofile-config-3.0/microprofile-config-spec-3.0.html#_dynamic_configsource[Dynamic ConfigSource]. The previous example can be modified to use a dynamic ConfigSource by using the `Provider` type for the `exampleConfig` variable: + +[source,java] +---- + @Inject + @ConfigProperty(name = "example_config", defaultValue = "theDefault") + Provider exampleConfig; +---- + +Each call to the `get()` method of the `Provider` returns the current value of the `ConfigProperty` annotation. This behavior allows the application to access the updated configuration value when the application process is restored during the InstantOn application container run. + +[#datasource] +== Injecting a DataSource too early +If an application has early startup code and you are using the `afterAppStart` option, it might get injected with `DataSource` before a checkpoint is performed for the application process. In a cloud environment, the configuration of the `DataSource` likely needs to change at the time the application image container is run. Consider the following `Servlet` example: + +[source,java] +---- +@WebServlet(urlPatterns = "/ExampleServlet", loadOnStartup = 1) +public class ExampleServlet extends HttpServlet { + @Resource(shareable = false) + private DataSource exampleDataSource; + ... +} +---- + +This `Servlet` example uses the `loadOnStartup = 1` attribute. When you are using the `afterAppStart` option, this attribute initializes the servlet before the checkpoint. The deployment information related to the `DataSource` might need to be configured when you deploy the application to the cloud. Consider the following Open Liberty `server.xml` configuration. + +[source,xml] +---- + + + + + + + + + + + + + + + +---- + +This configuration uses placeholder values for things like the database name, hostname, ports, user, and password. This configuration allows the values to be updated with environment variable values or other configuration mechanisms, as described in link:/guides/kubernetes-microprofile-config.html[Configuring microservices running in Kubernetes]. These configurations must not be hardcoded into an application image and must be able to be updated when you deploy the application to the cloud. + +If an application is injected with a `DataSource` before the checkpoint and the configuration of the `DataSource` changes, the application is restarted when the InstantOn application container image is run with the updated configuration. You can avoid this scenario by using the `beforeAppStart` option or by modifying the component not to be early startup code. In this example, that modification is to remove the `loadOnStartup = 1` attribute. + +[#features] +== Using product extensions, user features, or features that are not supported by InstantOn +InstantOn supports only a subset of Open Liberty features, as described in xref:instanton.adoc#supported-features[Open Liberty InstantOn supported features]. Any public features that are enabled outside of the supported set of features for InstantOn cause checkpoint to fail with an error message like the following example: + +[source,sh] +---- +CWWKC0456E: A checkpoint cannot be taken because the following features configured in the server.xml file are not supported for checkpoint: [usr:exampleFeature-1.0] +---- + +This error occurs for any configured features that are not supported for InstantOn. This limitation includes Liberty product extension and Liberty user features. + +[#boot] +== Updating configuration with a bootstrap.properties file +When an InstantOn application container image is run, the xref:reference:bootstrap-properties.adoc[bootstrap.properties] file is not read. Values that must be able to be configured when you run an InstantOn application container image must come from alternative sources. For example, you might use environment variables or other configuration mechanisms, as described link:/guides/kubernetes-microprofile-config.html[Configuring microservices running in Kubernetes]. + +[#securitymanager] +== Java SecurityManager is not supported +If Open Liberty is configured to run with the `SecurityManager`, InstantOn detects this configuration during a checkpoint and fails with the following message: + +[source,sh] +---- +CWWKE0958E: The server checkpoint request failed because the websphere.java.security property was set in the bootstrap.properties file. This property enables the Java Security Manager and is not valid when a server checkpoint occurs. +---- + +[#jvm] +== Updating JVM options +InstantOn does not support changing the xref:reference:config/server-configuration-overview.adoc#jvm-options[jvm.options] when you restore the InstantOn application process. Any JVM options that are required to be set for the JVM process must be defined during the InstantOn container image build. + +The IBM Semeru JVM does have limited support for setting JVM options on restore with the use of the `OPENJ9_RESTORE_JAVA_OPTIONS` environment variable. For more information, see the Java link:https://www.eclipse.org/openj9/docs/criusupport/[Checkpoint/Restore In Userspace (CRIU) support] documentation. + +[#se] +== SELinux limitations +If link:https://www.redhat.com/en/topics/linux/what-is-selinux[SELinux] mode is set to `enforcing`, SELinux might prevent InstantOn from performing a checkpoint of the application process when you use the xref:instanton.adoc#checkpoint_script[checkpoint.sh script] in the image template `Dockerfile` or `Containerfile`. If the `virt_sandbox_use_netlink` SELinux setting is disabled, the required `netlink` Linux system calls are blocked. This block prevents InstantOn from performing a checkpoint of the application process during the container image build. Open Liberty InstantOn detects this limitation and logs the following messages: + +[source,sh] +---- +CWWKE0962E: The server checkpoint request failed. The following output is from the CRIU /logs/checkpoint/checkpoint.log file that contains details on why the checkpoint failed. +Warn (criu/kerndat.c:1103): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file +Error (criu/libnetlink.c:84): Can't send request message: Permission denied +.. +Error (criu/cr-dump.c:2099): Dumping FAILED. +CWWKE0963E: The server checkpoint request failed because netlink system calls were unsuccessful. If SELinux is enabled in enforcing mode, netlink system calls might be blocked by the SELinux "virt_sandbox_use_netlink" policy setting. Either disable SELinux or enable the netlink system calls with the "setsebool virt_sandbox_use_netlink 1" command. +---- + +To work around this limitation, you can either enable the `virt_sandbox_use_netlink` SELinux setting with the `setsebool virt_sandbox_use_netlink 1` command or disable SELinux `enforcing` mode. Another option to work around this issue is to use the xref:instanton.adoc#three_step_process[three-step process] to build the InstantOn image. The three-step process requires the use of a `--privileged` container that grants access to the `netlink` system calls to the running container that performs the application process checkpoint. + +[#yama] +== Yama Linux Security Module limitations +If link:https://www.kernel.org/doc/Documentation/security/Yama.txt[Yama] is configured with one of the following modes, InstantOn cannot checkpoint or restore the application process in running containers: + +- `2` - admin-only attach +- `3` - no attach + +When this configuration is present, the `/logs/checkpoint/restore.log` contains the following error: + +[source,sh] +---- +Error (criu/arch/x86/kerndat.c:178): 32: ptrace(PTRACE_TRACEME) failed: Operation not permitted +---- + +For InstantOn checkpoint and restore to work, Yama must be configured with one of the following modes: + +- `0` - classic ptrace permissions +- `1` - restricted ptrace + +The following supported public cloud Kubernetes services have the default for Yama set to the `1` mode, which allows InstantOn to checkpoint and restore by default: + +- link:https://aws.amazon.com/eks/[Amazon Elastic Kubernetes Service (EKS)] +- link:https://azure.microsoft.com/en-us/products/kubernetes-service[Azure Kubernetes Service (AKS)] + +[#linux-calls] +== Access to Linux system calls +As described in xref:instanton.adoc#required-system-calls[Required Linux system calls], CRIU requires several Linux system calls to restore the application process. This requirement might require extra configuration to grant the required system calls to the running container when you use InstantOn. + +The following examples are errors that are logged to the `/logs/checkpoint/restore.log` file when access to specific system calls is blocked. + +[source,sh] +.Blocked clone3 system call +---- +Error (criu/kerndat.c:1377): Unexpected error from clone3: Operation not permitted +---- + +[source,sh] +.Blocked to ptrace system call +---- +Error (criu/arch/x86/kerndat.c:178): 28: ptrace(PTRACE_TRACEME) failed: Operation not permitted +---- + +[source,sh] +.Blocked to vmsplice system call +---- +Error (criu/pipes.c:184): 0x4c11a: Error splicing data: Operation not permitted +---- + +The supported public cloud Kubernetes Service environments currently allow the required system calls used by CRIU by default. No additional configuration is required when you use the following cloud providers: + +- Amazon Elastic Kubernetes Service (EKS) +- Azure Kubernetes Service (AKS) + +[#linux] +== Running without the necessary Linux capabilities +Errors occur during checkpoint and restore if the required xref:instanton.adoc#linux-capabilities[Linux capabilities] are not granted. If the required capabilities are not granted for checkpoint, then the following error occurs during the InstantOn container image build: + +[source,sh] +---- +Can't exec criu swrk: Operation not permitted +Can't read request: Connection reset by peer +Can't receive response: Invalid argument +[ERROR ] CWWKC0453E: The server checkpoint request failed with the following message: Could not dump the JVM processes, err=-70 +---- + +The `Operation not permitted` message indicates that the required Linux capabilities are not granted. If you are using the xref:instanton.adoc#checkpoint_script[checkpoint.sh script], the following error occurs during the `RUN checkpoint.sh` instruction: + +[source,sh] +---- +Error: building at STEP "RUN checkpoint.sh afterAppStart": while running runtime: exit status 74 +---- + +To avoid this error, grant the container image build the `CHECKPOINT_RESTORE`, `SYS_PTRACE`, and `SETPCAP Linux capabilities`. If you use the xref:instanton.adoc#three_step_process[three-step process] to build the container image, make sure the container that is running the checkpoint step is a `--privileged` container. + +If the required capabilities are not granted for restore, the following error occurs when you try to run the InstantOn application container image: + +[source,sh] +---- +/opt/ol/wlp/bin/server: line 1430: /opt/criu/criu: Operation not permitted +CWWKE0961I: Restoring the checkpoint server process failed. Check the /logs/checkpoint/restore.log log to determine why the checkpoint process was not restored. Launching the server without using the checkpoint image. +---- + +The `Operation not permitted` message is an indication that the required Linux capabilities are not granted for restore. + +[#processors] +== Supported processors +Currently, the only supported processor is X86-64/AMD64. Other processors are expected to be supported in later releases of Open Liberty InstantOn. + diff --git a/modules/ROOT/pages/instanton-sycalls-json.adoc b/modules/ROOT/pages/instanton-sycalls-json.adoc new file mode 100644 index 0000000000..4e5fcf9fdd --- /dev/null +++ b/modules/ROOT/pages/instanton-sycalls-json.adoc @@ -0,0 +1,851 @@ +// Copyright (c) 2022 IBM Corporation and others. +// Licensed under Creative Commons Attribution-NoDerivatives +// 4.0 International (CC BY-ND 4.0) +// https://creativecommons.org/licenses/by-nd/4.0/ +// +// Contributors: +// IBM Corporation +// +:page-description: Open Liberty InstantOn system calls +:seo-title: Open Liberty InstantOnsystem calls +:seo-description: Open Liberty InstantOnsystem calls +:page-layout: general-reference +:page-type: general += Open Liberty InstantOn System Calls + +The `criuRequiredSysCalls.json` file grants access to all the Linux system calls that are required by Checkpoint/Restore In Userspace (CRIU) to restore an InstantOn application process. You can assign this file to specify the policy for the container by using the Podman or Docker command options. + +For more information, see xref:instanton.adoc#required-system-calls[InstantOn: Required Linux system calls]. + +The following example shows a `criuRequiredSysCalls.json` file that grants the necessary access to restore an InstantOn application process. + +[#sys-calls-json] +[source,json] +.criuRequiredSysCalls.json +---- +{ + "defaultAction": "SCMP_ACT_ERRNO", + "defaultErrnoRet": 1, + "archMap": [ + { + "architecture": "SCMP_ARCH_X86_64", + "subArchitectures": [ + "SCMP_ARCH_X86", + "SCMP_ARCH_X32" + ] + }, + { + "architecture": "SCMP_ARCH_AARCH64", + "subArchitectures": [ + "SCMP_ARCH_ARM" + ] + }, + { + "architecture": "SCMP_ARCH_MIPS64", + "subArchitectures": [ + "SCMP_ARCH_MIPS", + "SCMP_ARCH_MIPS64N32" + ] + }, + { + "architecture": "SCMP_ARCH_MIPS64N32", + "subArchitectures": [ + "SCMP_ARCH_MIPS", + "SCMP_ARCH_MIPS64" + ] + }, + { + "architecture": "SCMP_ARCH_MIPSEL64", + "subArchitectures": [ + "SCMP_ARCH_MIPSEL", + "SCMP_ARCH_MIPSEL64N32" + ] + }, + { + "architecture": "SCMP_ARCH_MIPSEL64N32", + "subArchitectures": [ + "SCMP_ARCH_MIPSEL", + "SCMP_ARCH_MIPSEL64" + ] + }, + { + "architecture": "SCMP_ARCH_S390X", + "subArchitectures": [ + "SCMP_ARCH_S390" + ] + }, + { + "architecture": "SCMP_ARCH_RISCV64", + "subArchitectures": null + } + ], + "syscalls": [ + { + "names": [ + "accept", + "accept4", + "access", + "adjtimex", + "alarm", + "bind", + "brk", + "capget", + "capset", + "chdir", + "chmod", + "chown", + "chown32", + "clock_adjtime", + "clock_adjtime64", + "clock_getres", + "clock_getres_time64", + "clock_gettime", + "clock_gettime64", + "clock_nanosleep", + "clock_nanosleep_time64", + "close", + "close_range", + "connect", + "copy_file_range", + "creat", + "dup", + "dup2", + "dup3", + "epoll_create", + "epoll_create1", + "epoll_ctl", + "epoll_ctl_old", + "epoll_pwait", + "epoll_pwait2", + "epoll_wait", + "epoll_wait_old", + "eventfd", + "eventfd2", + "execve", + "execveat", + "exit", + "exit_group", + "faccessat", + "faccessat2", + "fadvise64", + "fadvise64_64", + "fallocate", + "fanotify_mark", + "fchdir", + "fchmod", + "fchmodat", + "fchown", + "fchown32", + "fchownat", + "fcntl", + "fcntl64", + "fdatasync", + "fgetxattr", + "flistxattr", + "flock", + "fork", + "fremovexattr", + "fsetxattr", + "fstat", + "fstat64", + "fstatat64", + "fstatfs", + "fstatfs64", + "fsync", + "ftruncate", + "ftruncate64", + "futex", + "futex_time64", + "futex_waitv", + "futimesat", + "getcpu", + "getcwd", + "getdents", + "getdents64", + "getegid", + "getegid32", + "geteuid", + "geteuid32", + "getgid", + "getgid32", + "getgroups", + "getgroups32", + "getitimer", + "getpeername", + "getpgid", + "getpgrp", + "getpid", + "getppid", + "getpriority", + "getrandom", + "getresgid", + "getresgid32", + "getresuid", + "getresuid32", + "getrlimit", + "get_robust_list", + "getrusage", + "getsid", + "getsockname", + "getsockopt", + "get_thread_area", + "gettid", + "gettimeofday", + "getuid", + "getuid32", + "getxattr", + "inotify_add_watch", + "inotify_init", + "inotify_init1", + "inotify_rm_watch", + "io_cancel", + "ioctl", + "io_destroy", + "io_getevents", + "io_pgetevents", + "io_pgetevents_time64", + "ioprio_get", + "ioprio_set", + "io_setup", + "io_submit", + "io_uring_enter", + "io_uring_register", + "io_uring_setup", + "ipc", + "kill", + "landlock_add_rule", + "landlock_create_ruleset", + "landlock_restrict_self", + "lchown", + "lchown32", + "lgetxattr", + "link", + "linkat", + "listen", + "listxattr", + "llistxattr", + "_llseek", + "lremovexattr", + "lseek", + "lsetxattr", + "lstat", + "lstat64", + "madvise", + "membarrier", + "memfd_create", + "memfd_secret", + "mincore", + "mkdir", + "mkdirat", + "mknod", + "mknodat", + "mlock", + "mlock2", + "mlockall", + "mmap", + "mmap2", + "mprotect", + "mq_getsetattr", + "mq_notify", + "mq_open", + "mq_timedreceive", + "mq_timedreceive_time64", + "mq_timedsend", + "mq_timedsend_time64", + "mq_unlink", + "mremap", + "msgctl", + "msgget", + "msgrcv", + "msgsnd", + "msync", + "munlock", + "munlockall", + "munmap", + "nanosleep", + "newfstatat", + "_newselect", + "open", + "openat", + "openat2", + "pause", + "pidfd_open", + "pidfd_send_signal", + "pipe", + "pipe2", + "poll", + "ppoll", + "ppoll_time64", + "prctl", + "pread64", + "preadv", + "preadv2", + "prlimit64", + "process_mrelease", + "pselect6", + "pselect6_time64", + "pwrite64", + "pwritev", + "pwritev2", + "read", + "readahead", + "readlink", + "readlinkat", + "readv", + "recv", + "recvfrom", + "recvmmsg", + "recvmmsg_time64", + "recvmsg", + "remap_file_pages", + "removexattr", + "rename", + "renameat", + "renameat2", + "restart_syscall", + "rmdir", + "rseq", + "rt_sigaction", + "rt_sigpending", + "rt_sigprocmask", + "rt_sigqueueinfo", + "rt_sigreturn", + "rt_sigsuspend", + "rt_sigtimedwait", + "rt_sigtimedwait_time64", + "rt_tgsigqueueinfo", + "sched_getaffinity", + "sched_getattr", + "sched_getparam", + "sched_get_priority_max", + "sched_get_priority_min", + "sched_getscheduler", + "sched_rr_get_interval", + "sched_rr_get_interval_time64", + "sched_setaffinity", + "sched_setattr", + "sched_setparam", + "sched_setscheduler", + "sched_yield", + "seccomp", + "select", + "semctl", + "semget", + "semop", + "semtimedop", + "semtimedop_time64", + "send", + "sendfile", + "sendfile64", + "sendmmsg", + "sendmsg", + "sendto", + "setfsgid", + "setfsgid32", + "setfsuid", + "setfsuid32", + "setgid", + "setgid32", + "setgroups", + "setgroups32", + "setitimer", + "setpgid", + "setpriority", + "setregid", + "setregid32", + "setresgid", + "setresgid32", + "setresuid", + "setresuid32", + "setreuid", + "setreuid32", + "setrlimit", + "set_robust_list", + "setsid", + "setsockopt", + "set_thread_area", + "set_tid_address", + "setuid", + "setuid32", + "setxattr", + "shmat", + "shmctl", + "shmdt", + "shmget", + "shutdown", + "sigaltstack", + "signalfd", + "signalfd4", + "sigprocmask", + "sigreturn", + "socket", + "socketcall", + "socketpair", + "splice", + "stat", + "stat64", + "statfs", + "statfs64", + "statx", + "symlink", + "symlinkat", + "sync", + "sync_file_range", + "syncfs", + "sysinfo", + "tee", + "tgkill", + "time", + "timer_create", + "timer_delete", + "timer_getoverrun", + "timer_gettime", + "timer_gettime64", + "timer_settime", + "timer_settime64", + "timerfd_create", + "timerfd_gettime", + "timerfd_gettime64", + "timerfd_settime", + "timerfd_settime64", + "times", + "tkill", + "truncate", + "truncate64", + "ugetrlimit", + "umask", + "uname", + "unlink", + "unlinkat", + "utime", + "utimensat", + "utimensat_time64", + "utimes", + "vfork", + "vmsplice", + "wait4", + "waitid", + "waitpid", + "write", + "writev", + "arch_prctl", + "chroot", + "clone", + "clone3", + "fallocate", + "fanotify_init", + "fsconfig", + "fsmount", + "fsopen", + "guarded_storage", + "kcmp", + "lseek", + "mmap", + "mount", + "open", + "open_by_handle_at", + "openat", + "pivot_root", + "preadv", + "process_vm_readv", + "ptrace", + "readdir", + "s390_runtime_instr", + "setns", + "sigaction", + "signal", + "syscall", + "umount", + "umount2", + "unshare", + "userfaultfd", + "wait" + ], + "action": "SCMP_ACT_ALLOW" + }, + { + "names": [ + "process_vm_readv", + "process_vm_writev", + "ptrace" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "minKernel": "4.8" + } + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 0, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 8, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 131072, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 131080, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "personality" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 4294967295, + "op": "SCMP_CMP_EQ" + } + ] + }, + { + "names": [ + "sync_file_range2", + "swapcontext" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "ppc64le" + ] + } + }, + { + "names": [ + "arm_fadvise64_64", + "arm_sync_file_range", + "sync_file_range2", + "breakpoint", + "cacheflush", + "set_tls" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "arm", + "arm64" + ] + } + }, + { + "names": [ + "arch_prctl" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "amd64", + "x32" + ] + } + }, + { + "names": [ + "modify_ldt" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "amd64", + "x32", + "x86" + ] + } + }, + { + "names": [ + "s390_pci_mmio_read", + "s390_pci_mmio_write", + "s390_runtime_instr" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "s390", + "s390x" + ] + } + }, + { + "names": [ + "riscv_flush_icache" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "arches": [ + "riscv64" + ] + } + }, + { + "names": [ + "open_by_handle_at" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_DAC_READ_SEARCH" + ] + } + }, + { + "names": [ + "bpf", + "clone", + "clone3", + "fanotify_init", + "fsconfig", + "fsmount", + "fsopen", + "fspick", + "lookup_dcookie", + "mount", + "mount_setattr", + "move_mount", + "name_to_handle_at", + "open_tree", + "perf_event_open", + "quotactl", + "quotactl_fd", + "setdomainname", + "sethostname", + "setns", + "syslog", + "umount", + "umount2", + "unshare" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_ADMIN" + ] + } + }, + { + "names": [ + "clone" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 0, + "value": 2114060288, + "op": "SCMP_CMP_MASKED_EQ" + } + ], + "excludes": { + "caps": [ + "CAP_SYS_ADMIN" + ], + "arches": [ + "s390", + "s390x" + ] + } + }, + { + "names": [ + "clone" + ], + "action": "SCMP_ACT_ALLOW", + "args": [ + { + "index": 1, + "value": 2114060288, + "op": "SCMP_CMP_MASKED_EQ" + } + ], + "comment": "s390 parameter ordering for clone is different", + "includes": { + "arches": [ + "s390", + "s390x" + ] + }, + "excludes": { + "caps": [ + "CAP_SYS_ADMIN" + ] + } + }, + { + "names": [ + "clone3" + ], + "action": "SCMP_ACT_ERRNO", + "errnoRet": 38, + "excludes": { + "caps": [ + "CAP_SYS_ADMIN" + ] + } + }, + { + "names": [ + "reboot" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_BOOT" + ] + } + }, + { + "names": [ + "chroot" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_CHROOT" + ] + } + }, + { + "names": [ + "delete_module", + "init_module", + "finit_module" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_MODULE" + ] + } + }, + { + "names": [ + "acct" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_PACCT" + ] + } + }, + { + "names": [ + "kcmp", + "pidfd_getfd", + "process_madvise", + "process_vm_readv", + "process_vm_writev", + "ptrace" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_PTRACE" + ] + } + }, + { + "names": [ + "iopl", + "ioperm" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_RAWIO" + ] + } + }, + { + "names": [ + "settimeofday", + "stime", + "clock_settime" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_TIME" + ] + } + }, + { + "names": [ + "vhangup" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_TTY_CONFIG" + ] + } + }, + { + "names": [ + "get_mempolicy", + "mbind", + "set_mempolicy" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYS_NICE" + ] + } + }, + { + "names": [ + "syslog" + ], + "action": "SCMP_ACT_ALLOW", + "includes": { + "caps": [ + "CAP_SYSLOG" + ] + } + } + ] +} +---- + + diff --git a/modules/ROOT/pages/instanton.adoc b/modules/ROOT/pages/instanton.adoc new file mode 100644 index 0000000000..4a361be0d3 --- /dev/null +++ b/modules/ROOT/pages/instanton.adoc @@ -0,0 +1,359 @@ +// Copyright (c) 2023 IBM Corporation and others. +// Licensed under Creative Commons Attribution-NoDerivatives +// 4.0 International (CC BY-ND 4.0) +// https://creativecommons.org/licenses/by-nd/4.0/ +// +// Contributors: +// IBM Corporation +// +:page-description: Open Liberty InstantOn provides incredibly fast startup times for MicroProfile and Jakarta EE applications. +:seo-title: Open Liberty InstantOn +:seo-description: Open Liberty InstantOn provides incredibly fast startup times for MicroProfile and Jakarta EE applications. +:page-layout: general-reference +:page-type: general += Fast startup for containerized applications with Open Liberty InstantOn + +Open Liberty InstantOn provides fast startup times for MicroProfile and Jakarta EE applications. With InstantOn, your applications can start in milliseconds, without compromising on throughput, memory, development-production parity, or Java language features. + +InstantOn uses the Checkpoint/Restore In Userspace (link:https://criu.org/[CRIU]) feature of the Linux kernel to take a checkpoint of the JVM that can be restored later. With InstantOn, the application is developed as normal, and then InstantOn makes a checkpoint of the application process when the application container image is built. When the application is restored, it runs in the same JVM, which provides complete parity between development and production. Because the checkpoint process takes only seconds, your CI/CD process is barely affected. + +When you build an application container image, InstantOn creates an extra image layer that contains a checkpoint of the Open Liberty application process. When the application image starts, the Open Liberty runtime recognizes the InstantOn layer and restores the application process to the checkpoint that was created during the application container image build. + +Open Liberty and link:https://blog.openj9.org/2022/10/14/openj9-criu-support-a-look-under-the-hood/[OpenJ9] contain hooks that participate in the checkpoint and restore process. These hooks allow the Open Liberty runtime to prepare the process for checkpoint such that persisting the checkpoint process in the container image is safe. This preparation step ensures that the process can be restored from the same persistent state in the container image into multiple, possibly concurrent, instances of the application. For more information on the CRIU support in OpenJ9 and the IBM Semeru JVM, see the Java link:https://www.eclipse.org/openj9/docs/criusupport/[CRIU Support] documentation. + +InstantOn cannot be used outside of a container image build. An application container image provides a consistent environment, which is required to ensure a reliable restore of an Open Liberty application process. The InstantOn container layer is the last layer of the application container image. This configuration ensures that the resources in the underlying layers of the image do not change from the time the checkpoint is taken to the time the image is started with InstantOn. + +The following sections describe the prerequisites and processes to build and run an InstantOn application image: + +- <<#beforeAppStart, When to take a checkpoint: beforeAppStart or afterAppStart>> +- <<#prereq, Runtime and host build system prerequisites>> +- <<#build, Building an InstantOn application image>> +- <<#run, Running and deploying an InstantOn application image>> +- <<#supported-features, Open Liberty InstantOn supported features>> + +[#beforeAppStart] +== When to make a checkpoint: beforeAppStart or afterAppStart + +When an InstantOn checkpoint occurs, the Open Liberty runtime starts. During this startup, the runtime processes the configuration, loads all the enabled features, and starts processing the configured application. This startup sequence varies depending on the size and complexity of the application and the number of Open Liberty features it requires. For small simple applications, this process can take less than 1 second. For more typical applications, it can take several seconds. InstantOn takes a checkpoint after the Open Liberty runtime completes startup. Two options are available to determine whether the checkpoint occurs before or after the application itself starts, either of which can be specified as part of the InstantOn `RUN` configuration: + +- `afterAppStart` - Performs a checkpoint after the application is started. This option provides the fastest startup, but might not be suitable for some applications. +- `beforeAppStart` - Performs a checkpoint before any application code runs. Use this option in cases where the application code is not compatible with the `afterAppStart` option. + + +Which of these options you choose depends on the code your application must run. +link:https://jakarta.ee/[Jakarta EE] and link:https://microprofile.io/[MicroProfile] applications might contain application code that gets run as the application is started, such as the following examples: + +- A servlet that uses the `loadOnStartup` attribute +- An EJB that uses the `@Startup` annotation +- A CDI bean that uses `@Observes @Initialized(ApplicationScoped.class)` annotations + +Sometimes, the application code that runs as the application starts might not be suited for performing an InstantOn checkpoint. For such cases, use the `beforeAppStart` option. For example, the following application code scenarios must be avoided before an InstantOn checkpoint, and are therefore better suited for the `beforeAppStart` option: + +- Accessing a remote resource, such as a database. The correct data source is unlikely to be available to connect to during an application container build. +- Creating a transaction. Currently, transactions are prohibited before an InstantOn checkpoint. +- Reading configuration that is expected to change when the application is deployed, for example configuration from MicroProfile Config. + +Using the `beforeAppStart` option in these cases ensures that the application code is run only after the InstantOn checkpoint process is restored. This option might result in slower restore times because it must run more code before the application is ready to service any incoming requests. +If the application early start code is determined to be safe and acceptable for checkpoint, then the `afterAppStart` checkpoint option can be used. This option provides for the fastest startup time when the application process is restored. + +If an application has no code that is run as the application is started, then the `beforeAppStart` and `afterAppStart` checkpoints are equivalent. In these cases, both checkpoint options perform a checkpoint of the process before the configured ports are enabled for servicing requests. This sequence ensures that the transport protocols for the application are enabled only after the InstantOn checkpoint process is restored. + +For more information about limitations with early startup code and possible workarounds, see xref:instanton-limitations.adoc[InstantOn limitations and known issues]. + +[#prereq] +== Runtime and host build system prerequisites + +Starting with Open Liberty version 23.0.0.6, all X86-64/AMD64 UBI xref:container-images.adoc[Open Liberty container images] include the prerequisites for InstantOn to checkpoint and restore Open Liberty application processes. Open Liberty Ubuntu container images are not enabled for InstantOn. + +Currently, InstantOn is supported with IBM Semeru Java version 11.0.19+ and IBM Semeru Java version 17.0.7+. InstantOn is expected to support new versions of IBM Semeru Java as they are released. Currently, InstantOn is not supported on other Java vendor implementations. + +To use InstantOn during an application container image build, the host machine must satisfy the following prerequisites: + +- Uses the Linux Operating System with kernel version 5.9 or greater +- Uses the X86-64/AMD64 processor architecture +- Allows execution of privileged container builds by using Podman or Docker + +The use of privileged containers in the application container image build is required only when you build the container image with InstantOn. Running the resulting InstantOn application image does not require the use of privileged containers. + +To use Docker, a version 23.0 or greater is required. Earlier versions of Docker did not support the `CHECKPOINT_RESTORE` Linux capability, which prevents InstantOn from doing a checkpoint or restore of a process in container. + +[#linux-capabilities] +=== Linux capability prerequisites for checkpoint and restore + +Unprivileged (non-root) users are supported by CRIU for checkpointing and restoring application processes. To checkpoint and restore, CRIU requires different link:https://man7.org/linux/man-pages/man7/capabilities.7.html[Linux capabilities]. + +To perform an application process checkpoint, CRIU requires the following Linux capabilities: + +- `CHECKPOINT_RESTORE` - This capability was added in Linux 5.9 to separate checkpoint/restore functions from the overloaded SYS_ADMIN capability. +- `SETPCAP` - Allows CRIU to drop capabilities from the restored application process. Because the CRIU binary is granted more capabilities, like `CHECKPOINT_RESTORE`, it needs these capabilities so it can drop such capabilities from the final process that CRIU restored. +- `SYS_PTRACE` - A powerful capability needed by CRIU to be able to capture and record the full process state. This capability is necessary only when CRIU is checkpointing an application process. For the restore process, only the `CHECKPOINT_RESTORE` and `SETPCAP` capabilities are necessary. + +[#build] +== Building an InstantOn application image + +Two options are available to build an application container image that uses InstantOn: + +- Add a special `RUN` instruction at end of a `Dockerfile` or `Containerfile` that runs the <<#checkpoint_script,checkpoint.sh script>> to perform an application checkpoint at container image build time. This option requires you to use link:https://podman.io/[Podman]. +- Use a <<#three_step_process,three-step process>> to build the application image, run the checkpoint, and commit the final result into an InstantOn application container image. This option requires you to use either Podman or Docker version 23.0 or later. + +To run the `checkpoint.sh` script, you must use Podman to build the application container image. Currently, you cannot use Docker to build the InstantOn application container image because Docker does not provide a way to grant the container build the necessary Linux capabilities. To use Docker to build an InstantOn application container image, you must follow the three-step build process. + +[#checkpoint_script] +=== Building the InstantOn image with Podman and the checkpoint.sh script + +You can use the `checkpoint.sh` script to perform the application checkpoint by adding the `RUN checkpoint.sh` instruction to the end of your `Dockerfile` or `Containerfile` file. The execution of the `checkpoint.sh` must be the last `RUN` instruction during your container image build. This configuration performs the application process checkpoint and stores the process data as the last layer of the application container image. Currently, this script requires you to use Podman rather than Docker because Docker cannot grant the necessary Linux capabilities. + +The following image template example uses the `kernel-slim-java17-openj9-ubi` tag to build an image that uses the latest Open Liberty release with the IBM Semeru distribution of Java 17. This example uses the `afterAppStart` checkpoint option. + +[source,dockerfile] +.Dockerfile +---- +FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi + +# Add a Liberty server configuration that includes all necessary features +COPY --chown=1001:0 server.xml /config/ + +# This script adds the requested XML snippets to enable Liberty features and grow the image to be fit-for-purpose. +# This option is available only in the 'kernel-slim' image type. The 'full' and 'beta' tags already include all features. +RUN features.sh + +# Add interim fixes (optional) +COPY --chown=1001:0 interim-fixes /opt/ol/fixes/ + +# Add an application +COPY --chown=1001:0 Sample1.war /config/dropins/ + +# This script adds the requested server configuration, applies any interim fixes, and populates caches to optimize the runtime. +RUN configure.sh + +# This script performs an InstantOn checkpoint of the application. +# The application can use beforeAppStart or afterAppStart to do the checkpoint. +# The default is beforeAppStart when not specified +RUN checkpoint.sh afterAppStart +---- + +Use the following Podman command to build the InstantOn application container image. To grant the necessary Linux capabilities to the container image build, this command must be run either as the `root` user or by using the `sudo` utility. + +[source,sh] +---- +podman build \ + -t dev.local/liberty-app-instanton \ + --cap-add=CHECKPOINT_RESTORE \ + --cap-add=SYS_PTRACE\ + --cap-add=SETPCAP \ + --security-opt seccomp=unconfined . +---- + +The three `--cap-add` options grant the three Linux capabilities that CRIU requires to perform the application process checkpoint during the container image build. The `--security-opt` option grants access to all Linux system calls to the container image build. + +[#three_step_process] +=== Building the InstantOn image by using the three-step process with Docker or Podman + +If you cannot use Podman to run the `checkpoint.sh` during the container image build, you can use the following three-step process to build the InstantOn application container image: + +1. Build the application container image without the InstantOn layer +2. Run the application container to perform a checkpoint of the application in the running container +3. Commit the stopped container with the checkpoint process data into an InstantOn application container image + +You can use these steps with either Podman and Docker to build an InstantOn application image. For Docker, version 23.0 or later is required. The following examples assume that you are using Docker to build an application image that is named `liberty-app`. + +==== 1. Build the application container image + +Set the image template (`Dockerfile` or `Containerfile`) similar to the following example. This example uses the `kernel-slim-java17-openj9-ubi` tag to build an image that uses the latest Open Liberty release with the IBM Semeru distribution of Java 17. This template does not run the `checkpoint.sh` script. + +[source,dockerfile] +.Dockerfile +---- +FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi + +# Add a Liberty server configuration that includes all necessary features +COPY --chown=1001:0 server.xml /config/ + +# This script adds the requested XML snippets to enable Liberty features and grow the image to be fit-for-purpose. +# This option is available only in the 'kernel-slim' image type. The 'full' and 'beta' tags already include all features. +RUN features.sh + +# Add interim fixes (optional) +COPY --chown=1001:0 interim-fixes /opt/ol/fixes/ + +# Add an application +COPY --chown=1001:0 Sample1.war /config/dropins/ + +# This script adds the requested server configuration, applies any interim fixes, and populates caches to optimize the runtime. +RUN configure.sh +---- + +To build the application container image with Docker, run the following command: + +[source,sh] +---- +docker build -t liberty-app . +---- + + +The resulting application container image, which is tagged `liberty-app`, does not contain the InstantOn checkpoint process layer. + +==== 2. Run the application container to perform a checkpoint + +Run the application container image to perform a checkpoint of the application process within the running container. The following example uses the `liberty-app` application image to run the checkpoint of the application process with the `afterAppStart` option: + +[source,sh] +---- +docker run \ + --name liberty-app-checkpoint-container \ + --privileged \ + --env WLP_CHECKPOINT=afterAppStart \ + liberty-app +---- + +This command runs the application within a container and performs an application process checkpoint. The `--env` option sets a `WLP_CHECKPOINT` environment variable to specify the checkpoint `afterAppStart` option. When the application process checkpoint completes, the `liberty-app-checkpoint-container` application container is stopped and exits. + +==== 3. Commit the stopped container with the checkpoint process data + +The stopped `liberty-app-checkpoint-container` container from the previous step contains the data from the InstantOn checkpoint process. Lastly, take this checkpoint process data and commit it to an application container image layer by running the following commit commands: + +[source,sh] +---- +docker commit liberty-app-checkpoint-container liberty-app-instanton +docker rm liberty-app-checkpoint-container +---- + +You now have two application images: `liberty-app` and `liberty-app-instanton`. Starting a container with the `liberty-app-instanton` container image shows a faster startup time than the original `liberty-app` image. The `liberty-app-checkpoint-container` stopped container is no longer needed and can safely be removed. + +[#run] +== Running and deploying an InstantOn application image + +Special considerations are required to run an InstantOn application image locally or when it is deployed to a public cloud. The following prerequisites are required to restore the InstantOn checkpoint process. + +[#required-to-restore] +1. The host that is running the container image must use Linux kernel 5.9 or greater +2. The Linux capabilities CHECKPOINT_RESTORE and SETPCAP must be granted to the running container +3. The necessary system calls must be granted to the running container +4. The host processor must be X86-64/AMD64 + +=== Running an InstantOn application image locally + +If a host system is running the Linux kernel 5.9 or greater with the X86-64/AMD64 processor, you can run an InstantOn application image by using Podman or Docker locally. The following command runs the `liberty-app-instanton` InstantOn application image with Podman: + +[source,sh] +---- +podman run \ + --rm \ + --cap-add=CHECKPOINT_RESTORE \ + --cap-add=SETPCAP \ + --security-opt seccomp=unconfined \ + -p 9080:9080 \ + liberty-app-instanton +---- + +The following command runs the `liberty-app-instanton` InstantOn application image with Docker: + +[source,sh] +---- +docker run \ + --rm \ + --cap-add=CHECKPOINT_RESTORE \ + --cap-add=SETPCAP \ + --security-opt seccomp=unconfined \ + -p 9080:9080 \ + liberty-app-instanton +---- + +In both cases, the `--cap-add` option grants the `CHECKPOINT_RESTORE` and `SETPCAP` capabilities. The `SYS_PTRACE` capability is not required to run the InstantOn application container image. + +[#required-system-calls] +==== Required Linux system calls + +The `--security-opt` option grants the running container access to all Linux system calls. Depending on the defaults of the container engine, the `--security-opt` with the `seccomp-unconfined` setting might not be required. For CRIU to restore the InstantOn application process, the container must have access to `clone3`, `ptrace`, and other system calls. This requirement is true even though the elevated Linux capability of `SYS_PTRACE` is not required to restore the process. You can update the defaults of the container engine to include all the required system calls. + +Alternatively, you can specify a file with the `--security-opt seccomp` option that specifies the policy for the container. Use the following command to specify a JSON policy file for `seccomp`: + +[source,sh] +---- +podman run \ + --rm \ + --cap-add=CHECKPOINT_RESTORE \ + --cap-add=NET_ADMIN \ + --cap-add=SYS_PTRACE \ + --security-opt seccomp=criuRequiredSysCalls.json \ + -p 9080:9080 \ + liberty-app-instanton +---- + +The resulting xref:instanton-sycalls-json.adoc[criuRequiredSysCalls.json] file grants access to all the Linux system calls that are required by CRIU to restore an InstantOn application process. + +==== Recovering from a failed InstantOn restore +If restoration of the InstantOn application process fails, Open Liberty starts the server without using the InstantOn checkpoint process. In such cases, the Open Liberty application starts as if no InstantOn checkpoint process layer exists, which takes longer than a successfully restored InstantOn process. This recovery launch from a failed InstantOn restore can be disabled by setting the following environment variable: + +[source,sh] +---- +CRIU_RESTORE_DISABLE_RECOVERY=true +---- + +After you build an InstantOn application container image, you can verify a successful restore by setting this environment variable to run locally. For example, you can run the following Podman command: + +[source,sh] +---- +podman run \ + --rm \ + --cap-add=CHECKPOINT_RESTORE \ + --cap-add=SETPCAP \ + --security-opt seccomp=unconfined \ + --env CRIU_RESTORE_DISABLE_RECOVERY=true \ + -p 9080:9080 \ + liberty-app-instanton +---- + +To avoid cloud environments from continuously trying to restart the failed start of an application container image, the default value of the `CRIU_RESTORE_DISABLE_RECOVERY` variable is `false`. + +=== Deploying an InstantOn application to Kubernetes services + +Currently, Open Liberty InstantOn is tested and supported on the following public cloud Kubernetes services: + +- link:https://aws.amazon.com/eks/[Amazon Elastic Kubernetes Service (EKS)] +- link:https://azure.microsoft.com/en-us/products/kubernetes-service[Azure Kubernetes Service (AKS)] + +Other public cloud Kubernetes services might also work if they have the <<#required-to-restore,prerequisites>> to allow the InstantOn application process to restore. + +When you deploy to Kubernetes, the container must be granted the `CHECKPOINT_RESTORE` and the `SETPCAP` Linux capabilities to allow the InstantOn application process to restore. You can configure these capabilities in the deployment YAML file by specifying the following `securityContext` for the container: + +[source,yaml] +---- + securityContext: + allowPrivilegeEscalation: true + privileged: false + runAsNonRoot: true + capabilities: + add: + - CHECKPOINT_RESTORE + - SETPCAP + drop: + - ALL +---- + +[#supported-features] +== Open Liberty InstantOn supported features + +InstantOn supports a subset of Open Liberty features. If a feature is enabled that InstantOn does not support, a failure occurs when you try to perform a checkpoint of an application process. InstantOn supports the following Jakarta EE and MicroProfile xref:reference:feature/feature-overview.adoc#conv[convenience features]: + +- Jakarta EE Web Profile versions feature:webProfile-8.0[display=8.0] and later +- MicroProfile versions feature:microProfile-4.1[display=4.1] and later + +You can individually enable the Open Liberty public features that ae enabled by the feature:webProfile[display=Jakarta EE Web Profile] and feature:microProfile[display=MicroProfile] features, depending on the needs of your application. This option avoids enabling the complete set of features that are enabled by the convenience features. + +In addition to the features that are enabled in the convenience features, InstantOn also supports the following features: + +- feature:audit-1.0[] +- feature:bells-1.0[] +- feature:distributedMap-1.0[] +- feature:federatedRegistry-1.0[] +- feature:ldapRegistry-3.0[] +- feature:monitor-1.0[] +- feature:openidConnectClient-1.0[] +- feature:passwordUtilities-1.1[] +- feature:restConnector-2.0[] +- feature:sessionDatabase-1.0[] +- feature:socialLogin-1.0[] +- feature:webCache-1.0[] + +For more information about limitations, see xref:instanton-limitations.adoc[InstantOn limitations and known issues]. \ No newline at end of file diff --git a/modules/ROOT/pages/integration-testing.adoc b/modules/ROOT/pages/integration-testing.adoc index 6c91f9589f..15597ff189 100644 --- a/modules/ROOT/pages/integration-testing.adoc +++ b/modules/ROOT/pages/integration-testing.adoc @@ -1,4 +1,4 @@ -// Copyright (c) 2020 IBM Corporation and others. +// Copyright (c) 2023 IBM Corporation and others. // Licensed under Creative Commons Attribution-NoDerivatives // 4.0 International (CC BY-ND 4.0) // https://creativecommons.org/licenses/by-nd/4.0/ @@ -10,24 +10,18 @@ // :page-layout: general-reference :page-type: general -:page-description: MicroShed testing helps you to write integration tests by using Testcontainers framework for Java microservice applications. With MicroShed testing you can test your Open Liberty application from outside the container so you are testing the exact same image that runs in production. -:page-categories: MicroShed testing -:seo-title: Testing in a container with MicroShed testing -:seo-description: MicroShed testing helps you to write integration tests by using Testcontainers for Java microservice applications. With MicroShed testing, you can test your Open Liberty application from outside the container so you are testing the exact same image that runs in production. -= True-to-production integration testing +:page-description: Testcontainers is a Java library that facilitates the creation of integration tests that closely resemble production environments. It enables developers to test their applications within the same container used in production, thus reducing discrepancies between development and deployment settings. +:page-categories: integration testing +:seo-title: True-to-production integration testing +:seo-description: Testcontainers is a Java library that facilitates the creation of integration tests that closely resemble production environments. It enables developers to test their applications within the same container used in production, thus reducing discrepancies between development and deployment settings. -MicroShed Testing is a Java library that helps you write true-to-production integration tests for your application in an environment similar to production. -To minimize the parity issues between development and production, you can test your application in the same container that you use in production. += True-to-production integration testing -https://microshed.org/microshed-testing/[MicroShed Testing] uses the https://www.testcontainers.org/[Testcontainers] framework to analyze your application from outside the container without accessing the application internals. -You can use MicroShed Testing to develop integration tests for your Open Liberty application. +Testcontainers is a Java library that facilitates the creation of integration tests that closely resemble production environments. It enables developers to test their applications within the same container used in production, thus reducing discrepancies between development and deployment settings. -You need to make sure that your application works accurately in production, similar to the development environment. -Integration tests assess multiple test classes and components, but integration tests take longer to set up and configure than unit tests. -Alternatively, unit tests are shorter and involve testing individual modules of an application. -With shorter development cycles and limited time available, developers often run unit tests. -MicroShed Testing helps you write and run true-to-production integration tests for your applications, and streamlines your integration tests with the Testcontainers framework for an efficient workflow. +By using the https://www.testcontainers.org/[Testcontainers] framework, developers can analyze their applications externally, without accessing internal application details. This approach ensures that integration tests accurately evaluate various test classes and components. Although integration tests require more setup and configuration time compared to unit tests, they provide a comprehensive assessment of the application’s behavior in a realistic environment. +Unit tests focus on testing individual modules or components of an application and are quicker to run. Due to shorter development cycles and time constraints, developers often prioritize running unit tests. However, with Testcontainers, developers can efficiently write and run integration tests that closely resemble real-world production scenarios. == Development and production parity @@ -36,54 +30,102 @@ The idea behind development and production parity is to keep the development, st To simplify the development process, developers often use tools in development that are different from production. For example, you might use a local Maven build to build a project for your application in development, but the application might be deployed to a container in production. Differences between the environments can cause a test to fail in production, though the test passes in the development environment. -MicroShed Testing helps achieve development and production parity by testing the application in an environment similar to production. +Testcontainers helps achieve development and production parity by testing the application in an environment similar to production. -== Writing integration tests with the Microshed Testing library +== Writing integration tests with the Testcontainers framework -You can deploy your applications in different environments with containers. -With the Testcontainers framework, you can use containers in a test environment. -Microshed Testing is a Java library for MicroProfile and https://jakarta.ee/[Jakarta EE] developers to test their applications in an environment similar to production. -Microshed Testing implements the Testcontainers framework to support development and production parity in testing. +Testcontainers is a Java library that enables you to thoroughly test your https://microprofile.io/[MicroProfile] and https://jakarta.ee/[Jakarta EE] applications in a testing environment that closely resembles your production setup. -With MicroShed Testing, you can write an integration test that looks like the following example: +In the following example, Testcontainers is used to start a Docker container that is running an Open Liberty server with a test application deployed. It uses https://rest-assured.io/[REST Assured] to send HTTP requests to the service. This test case adds a person, retrieves all people, and verifies the correctness of the data. ```java -@MicroShedTest -public class BasicJAXRSServiceTest { +@Testcontainers +public class ServiceTest { - @Container - public static ApplicationContainer app = new ApplicationContainer() - .withAppContextRoot("/myservice"); + // Latest liberty image + static final DockerImageName libertyImage = DockerImageName.parse("open-liberty:23.0.0.4-full-java11-openj9"); - @RESTClient - public static PersonService personSvc; - - @Test - public void testGetPerson() { - Long bobId = personSvc.createPerson("Bob", 24); - Person bob = personSvc.getPerson(bobId); - - assertEquals("Bob", bob.name); - assertEquals(24, bob.age); - assertNotNull(bob.id); + // Create container and copy application + server.xml + @Container + static final GenericContainer liberty = new GenericContainer<>(libertyImage) + .withExposedPorts(9080, 9443) + .withCopyFileToContainer(MountableFile.forHostPath("build/libs/testapp.war"), "/config/dropins/testapp.war") + .withCopyFileToContainer(MountableFile.forHostPath("build/resources/main/liberty/config/server.xml"), "/config/server.xml") + .waitingFor(Wait.forLogMessage(".*CWWKZ0001I: Application .* started in .* seconds.*", 1)) + .withLogConsumer(new LogConsumer(ServiceTest.class, "liberty")); + + // Setup RestAssured to query our service + static RequestSpecification requestSpecification; + + @BeforeAll + static void getServiceURL() { + String baseUri = "http://" + liberty.getHost() + ":" + liberty.getMappedPort(9080) + "/testapp/people"; + requestSpecification = new RequestSpecBuilder() + .setBaseUri(baseUri) + .build(); } + // Run our test @Test - public void testGetUnknownPerson() { - assertThrows(NotFoundException.class, () -> personSvc.getPerson(-1L)); + public void testAddPerson() { + // Add new person to service + given(requestSpecification) + .header("Content-Type", "application/json") + .queryParam("name", "bob") + .queryParam("age", "24") + .when() + .post("/") + .then() + .statusCode(200); + + // Get and verify only one person exists + List allIDs = given(requestSpecification) + .accept("application/json") + .when() + .get("/") + .then() + .statusCode(200) + .extract().body() + .jsonPath().getList("id"); + + assertEquals(1, allIDs.size()); + + // Verify person is bob + String actual = given(requestSpecification) + .accept("application/json") + .when() + .get("/" + allIDs.get(0)) + .then() + .statusCode(200) + .extract().body() + .jsonPath().getString("name"); + + assertEquals("bob", actual); } -} ``` -The `@MicroShedTest` annotation searches for a `Dockerfile` or `Containerfile` file in the repository, starts the application in a container, and waits for the application to be ready before the test starts. -The `@Container` annotation injects a REST Client proxy of the PersonService class, which helps you send HTTP requests on the running application container. -The `@RESTClient` annotation sends an HTTP POST request to the running container, which triggers the PersonService#createPerson endpoint and returns the generated ID. -By using the generated ID, you can send an HTTP GET request to read the record that was created. -The JSON response automatically converts to a Person object by using a JSON-B token. -The `@Test` annotation sends an HTTP GET request to find a Person object with the `-1` ID, which does not exist. -The annotation asserts that the application container returns an HTTP 404 exception. +=== Test Elements + +The `@Testcontainers` annotation is a custom Junit5 extension that controls the lifecycle of the Docker container (create, run, stop, and delete) within the scope of this test class. + +The `ServiceTest` class starts with a static field `libertyImage`, which represents the Docker image to be used for the test. + +The `@Container` annotation marks the field that constructs the Docker container by using the `libertyImage` image. It makes ports `9080` and `9443` accessible and copies the application and server configuration files into the container. It waits for a log message that indicates that the application started successfully and assigns a log consumer for logging container logs. + +The `@BeforeAll` annotation marks the method that creates the base URL for the service is constructed by using the host and mapped port of the liberty container. The request specification is created with the base URL. + +The `@Test` annotation marks the method that runs a series of tests on the application service. + +=== Test Logic + +Inside the test method, a new person is added to the service by sending a POST request with JSON payload that contains the name and age parameters. + +Upon adding the person, a GET request is sent to retrieve all people from the service. The response is validated to ensure a successful status code `(200)`. The IDs of all the people are extracted from the response by using JSONPath. + +The number of IDs is checked to ensure that only one person exists in the service. + +Another `GET` request is sent to retrieve the details of the person with the extracted ID. The response is validated to ensure a successful status code `(200)`. The name of the person is extracted from the response by using JSONPath. -== See also +Finally, the extracted name is compared with the expected value `bob` using the `assertEquals()` method. -Guide: https://openliberty.io/guides/microshed-testing.html[Testing a MicroProfile or Jakarta EE application] diff --git a/modules/ROOT/pages/verify-signatures-for-container-images-in-open-liberty.adoc b/modules/ROOT/pages/verify-signatures-for-container-images-in-open-liberty.adoc new file mode 100644 index 0000000000..30d6620302 --- /dev/null +++ b/modules/ROOT/pages/verify-signatures-for-container-images-in-open-liberty.adoc @@ -0,0 +1,104 @@ +// Copyright (c) 2023 IBM Corporation and others. +// Licensed under Creative Commons Attribution-NoDerivatives +// 4.0 International (CC BY-ND 4.0) +// https://creativecommons.org/licenses/by-nd/4.0/ +// +// Contributors: +// IBM Corporation +// +:page-description: +:seo-title: Verify images for Open Liberty Operator +:seo-description: All images that are associated with the Open Liberty operator are digitally signed. Verify the downloaded images and ensure that the images are created by using the image signatures and uploaded by Open Liberty. +:page-layout: +:page-type: + += Verify signatures for container images in Open Liberty + +All the Open Liberty container images are digitally signed. You can use the Open Liberty public key to check the signature, verify that the image was created by Open Liberty, and that it was not modified since its release. + +== Before you begin + +Before you can verify image signatures, you must install the following tools: + +* https://gnupg.org/software/index.html[GNU Privacy Guard v2] +* https://www.redhat.com/en/topics/containers/what-is-skopeo#overview[Skopeo] + + +The following Open Liberty public key must exist on the same machine as the command-line tools. Copy the following text box exactly as shown into a text editor and save it as a file named `ol.gpg`. + +---- +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBGPmbEgBEADngOaoWkCGOA6wEXSEiy0p/ccG0vmRdMQzm4o0sIPsZ+spdj1b +VQAZx23yGctE30JTu2YAaZEyH2cqgO+iEGMSz9yY6PDwcIn5y7N+tNXoUKdinPJ2 +6T4bTypJ1SileGncUVU3U1xAZIH2Cmcn5wmXS/9GKIbtkUQ708RQLKVEFESpjgjr +VKjpxn8pd+0niQniq5h7iNf6cYAQnCsdH/z8oifWnacipEuBDJ9lF30YRiLhK7m6 +FTOdBGeF7XohUOI/AGw0erEuTc80YvX/i+IUmf/hqHxY+A/hEjgFkvXxh8zAmCDU +It1v/LV+c9vdjz/hNb+jGI1te3z3xMDQz1WCMIYflyw6ex9nmP7EcFFPG/kRVAMx +cu+glm71QC7Q9ftx1ZvAfYXhNMCr6b91z1e3XYM/Whx9qNkyc8MnoZtq5lxJGRkL +rgB8RrWGP55R7oWbEn80ppT6MtINy6Ft7TZhnjsI6VEsEcB2qi6TC/0/29z9c2z6 +DuFFDQ/y/Naqb+mbDhdahhX1t5ABWTgcv+h+EZzs1QRV/ScBFqJMgV6O+BNbsXpZ +/1z0crs6yAHQujK4mgoHEyEiP7/3OVF1Wazlyap9PwU7Y4LKEfdrO2+CIR5AuivR +3PRNhU5NWYvEn9bnIXMsacAfLBTv5lFZqT15PFggtoonDZJHCIncnoQyQwARAQAB +tD5JbnRlcm5hdGlvbmFsIEJ1c2luZXNzIE1hY2hpbmVzIENvcnBvcmF0aW9uIDxw +c2lydEB1cy5pYm0uY29tPokCOgQTAQgAJAUCY+ZsSAIbDwULCQgHAgYVCgkICwIE +FgIDAQIeAQUJB4TN/wAKCRC9n9W+nmjKAGHmD/9QDewA9bYLE6SAePCyFg2/tWAe +U2Ot+nO2QtUn0amF0hPNIv2q4sgVoJ/4tfMyK2JytuvfHCbWlM6rGaUhAdOlR83Q +ATl8d1KhWEOnYDwiBrOlOkyn/Qw3Wv209bNDpd3Y6l1Kootv8F1xbCszkYc1FsJB +Y3i4Cu2asQR3eMCNqAQTIsfKQRWc6dtn9/L0allKR+OOAJRlHZ3woRUbb0NeOYjM +44awORcHvuO7nivIWNntKrqCg/twVVtQrVcLbr9p60adTLQ2kC/E6t5J2G0gy4sX +jy+GAHdZlXr8Yl0wUE36n5bmo016gfV1mlxj1StpHi4kVTB/8xEQBBxaPnaRB7TF +SCTIhAdTt6+Ko+QVLGMEChDJ5HSzDKzSVK/p6BhesFz5xADH2aNdMw4xzwf56Ooa +prAaTQtSE9/LlQQZ+GvAJaeUj8e1Dtu0yQrXanyrMkRl1iNywYNXTB/cbALp06SI +BcRRdyVghNF2kcm6sEhWKS2IyTqLNM7t3UV9SLZanBT1/PlBHifoC/GWR8mAVcUl +wWWLiisj04+mDuy/J9Gu6R8PNS+WQo6aHZPOjN3k1boHjrLzpc6/0DX5p+1AwsYo +Omz0ERJdt888LZ4HY+iEC/G2VMqCMn8iS1Q0KbZ+H8gS3Q194UqFUa6M3JOmfZdQ +rRtygUT5grK8h6/pFg== +=rXKG +-----END PGP PUBLIC KEY BLOCK----- +---- + +== Procedure + +. Import the Open Liberty public key file to GPG. ++ +---- +gpg2 --import ol.gpg +---- ++ +. Run the following command to export the fingerprint to make it available for the verification command. ++ +---- +export fingerprint=E70E5D6C3F1E452CB0F67DF1BD9FD5BE9E68CA00 +---- ++ +. To create a directory for an each image and pull the image into local storage using Skopeo, follow these steps: +* Create a directory for the image with a name of your choice, as shown in the following example: ++ +---- +mkdir OpenLiberty-image +---- ++ +* Use Skopeo to copy the container image to the newly created directory, as shown in the following example: ++ +---- +skopeo copy docker://icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi-amd64 dir:./OpenLiberty-image +---- ++ +This command pulls the `icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi-amd64` image and stores it in the `OpenLiberty-image` directory. Replace these values with the image that you want to verify. This command also copies the signature file that you use to verify the image. ++ +. Run the `standalone-verify` command to verify the image by using the downloaded signature and repository, as shown in the following example: ++ +---- +skopeo standalone-verify ./OpenLiberty-image/manifest.json icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi-amd64 ${fingerprint} ./OpenLiberty-image/signature-1 +---- ++ +The `OpenLiberty-image` refers to the location of the copied image. The `./OpenLiberty-image/signature-1` specifies the path to the signature file that is associated with the container image. + +== Result + +If the verification is successful, the following message is displayed: +---- +Signature verified +---- + diff --git a/modules/reference/pages/command/server-checkpoint-draft.adoc b/modules/reference/pages/command/server-checkpoint-draft.adoc new file mode 100644 index 0000000000..d41a4f135f --- /dev/null +++ b/modules/reference/pages/command/server-checkpoint-draft.adoc @@ -0,0 +1,97 @@ +// +// Copyright (c) 2020 IBM Corporation and others. +// Licensed under Creative Commons Attribution-NoDerivatives +// 4.0 International (CC BY-ND 4.0) +// https://creativecommons.org/licenses/by-nd/4.0/ +// +// Contributors: +// IBM Corporation +// +:page-layout: server-command +:page-type: command + +// This command is currently an implementation detail of the checkpoint script and so is not documented externally. Keeping this draft in case that changes in a future release. DM 6/23 + += server checkpoint command + +The `server checkpoint` command checkpoints an Open Liberty server to enable InstanOn startup. + +== Usage examples + +Checkpoint a server called `myserver` by using the default `beforeAppStart` checkpoint: + +---- +server checkpoint myserver +---- + +Checkpoint a server called `myserver` by using the `afterAppStart` checkpoint: + +---- +server checkpoint myserver --at=afterAppStart +---- + +== Syntax + +Run the command from the `_path_to_liberty_/wlp/bin` directory. + +[subs=+quotes] +---- +server checkpoint _serverName_ [_options_] +---- + +serverName:: +The name of the server to run the command against. If no server name is specified, the `defaultServer` server is used to checkpoint. If no `defaultServer` server exists, the command creates one and checkpoints it. + +== Options + +.Options for the server run command +[%header,cols=2*] +|=== +|Option +|Description + +|--at=_checkpoint_phase_ +|Specifies the checkpoint phase. +The valid values are `beforeAppStart` or `afterAppStart`. If no value is specified, the default value is `beforeAppStart`. + +|=== + + +== Exit codes + +The following exit codes are available for the `server checkpoint` command: + +.Exit codes for the server create command +[%header,cols="2,9"] +|=== + +|Code +|Explanation + +|0 +|This code indicates successful completion of the checkpoint operation. + + +|70 +|The Java version used does not support checkpoint. + +|71 +|The running Java instance is not enabled to support checkpoint. The Java commandline option `-XX:+EnableCRIUSupport` must be specified. + +|72 +|Open Liberty failed to prepare for checkpoint. + +|73 +|The JVM failed to prepare for checkpoint. + +|74 +|CRIU failed to checkpoint the process. + +|75 +|An unknown error occurred during the checkpoint of the process. + +|=== + +== See also + +* xref:ROOT:instanton.adoc[Fast startup for containerized applications Open Liberty InstantOn] diff --git a/modules/reference/pages/feature/openidConnectClient/examples.adoc b/modules/reference/pages/feature/openidConnectClient/examples.adoc index 5f55929427..54953cb4cf 100644 --- a/modules/reference/pages/feature/openidConnectClient/examples.adoc +++ b/modules/reference/pages/feature/openidConnectClient/examples.adoc @@ -8,6 +8,7 @@ The following examples demonstrate configuration options for the OpenID Connect - <<#jwe,Accept tokens in JWE format>> - <<#third,Configure a third-party OpenID Connect provider>> - <<#filter,Support Multiple OpenID Connect Providers>> +- <<#privatekey,Use Private Key JWT for client authentication>> - <<#ltpa,Disable LTPA cookies>> - <<#oauth,Accept an OAuth 2.0 bearer access token without redirecting to an OpenID Connect provider>> - <<#context,Modify the context root>> @@ -283,6 +284,27 @@ In the following example, authentication requests that contain the `/mywebapp/me For more information, see xref:ROOT:authentication-filters.adoc[Authentication filters]. +[#privatekey] +=== Use Private Key JWT for client authentication + +OpenID Connect clients in Open Liberty support the `private_key_jwt` client authentication method with OpenID Connect token endpoints. OpenID Connect clients need to provide authentication data to the OpenID Connect provider for accessing the provider's token endpoint. Clients can authenticate by using several different methods, but most of those methods require a client secret. The `private_key_jwt` authentication method allows clients to use asymmetric keys to create signed JWTs to authenticate instead of client secrets. By using this authentication method, OpenID Connect clients in Open Liberty no longer need to have a client secret. + +Server administrators can enable this function by using the `private_key_jwt` option for the `tokenEndpointAuthMethod` attribute, and the new `tokenEndpointAuthSigningAlgorithm` and `keyAliasName` attributes in the `openidConnectClient` element. + +[source,xml] +---- + +---- + +The `tokenEndpointAuthSigningAlgorithm` attribute specifies the signing algorithm to sign the JWT that is used for client authentication. The `keyAliasName` attribute points to the private key to use to sign the JWT. The private key must be present in the keystore that is specified by the `sslRef` attribute in the config:openidConnectClient[display=OpenID Connect client configuration]. OpenID Connect providers that support Private Key JWT client authentication typically provide an interface for administrators to upload their corresponding public key. + +For more information on `private_key_jwt` client authentication, see the https://openid.net/specs/openid-connect-core-1_0.html#ClientAuthentication[OpenID Connect core specification] and https://datatracker.ietf.org/doc/html/rfc7523[RFC 7523]. + [#ltpa] === Disable LTPA cookies diff --git a/modules/reference/pages/feature/socialLogin/examples.adoc b/modules/reference/pages/feature/socialLogin/examples.adoc index 2c8f9c2c1c..919a119323 100644 --- a/modules/reference/pages/feature/socialLogin/examples.adoc +++ b/modules/reference/pages/feature/socialLogin/examples.adoc @@ -3,6 +3,19 @@ The Social Media Login feature provides a set of built-in configuration elements for popular social media providers, in addition to generic configuration elements. The following examples show how to configure different scenarios in the `server.xml` file. + +- <<#socialmediaid,Request login with a social media ID>> +- <<#socialmediaproviders,Provide a choice of social media providers to the user>> +- <<#authenticatedusers,Require authenticated users to also be present in the configured registry>> +- <<#loginwithasocialmediaid,Request login with a social media ID or with their account for the configured registry>> +- <<#loginforsubset,Request login with a social media ID for only a subset of applications, URLs, browsers, or IP addresses>> +- <<#loginsasoptions,Provide other social media logins as options to the user>> +- <<#privatekeyjwt,Use Private Key JWT for client authentication>> +- <<#openshiftservice,Use OpenShift service accounts to authenticate and authorize protected resource requests>> +- <<#activedirectory,Use Active Directory as an authentication provider>> + + +[#socialmediaid] === Request login with a social media ID If multiple social media providers are configured, a customizable selection form is presented. @@ -24,12 +37,13 @@ In this example, `your_app_id` and `your_app_secret` are the Google client ID an ---- +[#socialmediaproviders] === Provide a choice of social media providers to the user You can configure multiple social media providers for users to choose from. The user receives a customizable selection form before authentication with the available providers included on the form. -The following example includes a choice of Google, GitHub, Facebook, LinkedIn, and Twitter: +The following example includes a choice of Google, GitHub, Facebook, LinkedIn, and Twitter. [source,xml] ---- @@ -49,13 +63,13 @@ The following example includes a choice of Google, GitHub, Facebook, LinkedIn, a ---- - +[#authenticatedusers] === Require authenticated users to also be present in the configured registry You can restrict the presentation of a social media provider to only users who are also in another configured registry. For example, use the `mapToUserRegistry` attribute to configure your application. -The user must authenticate through Google and also verify as an existing user in the company LDAP registry: +The user must authenticate through Google and also verify as an existing user in the company LDAP registry. [source,xml] ---- @@ -67,10 +81,11 @@ The user must authenticate through Google and also verify as an existing user in For more information on configuring an LDAP registry, see the feature:ldapRegistry[display=LDAP User Registry feature]. +[#loginwithasocialmediaid] === Request login with a social media ID or with their account for the configured registry You can give users the option of logging in with either a social media provider or with their account on the configured registry. -For example, use the `enableLocalAuthentication` attribute to configure your application so that users can log in with a Google ID or with their account on their company's LDAP registry: +For example, use the `enableLocalAuthentication` attribute to configure your application so that users can log in with a Google ID or with their account on their company's LDAP registry. [source,xml] ---- @@ -83,11 +98,12 @@ For example, use the `enableLocalAuthentication` attribute to configure your app ---- +[#loginforsubset] === Request login with a social media ID for only a subset of applications, URLs, browsers, or IP addresses To protect only a subset of applications, URLs, or IP addresses, use an xref:ROOT:authentication-filters.adoc[authentication filter]. The security configuration takes effect only when the conditions in the filter are met. For example, -you might want a web application to be secured with the Social Media Login feature and a microservice application to be secured with the feature:mpJwt[display=MicroProfile JWT feature]. In the following example, only requests that contain the `/mywebapp` URL pattern are secured with Google credentials: +you might want a web application to be secured with the Social Media Login feature and a microservice application to be secured with the feature:mpJwt[display=MicroProfile JWT feature]. In the following example, only requests that contain the `/mywebapp` URL pattern are secured with Google credentials. // tag::authfilter[] @@ -107,12 +123,13 @@ you might want a web application to be secured with the Social Media Login featu For more information, see the config:authFilter[display=authFilter element]. +[#loginsasoptions] === Provide other social media logins as options to the user To authenticate with a social media provider that isn't preconfigured with Open Liberty, use the config:oauth2Login[display=oauth2Login element] for OAuth providers or the config:oidcLogin[display=oidcLogin element] for OpenID Connect providers. These elements supply the configuration details that are needed to work with the provider. The details can usually be obtained from the developer instructions of the social media provider. -The following example configures Instagram as the social media provider: +The following example configures Instagram as the social media provider. [source,xml] ---- @@ -127,14 +144,31 @@ The following example configures Instagram as the social media provider: ---- +[#privatekeyjwt] +=== Use Private Key JWT for client authentication + +OpenID Connect clients that are configured by using the `oidcLogin` element in the Social Media Login feature support the `private_key_jwt` client authentication method with OpenID Connect token endpoints. The process for enabling this support in the Social Media Login feature is identical to the xref:reference:feature/openidConnectClient/examples.adoc#privatekey[OpenID Connect Client 1.0] feature. + +The following example shows how to use a private key JWT for client authentication. + +[source,xml] +---- + +---- +[#openshiftservice] === Use OpenShift service accounts to authenticate and authorize protected resource requests The Social Media Login feature can be configured to use OpenShift service accounts to authenticate and authorize protected resource requests. With this configuration, server administrators can secure endpoints, for example, monitoring and metrics endpoints, that might produce sensitive information. The service accounts can authenticate themselves by providing a service account token that was created within the OpenShift cluster in the request. -The following example shows how to configure the Social Media Login feature to use OpenShift service accounts as a single sign-on provider: +The following example shows how to configure the Social Media Login feature to use OpenShift service accounts as a single sign-on provider. [source,xml] ---- @@ -145,9 +179,10 @@ The following example shows how to configure the Social Media Login feature to u The `okdServiceLogin` element authenticates all protected resource requests received by Open Liberty. The OpenShift project that the service account is in, is used as the group for the service account for authorization decisions. +[#activedirectory] === Use Active Directory as an authentication provider -You can configure an Open Liberty server to use Active Directory as an authentication provider for protected resources. The `oidcLogin` element configures a social login by using the OpenID Connect protocol. With OpenID Connect, the discovery endpoint URL provides the information that the client needs to interact with the authentication provider, which in this case is Active Directory. In the following example, the `discoveryEndpoint` attribute specifies the endpoint URL for Active Directory: +You can configure an Open Liberty server to use Active Directory as an authentication provider for protected resources. The `oidcLogin` element configures a social login by using the OpenID Connect protocol. With OpenID Connect, the discovery endpoint URL provides the information that the client needs to interact with the authentication provider, which in this case is Active Directory. In the following example, the `discoveryEndpoint` attribute specifies the endpoint URL for Active Directory. [source,xml] ----