Support LTPA rotation without requiring planned outage

[NottyCode]

Description of the high level feature, including any external spec links:

Idea: make Liberty feature parity with traditional WAS to allow LtpaToken key rotation without having to restart the Liberty profile.
LtpaToken SSO key rotation in Liberty currently requires a planned outage, unlike traditional WAS which can rotate the key on the fly. The traditional WAS administrative console setting controls for automatic key re-generation filled a function in high-security sites to make mounting a session attack prohibitively costly. The feature created a new key without requiring any administrator intervention, nor restarting of WAS profiles.
There is no way to re-generate the LTPA keys on-demand without a Liberty profile restart so it can be automated out-of-band of Liberty; and even that procedure is not shown in the product documentation as far as I can tell.
Thus, using third-party information sources and not the product documentation, as best as I can tell the only way to implement a high-security configuration with Liberty profiles without interrupting the application is only under applications like IBM Workload Scheduler that support session persistence. We would have to architect a load-balancer-fronted multiple-master configuration that fails over to a standby, transparently carrying sessions with it, drop the target master from the load balancer rotation, shut down the profile, force the password change of the LTPA key, start up the profile which generates a new LTPA key, and then return the node to the load balancer rotation. That's a big increase in operational complexity to rotate the LTPA key with the same no-restart functionality as traditional WAS.

---

When complete & mandatory, add links to the UFO (Upcoming Feature Overview) document, FTS (Feature Test Summary), blogs post issues(s), and Aha (externally raised RFEs):

• UFO: https://ibm.box.com/s/5xphq0qr7mwhc1cu1riaqdm0uoygqvsd
• FTS: Feature Test Summary - for LTPA key rotation enhancement #25669
• Beta Blog Post (if applicable): BETA BLOG - Support LTPA keys rotation without requiring planned outage #26138
• Blog Post: GA BLOG - Support LTPA keys rotation without requiring planned outage #27048
• Aha:
  • https://cloud-platform.ideas.ibm.com/ideas/LIBERTY-I-12

---

List of Steps to complete or get approvals / sign-offs for Onboarding to the Liberty release (GM date)

Instructions:

• Do the actions below and mark them complete in the checklist when they are done.
• Make sure all feature readiness approvers put the appropriate tag on the epic to indicate their approval.

Design

Before Development Starts or 8 weeks before Onboarding

• POC Design / UFO Review Scheduled (David Chang) or N/A.
• POC Design / UFO Reviewed (Feature Owner) or N/A.
• Complete any follow-ons from the POC Review.
• Design / UFO Approval (Alasdair Nottingham) or N/A.
• N/A - No Design / No UFO Approval (Alasdair Nottingham) or N/A.
• SVT Requirements identified. (Epic owner / Feature owner with SVT focal point)
• ID Requirements identified (Documenting Open Liberty). (Epic owner / Feature owner with ID focal point)
• Create a child task of this epic entitled "Feature Test Summary" via this template. Add the link in above.

Before proceeding to any items below (active development), this feature MUST be prioritized on the backlog, and have been socialized (e.g., UFO Review). Follow the Feature and UFO Approval Process.

Development

When active development has begun

• Add the "In Progress" label to this issue.

Beta

In order to facilitate early feedback from users, all new features and functionality should first be released as part of a beta release.

Beta Code

• Beta fence the functionality
  • kind=beta, ibm:beta, ProductInfo.getBetaEdition()
  • add beta tag for LTPA key rotation #26060
• Beta development complete and feature ready for inclusion in a beta release
  • Add label target:beta and the appropriate target:YY00X-beta (where YY00X is the targeted beta version).
• Feature delivered into beta
  • (OpenLiberty/release-manager) adds label release:YY00X-beta (where YY00X is the first beta version that included the functionality).

Beta Blog (Complete 1.5 weeks before beta eGA)

• Beta blog issue created and populated using the Open Liberty BETA blog post template.
  • Add a link to the beta blog issue in the Documents section.
  • Note: This is for inclusion into the overall beta release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

Legal

3 weeks before Onboarding

• Identify all open source libraries that are changing or are new. Work with Legal Release Services (Cass Tucker or Release PM) to get open source cleared and approved. Or N/A. (Epic Owner). New or changed open source impacts license and Certificate of Originality.

Translation

3 weeks before Onboarding

• All new or changed PII messages are checked into the integration branch, before the last translation shipment out. (Epic Owner)

Feature Complete

2 weeks before Onboarding

• Implementation complete. (Epic owner / Feature owner)
• All function tests complete. Ready for FAT Approval. (Epic owner / Feature owner)
• Review all known issues for Stop Ship. (Epic owner / Feature owner / PM)

Focal Point Approvals

2 to 1 week before Onboarding

You MUST have the Design Approved or No Design Approved label before requesting focal point approvals.

All features (both "Design Approved" and "No Design Approved")

• FAT - (OpenLiberty/fat-approvers). SOE FATS are running successfully or N/A . Approver adds label focalApproved:fat to the Epic in Github.
• Demo - (Tom Evans or Chuck Bridgham). Demo is scheduled for an upcoming EOI. Approver adds label focalApproved:demo to the Epic in Github.
• Globalization (Sam Wong - Liberty / Simy Cheeran - tWAS). Translation is complete or N/A. TVT - complete or N/A. Approver adds label focalApproved:globalization to the Epic in Github.

"Design Approved" features

• Accessibility - (Steven Zvonek). Accessibility testing is complete or N/A. Approver adds label focalApproved:accessibility to the Epic in Github.
• ID - (Karen Deen). Documentation work is complete or N/A . Approver adds label focalApproved:id to the Epic in Github.
• Performance - (Jared Anderson). Performance testing is complete with no high severity defects or N/A . Approver adds label focalApproved:performance to the Epic in Github.
• Serviceability - (Don Bourne). Serviceability has been addressed.
• STE - (Swati Kasundra). STE chart deck is complete or N/A . Approver adds label focalApproved:ste to the Epic in Github.
• SVT - (Brian Hanczaryk - APS). SVT is complete or N/A . Approver adds label focalApproved:svt to the Epic in Github.

Ready for GA

1 week before Onboarding

• No Stop Ship issues for the feature. (Epic owner / Feature owner / Release PM)
• Ship Readiness Review and Release Notes completed (Epic owner / Feature owner / Release PM)
• Github Epic and Epic's issues are closed / complete. All PRs are committed to the release branch. (Epic owner / Feature owner / Backlog Subtribe PM)

1 week before GA

• Create, populate, and link to the Blog post issue

Other deliverbles

• OL Guides - (Yee-Kang Chang). Assessment for OL Guides is complete or N/A.
• WDT - (Leonard Theivendra). WDT work complete or N/A.
• Blog - (Laura Cowen) Blog article writeup (Epic owner / Feature owner / Laura Cowen)

[utle]

Meeting notes:

• Cover page: Typo planed >>> planned

p1: Fixed typo

• P 3: Question - Are LTPA keys unique per server instance? Tom noticed that the keys are stored in the container when created. We cannot leave an LTPA key in the container. Verify if the configure.sh is deleting the LTPA keys. Will take offline but that discussion needs result in updates on page 31 for container considerations.

p33: Added the following for container considerations:
Liberty operator will provide a better user experience
https://github.com/WASdev/websphere-liberty-operator/issues/380

• P 7: Add JWT user >>> LTPA/JWT SSO user.

p7: Added JWT in bullets two

• P 12: Ut mentioned there was a typo, but I didn't catch it. He also mentioned that step 7 should return 200 code.

p12: Fixed typo.

• P 13: Question on how do you get the LTPA key2 and how this is updated in a container? Steps on page 11 talk about using config MBean to update. Operators don’t have any LTPA support to manage. Need day2 operation to recycle LTPA keys. Operator needs to handle across all pods. Open issue for operator. Does file monitoring help? Are there other patterns other than file based? tWAS uses a keystore. Should some other approach be used? Certificates might be able to be used? Could LTPA keys stored elsewhere?

p11: Added the following bullet:
Pre-req - Config update Trigger with mbean

p33: added  the following for container considerations:
The Liberty operator will provide a better user experience  by automating the steps defined in the p11
https://github.com/WASdev/websphere-liberty-operator/issues/380

• Can you have a directory to where the keys are placed? Can config dropins be used? - not for keys, but could for the config pointing to the keys. But not a best practice if auto update is disabled.

This needs further discussion to be taken offline with updates to the cloud considerations page.

p19: Added monitoryDirectory attribute so they can place the LTPA keys file in the directory

• P 19: Config page Used to verify incoming LTPA token Config naming updates requested - change:

• verifyKeysFile to validationKeys name to fileName Discussion also on expirationDate. Should that be validUntil? It was asked whether it should be a duration, but that way the key file would have to be read to validate. No final decision there. Noted that the implementation will ignore expired keys. Discussion - Is expirationDate required? No There is a use case where expirationData is not required.

p19: 
- Used element name validationKeys.  
- Changed expirationDate to notUseAfterDate. It's optional.

• P 22: Add a config example in its proper context.

p20, p21 and p22: Added description for all examples

• In the first bullet, "ltpa" should be "LTPA/JWT" main bullet point is misleading - no auto generation. Concern that it doesn’t say which cookie it applies to - just SSO cookie Rename useContextRootAsCookiePath to useContextRootAsSSOCookiePath Question - is webAppSecurity per application - No - applies to all applications. Does changing the config values cause application restart? No

p24: Changed attribute useContextRootAsCookiePath to useContextRootAsSSOCookiePath

• P 31: Add container/operator considerations as previously brought up in the UFO discussion.

p33: Updated the slide with the container considerations issue.

• P 33: Serviceability

• The third bullet - message should be an INFO message (maybe warning)? Add another new ERROR message if the expirationDate value is bad and is not a valid ISO date. p 27: On the Beta slide - Add that new config properties need the IBM Beta tag.

p35:
- Changed the error message to information message.
- Added new error message for the validationKeys passed the specified notUseAfterData. 

p29:  Added beta tag.

• This might need a comeback after updates depending on the amount of changes that go in. Review with Alasdair after updates are made.

• Notes from chat: from Adam Yoho (IBM) to Everyone: 10:35 AM file containing keys used to verify LTPA tokens from Adam Yoho (IBM) to Everyone: 10:36 AM
  from Joe Grassel (IBM) to Everyone: 10:40 AM So what happens should a key expire while an application in service is still using it? from Nichole Stewart (IBM) to Everyone: 10:41 AM The user is required to log back in. from Nichole Stewart (IBM) to Everyone: 10:42 AM Then the user gets an ltpa token from a valid key file. from Joe Grassel (IBM) to Everyone: 10:43 AM I mean more on the app config side, like what happens to the app at 2023-04-30 8am, especially when there are no more valid keys to use. Is there some kind of notification? It's not uncommon for people to forget to maintain their DNS registry or root certs, I'm sure this would be even more common from Eric Covener (IBM) to Everyone: 10:44 AM IIUC The first key is always used for signing/encryption and verifying/decryption. The expiration at the top level is how long the token is valid, not the key. from Ajay Reddy Karkala (IBM) to Everyone: 10:44 AM right. token expiration from Ajay Reddy Karkala (IBM) to Everyone: 10:55 AM useContextRootAsSSOCookiePath

p22: 
- Changed useContextRootAsCookiePath to useContextRootAsSSOCookiePath
- Other comments are already addressed.

[NottyCode]

@utle I'm thinking about the steps a user would have to go through to get an application deployed to multiple servers updated with new LTPA keys without taking a login outage (i.e. LTPA tokens are issued that a server cannot access).

At a high level I think the steps are fairly simple, but I'm not sure of the detail and the UFO doesn't include it. I'm worried that the steps are overly burdensome. I think they would be:

1. Call createLTPAKeys
2. Copy LTPA keys to all servers
3. Configure all servers to use the new LTPA key as a validation source
4. Move primary LTPA keys to be a validation source (as well as primary)
5. Update all servers primary LTPA key for issuing

I worry going from 4 to 5 would require significant updates to the server.xml since it would involve updating the primary and deleting an entry from the validationKeys.

Do you agree that these are the steps? If so we should document the process in the UFO, however I'm concerned that flow is quite complicated for users to script around and perhaps we need to simplify the externals here.

[utle]

@NottyCode
We have two options
Option 1: Do not enable monitoryDirectory; Just likes what you have above.

Option 2: Enable monitorDirectory; Primary and validation keys must be in the same directory.

1. Call createLTPAKeys to create a new ltpa.keys file.
2. Enable monitory directory for all servers.
3. Rename the existing ltpa.keys file to validation.keys for all servers.
4. Update all servers ltpa.keys file with the new ltpa.keys file.

Added pg22 and pg 23 in the UFO with these two options.

[utle]

Use Application context root for JWT/LTPA SSO cookie path:
#25431
#25432

[Zech-Hein]

@OpenLiberty/demo-approvers Demo scheduled for EOI 23.16

[utle]

ID:
OpenLiberty/docs#6821

[Zech-Hein]

Product Code delivered: #25826

[Zech-Hein]

FAT test code delivered: #26017

[nstewart0206]

SVT: Issue 30215

[Zech-Hein]

@OpenLiberty/serviceability-approvers - See the completed questions below. Please let me know if serviceability approval can be granted or if anything else is needed.

1. UFO -- does the UFO identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?

• Yes, the UFO has a slide for Serviceability that addresses new messages being added and for common error scenario messages. These messages have all been implemented.

2. Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. L2, test team, or another development team).

a) What problem paths were tested and demonstrated?

• validation keys fileName that doesn't exist
• validation keys password that is incorrect
• validation keys notUseAfterDate set in the past
• validation keys notUseAfterDate set with an invalid value

b) Who did you demo to?

• Demo'd to the core-security-squad test team (Malhar Shah)

  c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that L2 should be able to quickly address those problems without need to engage L3?

• Yes

3. SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered.
   a) Who conducted SVT tests for this feature?

• Nichole Stewart

  b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that L2 should be able to quickly address those problems without need to engage L3?

• Yes

4. Which L2 / L3 queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective L2/L3 teams know they are supporting it. Ask Don Bourne if you need links or more info.

• WAS L2: SEC
• WAS L3: Core Security

5. Does this feature add any new metrics or emit any new JSON events? If yes, have you updated the JMX metrics reference list / Metrics reference list / JSON log events reference list in the Open Liberty docs?

• No

[Zech-Hein]

@OpenLiberty/ste-approvers the STE slides have been uploaded to the STE Box Folder

[tngiang73]

@Zech-Hein : WASSEC L2 reviewed STE slides and they good good. Thanks.

[chirp1]

ID has received initial content for this feature at OpenLiberty/docs#6821 and written a draft. Approving.

[Zech-Hein]

UpdateTrigger being added per Alasdair's request: #26731

[Zech-Hein]

Beta guards are removed: #27009