Changes required for adoption of the cmsweb-preprod cluster

[amaltaro]

Impact of the new feature
WMCore in general

Is your feature request related to a problem? Please describe.
We have recently decided to have a 3rd cluster for integration tests, called cmsweb-preprod. A short summary is:
0) cmsweb-testX clusters: a developer's playground environment to test, patch, change, and do whatever we need when developing for central services.

1. cmsweb-testbed is a level up from the testX clusters. Meant for initial integration tests with more stable code
2. cmsweb-preprod is yet another level up from testbed cluster. Meant for pre-production code, so code that we are actually preparing to push to production. It is meant to be stable!
3. cmsweb-prod/cmsweb is the production environment.

Having said that, we are moving towards the adoption of cmsweb-preprod in CMS (and WM). So this ticket is meant to track all the relevant changes that we need to either a) revise or b) update the codebase for.

Note that some cmsweb-preprod details are still under discussion, e.g. dbs url to be used and separation of CouchDB (and MongoDB) backends.

Describe the solution you'd like

1. https://github.com/dmwm/WMCore/blob/2748054/src/python/WMCore/WorkQueue/DataLocationMapper.py and

   WMCore/src/python/WMCore/WorkQueue/Policy/Start/StartPolicyInterface.py

   Line 285 in 2748054

   pileUpinstance = '-testbed' if 'cmsweb-testbed' in dbsUrl else '-prod'

   using a code like:

                pileupInstance = '-testbed' if 'cmsweb-testbed' in dbsUrl else '-prod'

2. Create new MongoDB cluster (or database under cmsweb-testbed)

3. Create new CouchDB cluster

4. Considerations for the data stored in the database? I would suggest moving cmsweb-testbed data over to cmsweb-preprod and start with a fresh database in cmsweb-testbed

5. testbed agents will have to move to cmsweb-preprod

In terms of WMCore codebase, I don't think there is anything else to consider. We might want to revisit the workflow json templates though.

Describe alternatives you've considered
None

Additional context
None

[mapellidario]

After a private discussion with Alan, we agreed that we should start using the new cmsweb-preprod cluster for our final validations and integrations before a deployment to prod. The main blocker for usins cmsweb-preprod consistently at the moment is that cmsweb-preprod shares the same couchdb and mongo databases with cmsweb-testbed

we have two options:

1. decomission cmsweb-testbed, only use cmsweb-preprod.
   • pro: the need for two separate integration clusters is not clear yet. maybe only use cmsweb-preprod is enough for us
   • cons: many teams rely on cmsweb-testbed, for example cms@home. decomissioning this cluster will require careful communications with other teams.
2. completely decouple cmsweb-testbed and cmsweb-preprod
   • ask aroosha to deploy a new couchdb instance in cmsweb-preprod
   • ask aroosha to deploy a new mongodb cluster for central services running in cmsweb-preprod