Enable use of customDefaults from Project Config also in Groovy (#1521)

This change enables the setupCommonPipelineEnvironment step to handle custom default configurations defined in customDefaults parameter of the project configuration. Previously, only the getConfig Go step was able to incorporate custom default configurations. Update documentation on custom defaults and sharing between projects. Co-authored-by: Stephan Aßmus <[email protected]>
SAP · May 12, 2020 · d7985dd · d7985dd
1 parent b512301
commit d7985dd
Show file tree

Hide file tree

Showing 10 changed files with 315 additions and 63 deletions.
diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
@@ -9,6 +9,7 @@
 1. [Testing](#testing)
 1. [Debugging](#debugging)
 1. [Release](#release)
+1. [Pipeline Configuration](#pipeline-configuration)
 
 ## Getting started
 
@@ -428,3 +429,34 @@ We release on schedule (once a week) and on demand.
 To perform a release, the respective action must be invoked for which a convenience script is available in `contrib/perform-release.sh`.
 It requires a personal access token for GitHub with `repo` scope.
 Example usage `PIPER_RELEASE_TOKEN=THIS_IS_MY_TOKEN contrib/perform-release.sh`.
+
+## Pipeline Configuration
+
+The pipeline configuration is organized in a hierarchical manner and configuration parameters are incorporated from multiple sources.
+In general, there are four sources for configurations:
+
+1. Directly passed step parameters
+1. Project specific configuration placed in `.pipeline/config.yml`
+1. Custom default configuration provided in `customDefaults` parameter of the project config or passed as parameter to the step `setupCommonPipelineEnvironment`
+1. Default configuration from Piper library
+
+For more information and examples on how to configure a project, please refer to the [configuration documentation](https://sap.github.io/jenkins-library/configuration/).
+
+### Groovy vs. Go step configuration
+
+The configuration of a project is, as of now, resolved separately for Groovy and Go steps.
+There are, however, dependencies between the steps responsible for resolving the configuration.
+The following provides an overview of the central components and their dependencies.
+
+#### setupCommonPipelineEnvironment (Groovy)
+
+The step `setupCommonPipelineEnvironment` initializes the `commonPipelineEnvironment` and `DefaultValueCache`.
+Custom default configurations can be provided as parameters to `setupCommonPipelineEnvironment` or via the `customDefaults` parameter in project configuration.
+
+#### DefaultValueCache (Groovy)
+
+The `DefaultValueCache` caches the resolved (custom) default pipeline configuration and the list of configurations that contributed to the result.
+On initialization, it merges the provided custom default configurations with the default configuration from Piper library, as per the hierarchical order.
+
+Note, the list of configurations cached by `DefaultValueCache` is used to pass path to the (custom) default configurations of each Go step.
+It only contains the paths of configurations which are **not** provided via `customDefaults` parameter of the project configuration, since the Go layer already resolves configurations provided via `customDefaults` parameter independently.
diff --git a/documentation/docs/configuration.md b/documentation/docs/configuration.md
@@ -14,6 +14,7 @@ Configuration of the Piper steps as well the Piper templates can be done in a hi
 1. Stage configuration parameters define a Jenkins pipeline stage dependent set of parameters (e.g. deployment options for the `Acceptance` stage)
 1. Step configuration defines how steps behave in general (e.g. step `cloudFoundryDeploy`)
 1. General configuration parameters define parameters which are available across step boundaries
+1. Custom default configuration provided by the user through a reference in the `customDefaults` parameter of the project configuration
 1. Default configuration comes with the Piper library and is always available
 
 ![Piper Configuration](images/piper_config.png)
@@ -78,3 +79,32 @@ commonPipelineEnvironment.configuration.general.gitSshKeyCredentialsId
 Within library steps the `ConfigurationHelper` object is used.
 
 You can see its usage in all the Piper steps, for example [newmanExecute](https://github.com/SAP/jenkins-library/blob/master/vars/newmanExecute.groovy#L23).
+
+## Custom default configuration
+
+For projects that are composed of multiple repositories (microservices), it might be desired to provide custom default configurations.
+To do that, create a YAML file which is accessible from your CI/CD environment and configure it in your project configuration.
+For example, the custom default configuration can be stored in a GitHub repository and accessed via the "raw" URL:
+
+```yaml
+customDefaults: ['https://my.github.local/raw/someorg/custom-defaults/master/backend-service.yml']
+general:
+  ...
+```
+
+Note, the parameter `customDefaults` is required to be a list of strings and needs to be defined as a separate section of the project configuration.
+In addition, the item order in the list implies the precedence, i.e., the last item of the customDefaults list has highest precedence.
+
+It is important to ensure that the HTTP response body is proper YAML, as the pipeline will attempt to parse it.
+
+Anonymous read access to the `custom-defaults` repository is required.
+
+The custom default configuration is merged with the project's `.pipeline/config.yml`.
+Note, the project's config takes precedence, so you can override the custom default configuration in your project's local configuration.
+This might be useful to provide a default value that needs to be changed only in some projects.
+An overview of the configuration hierarchy is given at the beginning of this page.
+
+If you have different types of projects, they might require different custom default configuration.
+For example, you might not require all projects to have a certain code check (like Whitesource, etc.) active.
+This can be achieved by having multiple YAML files in the _custom-defaults_ repository.
+Configure the URL to the respective configuration file in the projects as described above.
diff --git a/documentation/docs/pipelines/cloud-sdk/shared-config-between-projects.md b/documentation/docs/pipelines/cloud-sdk/shared-config-between-projects.md
diff --git a/documentation/mkdocs.yml b/documentation/mkdocs.yml
@@ -24,7 +24,6 @@ nav:
             - 'Introduction': pipelines/cloud-sdk/introduction.md
             - 'Build Tools': pipelines/cloud-sdk/build-tools.md
             - 'Cloud Qualities': pipelines/cloud-sdk/cloud-qualities.md
-            - 'Shared Configuration': pipelines/cloud-sdk/shared-config-between-projects.md
     - 'Scenarios':
         - 'Build and Deploy Hybrid Applications with Jenkins and SAP Solution Manager': scenarios/changeManagement.md
         - 'Build and Deploy SAP UI5 or SAP Fiori Applications on SAP Cloud Platform with Jenkins': scenarios/ui5-sap-cp/Readme.md

diff --git a/src/com/sap/piper/DefaultValueCache.groovy b/src/com/sap/piper/DefaultValueCache.groovy
@@ -1,7 +1,5 @@
 package com.sap.piper
 
-import com.sap.piper.MapUtils
-
 @API
 class DefaultValueCache implements Serializable {
     private static DefaultValueCache instance
@@ -40,24 +38,66 @@ class DefaultValueCache implements Serializable {
     }
 
     static void prepare(Script steps, Map parameters = [:]) {
-        if(parameters == null) parameters = [:]
-        if(!DefaultValueCache.getInstance() || parameters.customDefaults) {
-            def defaultValues = [:]
-            def configFileList = ['default_pipeline_environment.yml']
-            def customDefaults = parameters.customDefaults
-
-            if(customDefaults in String)
-                customDefaults = [customDefaults]
-            if(customDefaults in List)
-                configFileList += customDefaults
-            for (def configFileName : configFileList){
-                if(configFileList.size() > 1) steps.echo "Loading configuration file '${configFileName}'"
-                def configuration = steps.readYaml text: steps.libraryResource(configFileName)
-                defaultValues = MapUtils.merge(
-                        MapUtils.pruneNulls(defaultValues),
-                        MapUtils.pruneNulls(configuration))
+        if (parameters == null) parameters = [:]
+        if (!getInstance() || parameters.customDefaults) {
+            List defaultsFromResources = ['default_pipeline_environment.yml']
+            List customDefaults = Utils.appendParameterToStringList(
+                [], parameters, 'customDefaults')
+            defaultsFromResources.addAll(customDefaults)
+            List defaultsFromFiles = Utils.appendParameterToStringList(
+                [], parameters, 'customDefaultsFromFiles')
+            List defaultsFromConfig = Utils.appendParameterToStringList(
+                [], parameters, 'customDefaultsFromConfig')
+
+            Map defaultValues = [:]
+            defaultValues = addDefaultsFromLibraryResources(steps, defaultValues, defaultsFromResources)
+            defaultValues = addDefaultsFromFiles(steps, defaultValues, defaultsFromFiles)
+            defaultValues = addDefaultsFromFiles(steps, defaultValues, defaultsFromConfig)
+
+            // The "customDefault" parameter is used for storing which extra defaults need to be
+            // passed to piper-go. The library resource 'default_pipeline_environment.yml' shall
+            // be excluded, since the go steps have their own in-built defaults in their yaml files.
+            // And 'customDefaultsFromConfig' shall also be excluded, since piper-go handles this
+            // config parameter itself.
+            createInstance(defaultValues, customDefaults + defaultsFromFiles)
+        }
+    }
+
+    private static Map addDefaultsFromLibraryResources(Script steps, Map defaultValues, List resourceFiles) {
+        for (String configFileName : resourceFiles) {
+            if (resourceFiles.size() > 1) {
+                steps.echo "Loading configuration file '${configFileName}'"
+            }
+            Map configuration = steps.readYaml text: steps.libraryResource(configFileName)
+            defaultValues = mergeIntoDefaults(defaultValues, configuration)
+        }
+        return defaultValues
+    }
+
+    private static Map addDefaultsFromFiles(Script steps, Map defaultValues, List configFiles) {
+        for (String configFileName : configFiles) {
+            steps.echo "Loading configuration file '${configFileName}'"
+            try {
+                Map configuration = steps.readYaml file: ".pipeline/$configFileName"
+                defaultValues = mergeIntoDefaults(defaultValues, configuration)
+            } catch (Exception e) {
+                steps.error "Failed to parse custom defaults as YAML file. " +
+                    "Please make sure it is valid YAML, and if loading from a remote location, " +
+                    "that the response body only contains valid YAML. " +
+                    "If you use a file from a GitHub repository, make sure you've used the 'raw' link, " +
+                    "for example https://my.github.local/raw/someorg/shared-config/master/backend-service.yml\n" +
+                    "File path: ${configFileName}\n" +
+                    "Content: ${steps.readFile file: configFileName}\n" +
+                    "Exeption message: ${e.getMessage()}\n" +
+                    "Exception stacktrace: ${Arrays.toString(e.getStackTrace())}"
             }
-            DefaultValueCache.createInstance(defaultValues, customDefaults)
         }
+        return defaultValues
+    }
+
+    private static Map mergeIntoDefaults(Map defaultValues, Map configuration) {
+        return MapUtils.merge(
+            MapUtils.pruneNulls(defaultValues),
+            MapUtils.pruneNulls(configuration))
     }
 }
diff --git a/src/com/sap/piper/Utils.groovy b/src/com/sap/piper/Utils.groovy
@@ -173,3 +173,15 @@ static String evaluateFromMavenPom(Script script, String pomFileName, String pom
     }
     return resolvedExpression
 }
+
+static List appendParameterToStringList(List list, Map parameters, String paramName) {
+    def value = parameters[paramName]
+    List result = []
+    result.addAll(list)
+    if (value in CharSequence) {
+        result.add(value)
+    } else if (value in List) {
+        result.addAll(value)
+    }
+    return result
+}
diff --git a/test/groovy/PrepareDefaultValuesTest.groovy b/test/groovy/PrepareDefaultValuesTest.groovy
@@ -8,9 +8,7 @@ import com.sap.piper.DefaultValueCache
 import util.BasePiperTest
 import util.JenkinsLoggingRule
 import util.JenkinsReadYamlRule
-import util.JenkinsShellCallRule
 import util.JenkinsStepRule
-
 import util.Rules
 
 public class PrepareDefaultValuesTest extends BasePiperTest {
@@ -31,7 +29,7 @@ public class PrepareDefaultValuesTest extends BasePiperTest {
     public void setup() {
 
         helper.registerAllowedMethod("libraryResource", [String], { fileName ->
-            switch(fileName) {
+            switch (fileName) {
                 case 'default_pipeline_environment.yml': return "default: 'config'"
                 case 'custom.yml': return "custom: 'myConfig'"
                 case 'not_found': throw new hudson.AbortException('No such library resource not_found could be found')

diff --git a/test/groovy/SetupCommonPipelineEnvironmentTest.groovy b/test/groovy/SetupCommonPipelineEnvironmentTest.groovy
@@ -1,38 +1,66 @@
+import com.sap.piper.DefaultValueCache
 import org.junit.Before
 import org.junit.Rule
 import org.junit.Test
+import org.junit.rules.ExpectedException
 import org.junit.rules.RuleChain
 import org.yaml.snakeyaml.Yaml
 import util.BasePiperTest
+import util.JenkinsReadFileRule
+import util.JenkinsShellCallRule
 import util.JenkinsStepRule
 import util.JenkinsWriteFileRule
 import util.Rules
 
+import static org.hamcrest.Matchers.hasItem
 import static org.junit.Assert.assertEquals
 import static org.junit.Assert.assertNotNull
+import static org.junit.Assert.assertThat
+import static org.junit.Assert.assertTrue
 
 class SetupCommonPipelineEnvironmentTest extends BasePiperTest {
 
     def usedConfigFile
 
     private JenkinsStepRule stepRule = new JenkinsStepRule(this)
     private JenkinsWriteFileRule writeFileRule = new JenkinsWriteFileRule(this)
+    private ExpectedException thrown = ExpectedException.none()
+    private JenkinsShellCallRule shellRule = new JenkinsShellCallRule(this)
+    private JenkinsReadFileRule readFileRule = new JenkinsReadFileRule(this, "./")
 
     @Rule
     public RuleChain rules = Rules
         .getCommonRules(this)
         .around(stepRule)
         .around(writeFileRule)
+        .around(thrown)
+        .around(shellRule)
+        .around(readFileRule)
+
 
     @Before
     void init() {
 
         def examplePipelineConfig = new File('test/resources/test_pipeline_config.yml').text
 
+        helper.registerAllowedMethod("libraryResource", [String], { fileName ->
+            switch(fileName) {
+                case 'default_pipeline_environment.yml': return "default: 'config'"
+                case 'custom.yml': return "custom: 'myConfig'"
+                case 'notFound.yml': throw new hudson.AbortException('No such library resource notFound could be found')
+                default: return "the:'end'"
+            }
+        })
+
         helper.registerAllowedMethod("readYaml", [Map], { Map parameters ->
             Yaml yamlParser = new Yaml()
             if (parameters.text) {
                 return yamlParser.load(parameters.text)
+            } else if(parameters.file) {
+                if(parameters.file == '.pipeline/default_pipeline_environment.yml') return [default: 'config']
+                else if (parameters.file == '.pipeline/custom.yml') return [custom: 'myConfig']
+            } else {
+                throw new IllegalArgumentException("Key 'text' and 'file' are both missing in map ${m}.")
             }
             usedConfigFile = parameters.file
             return yamlParser.load(examplePipelineConfig)
@@ -68,5 +96,77 @@ class SetupCommonPipelineEnvironmentTest extends BasePiperTest {
         assertEquals('develop', nullScript.commonPipelineEnvironment.configuration.general.productiveBranch)
         assertEquals('my-maven-docker', nullScript.commonPipelineEnvironment.configuration.steps.mavenExecute.dockerImage)
     }
+
+    @Test
+    public void testAttemptToLoadNonExistingConfigFile() {
+
+        helper.registerAllowedMethod("fileExists", [String], { String path ->
+            switch(path) {
+                case 'default_pipeline_environment.yml': return false
+                case 'custom.yml': return false
+                case 'notFound.yml': return false
+                default: return true
+            }
+        })
+
+        helper.registerAllowedMethod("handlePipelineStepErrors", [Map,Closure], { Map map, Closure closure ->
+            closure()
+        })
+
+        // Behavior documented here based on reality check
+        thrown.expect(hudson.AbortException.class)
+        thrown.expectMessage('No such library resource notFound could be found')
+
+        stepRule.step.setupCommonPipelineEnvironment(
+            script: nullScript,
+            customDefaults: 'notFound.yml'
+        )
+    }
+
+    @Test
+    void testAttemptToLoadFileFromURL() {
+        helper.registerAllowedMethod("fileExists", [String], {String path ->
+            switch (path) {
+                case 'default_pipeline_environment.yml': return false
+                default: return true
+            }
+        })
+
+        String customDefaultUrl = "https://url-to-my-config.com/my-config.yml"
+        boolean urlRequested = false
+
+        helper.registerAllowedMethod("httpRequest", [Map], {Map parameters ->
+            switch (parameters.url) {
+                case customDefaultUrl:
+                    urlRequested = true
+                    return [status: 200, content: "custom: 'myRemoteConfig'"]
+                default:
+                    throw new IllegalArgumentException('wrong URL requested')
+            }
+        })
+
+        helper.registerAllowedMethod("readYaml", [Map], { Map parameters ->
+            Yaml yamlParser = new Yaml()
+            if (parameters.text) {
+                return yamlParser.load(parameters.text)
+            } else if (parameters.file) {
+                if (parameters.file == '.pipeline/config-with-custom-defaults.yml') {
+                    return [customDefaults: "${customDefaultUrl}"]
+                }
+                if (parameters.file == '.pipeline/custom_default_from_url_0.yml') {
+                    return [custom: 'myRemoteConfig']
+                }
+            }
+            throw new IllegalArgumentException("Unexpected invocation of readYaml step")
+        })
+
+        stepRule.step.setupCommonPipelineEnvironment(
+            script: nullScript,
+            customDefaults: 'custom.yml',
+            configFile: '.pipeline/config-with-custom-defaults.yml',
+        )
+        assertEquals("custom: 'myRemoteConfig'", writeFileRule.files['.pipeline/custom_default_from_url_0.yml'])
+        assertEquals('myRemoteConfig', DefaultValueCache.instance.defaultValues['custom'])
+    }
 }