diff --git a/README.md b/README.md index bf81af7..98c6fa9 100644 --- a/README.md +++ b/README.md @@ -9,51 +9,45 @@ _This is the maven databricks plugin, which uses the databricks rest api._ [![Javadocs](http://www.javadoc.io/badge/com.edmunds/databricks-maven-plugin.svg)](http://www.javadoc.io/doc/com.edmunds/databricks-maven-plugin) +## Prerequisites -### Building, Installing and Running +For Users: +- You have a databricks account +- You are somewhat familiar with Maven and have maven installed +- You have an s3 bucket (we will call databricksRepo) that you will use to store your artifacts. +- You have AWS keys that can write to this s3 bucket +- Databricks has access to an IAM Role that can read from this bucket. -How to build the project locally: -```mvn clean install``` - -- Not required! Because you can build and develop without it, but you will likely want Lombok configured with your IDEA: -https://projectlombok.org/setup/intellij - -How to run the project locally (if applicable): - - -## Running the tests +For Contributors: +- You need to be able execute an integration test that will actually do things on your databricks account. -```mvn clean test``` - - -### End-to-End testing +## Configuring -Please have these set in your .bash_profile -```bash -export DB_USER=myuser -export DB_PASSWORD=mypassword -export DB_URL=my-test-db-instance -export DB_TOKEN=my-db-token -export DB_REPO=my-s3-bucket/my-artifact-location -export INSTANCE_PROFILE_ARN=arn:aws:iam::123456789:instance-profile/MyDatabricksRole -``` +### System Properties +For databricks specific properties we also support system properties. This can be useful for when you don't want tokens or passwords +stored in a pom or a script and instead want it to be available on a build server. +Currently the following environment properties are supported: +DB_URL -> my-databrics.cloud.databricks.com +DB_TOKEN -> dapiceexampletoken +DB_USER -> my_user +DB_PASSWORD -> my_password -```bash -mvn clean -P run-its install -``` +We can continue to support more system properties in the future if users have a compelling reason for it. -Please note, this will: -1. create a job, if it does not exist, delete it if it does -2. start the job (e.g. run it once) -3. wait for the job to finish and ensure it's success +### AWS Credentials +For the upload mojo that uploads your artifact to s3, the default aws provider chain is used. As long as you have appropriate permissions on that chain +it should just work. -## Deployment +### All other properties -Since this code is a library, you do not need to deploy it anywhere, once it passes build, you can just use it in another project. +For all other properties we support configuration in the following ways: +1. via configuration in the mojo +2. via property configuration on the command line or in the section -## Configuring +### Examples -It is recommended that you use maven profiles to allow for credentials per an environment to be defined. +If you would like to setup default profiles for users, you can take the following approach. +NOTE: if you define like below, you cannot override via CLI args unless you use project properties as well. ```xml @@ -78,6 +72,15 @@ It is recommended that you use maven profiles to allow for credentials per an en ``` +Yet another option is to provide all of your credentials when you call the plugin. +You can even rely on System Properties or the default aws provider chain +for the host/user/password OR token for databricks rest client. Please see End to End testing section or the +BaseDatabricksMojo for information on these system properties. + +```sh +mvn databricks:upload-to-s3 -Ddatabricks.repo=my-repo -Denvironment=QA +``` + ## Instructions ### Use Case 1 - Uploading an Artifact to s3, for Databricks ```bash @@ -213,6 +216,49 @@ mvn databricks:job -Djob.command=RESTART ``` + +## Building, Installing and Running + +How to build the project locally: +```mvn clean install``` + +- Not required! Because you can build and develop without it, but you will likely want Lombok configured with your IDEA: +https://projectlombok.org/setup/intellij + +How to run the project locally (if applicable): + + +### Running the tests + +```mvn clean test``` + + +### End-to-End testing + +Please have these set in your .bash_profile. + +```bash +export DB_USER=myuser +export DB_PASSWORD=mypassword +export DB_URL=my-test-db-instance +export DB_TOKEN=my-db-token +export DB_REPO=my-s3-bucket/my-artifact-location +export INSTANCE_PROFILE_ARN=arn:aws:iam::123456789:instance-profile/MyDatabricksRole +``` + +```bash +mvn clean -P run-its install +``` + +Please note, this will: +1. create a job, if it does not exist, delete it if it does +2. start the job (e.g. run it once) +3. wait for the job to finish and ensure it's success + +## Releasing + +Please see the contributing section on how to RELEASE. + ## Contributing Please read [CONTRIBUTING.md](CONTRIBUTING.md) for the process for merging code into master. \ No newline at end of file diff --git a/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksJobMojo.java b/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksJobMojo.java index 408a797..c5519e3 100644 --- a/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksJobMojo.java +++ b/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksJobMojo.java @@ -129,25 +129,20 @@ String getJobSettingsFromTemplate(JobTemplateModel jobTemplateModel) throws Mojo JobTemplateModel getJobTemplateModel() throws MojoExecutionException { try { JobTemplateModel jobTemplateModel; + // TODO this if/else should be done with polymorphism. It isn't needed in local builds if (jobTemplateModelFile.exists()) { String jobTemplateModelJson = FileUtils.readFileToString(jobTemplateModelFile); jobTemplateModel = ObjectMapperUtils.deserialize(jobTemplateModelJson, JobTemplateModel.class); } else { if (isLocalBuild) { - jobTemplateModel = new JobTemplateModel(project); + if (databricksRepo == null) { + throw new MojoExecutionException("databricksRepo must be set!"); + } + jobTemplateModel = new JobTemplateModel(project, environment, databricksRepo, databricksRepoKey); } else { throw new MojoExecutionException(String.format("[%s] file was not found in the build. Please ensure prepare-package was ran during build.", MODEL_FILE_NAME)); } } - - // [BDD-3114] - we want the current environment, to honor what was passed into the build, and not what was serialized [SAE] - jobTemplateModel.setEnvironment(environment); - if (StringUtils.isBlank(databricksRepo)) { - throw new MojoExecutionException("missing property: databricks.repo"); - } - jobTemplateModel.getProjectProperties().setProperty("databricks.repo", databricksRepo); - jobTemplateModel.getProjectProperties().setProperty("databricks.repo.key", databricksRepoKey); - return jobTemplateModel; } catch (IOException e) { throw new MojoExecutionException(e.getMessage(), e); diff --git a/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksMojo.java b/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksMojo.java index eea51a1..6be0ca0 100644 --- a/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksMojo.java +++ b/src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksMojo.java @@ -50,13 +50,17 @@ public abstract class BaseDatabricksMojo extends AbstractMojo { * for example: * "my-bucket/artifacts" * - * For some reason, I couldn't use the "." syntax for the name. + * This property is not required due to the no project option. + * + * If both project property and mojo configuration is set, mojo configuration wins. */ - @Parameter(name = "databricksRepo", property = "databricks.repo", required = true) + @Parameter(name = "databricksRepo", property = "databricks.repo") protected String databricksRepo; /** * The prefix to load to. This is appended to the databricksRepo property. + * This is an artifact specific key and will by default be the maven style qualifier: + * groupId/artifactId/version/artifact-version.jar */ @Parameter(name= "databricksRepoKey", property = "databricks.repo.key", defaultValue = "${project.groupId}/${project.artifactId}/${project.version}/${project.build.finalName}" + @@ -72,28 +76,42 @@ public abstract class BaseDatabricksMojo extends AbstractMojo { /** * The environment name. Is used in freemarker templating for conditional job settings. */ - @Parameter(property = "environment") + @Parameter(name = "environment", property = "environment") protected String environment; - @Parameter(property = "host") + /** + * This property can be picked up via an environment property! + * DB_URL + */ + @Parameter(name = "host", property = "host") protected String host; /** + * This property can be picked up via an environment property! + * DB_TOKEN * NOTE: user+password authentication will take precedence over token based authentication if both are provided. */ - @Parameter(property = "token") + @Parameter(name = "token", property = "token") protected String token; - @Parameter(property = "user") + /** + * This property can be picked up via an environment property! + * DB_USER + */ + @Parameter(name = "user", property = "user") protected String user; - @Parameter(property = "password") + /** + * This property can be picked up via an environment property! + * DB_PASSWORD + */ + @Parameter(name = "password", property = "password") protected String password; /** * Whether or not you want to validate the databricks job settings file. */ - @Parameter(defaultValue = "true", property = "validate") + @Parameter(name = "validate", defaultValue = "true", property = "validate") protected boolean validate; @@ -174,6 +192,7 @@ protected void validateRepoProperties() throws MojoExecutionException { } String createArtifactPath() throws MojoExecutionException { + //TODO if we want databricksRepo to be specified via system properties, this is where it could happen. validateRepoProperties(); String modifiedDatabricksRepo = databricksRepo; String modifiedDatabricksRepoKey = databricksRepoKey; diff --git a/src/main/java/com/edmunds/tools/databricks/maven/PrepareDbResources.java b/src/main/java/com/edmunds/tools/databricks/maven/PrepareDbResources.java index ee823a2..7d9fa41 100644 --- a/src/main/java/com/edmunds/tools/databricks/maven/PrepareDbResources.java +++ b/src/main/java/com/edmunds/tools/databricks/maven/PrepareDbResources.java @@ -46,7 +46,7 @@ public void execute() throws MojoExecutionException { void prepareJobTemplateModel() throws MojoExecutionException { try { - FileUtils.writeStringToFile(jobTemplateModelFile, ObjectMapperUtils.serialize(new JobTemplateModel(project))); + FileUtils.writeStringToFile(jobTemplateModelFile, ObjectMapperUtils.serialize(getJobTemplateModel())); } catch (IOException e) { throw new MojoExecutionException(e.getMessage(), e); } diff --git a/src/main/java/com/edmunds/tools/databricks/maven/model/JobTemplateModel.java b/src/main/java/com/edmunds/tools/databricks/maven/model/JobTemplateModel.java index 576f2be..5ff3ed5 100644 --- a/src/main/java/com/edmunds/tools/databricks/maven/model/JobTemplateModel.java +++ b/src/main/java/com/edmunds/tools/databricks/maven/model/JobTemplateModel.java @@ -36,8 +36,13 @@ public class JobTemplateModel { private String groupId; private String artifactId; private String version; + // TODO This property should not be persisted with the template model private String environment; private String groupWithoutCompany; + // The rationale for persisting these properties is because a deployed artifact will already have been deployed + // to a specific place. You cannot change that after the fact! + private String databricksRepo; + private String databricksRepoKey; /** * Don't use this - it's for jackson deserialization only! @@ -45,13 +50,20 @@ public class JobTemplateModel { public JobTemplateModel() { } - public JobTemplateModel(MavenProject project) { + public JobTemplateModel(MavenProject project, + String environment, String databricksRepo, String databricksRepoKey) { this.groupId = project.getGroupId(); this.artifactId = project.getArtifactId(); this.projectProperties = project.getProperties(); this.systemProperties = System.getProperties(); this.version = defaultString(systemProperties.getProperty(DEPLOY_VERSION), project.getVersion()); this.groupWithoutCompany = stripCompanyPackage(project.getGroupId()); + this.databricksRepo = databricksRepo; + this.databricksRepoKey = databricksRepoKey; + this.environment = environment; + //TODO NEED TO GET RID OF this once we are ready. This is for backwards compatibility + projectProperties.setProperty("databricks.repo", databricksRepo); + projectProperties.setProperty("databricks.repo.key", databricksRepoKey); } public static String stripCompanyPackage(String path) { @@ -93,4 +105,20 @@ public String getGroupWithoutCompany() { public String toString() { return ReflectionToStringBuilder.toString(this); } + + public String getDatabricksRepo() { + return databricksRepo; + } + + public void setDatabricksRepo(String databricksRepo) { + this.databricksRepo = databricksRepo; + } + + public String getDatabricksRepoKey() { + return databricksRepoKey; + } + + public void setDatabricksRepoKey(String databricksRepoKey) { + this.databricksRepoKey = databricksRepoKey; + } } diff --git a/src/test/java/com/edmunds/tools/databricks/maven/DatabricksMavenPluginTestHarness.java b/src/test/java/com/edmunds/tools/databricks/maven/DatabricksMavenPluginTestHarness.java index 00fbc5f..87375f6 100644 --- a/src/test/java/com/edmunds/tools/databricks/maven/DatabricksMavenPluginTestHarness.java +++ b/src/test/java/com/edmunds/tools/databricks/maven/DatabricksMavenPluginTestHarness.java @@ -91,4 +91,13 @@ public T getOverridesMojo(String goal) throws Exc ret.setDatabricksServiceFactory(databricksServiceFactory); return ret; } + + public T getOverridesMojo(String goal, String variation) throws Exception { + File testPom = new File(getBasedir(), + String.format("src/test/resources/unit/basic-test/%s/test-overrides-plugin-config%s" + + ".xml", goal, variation)); + T ret = (T) lookupConfiguredMojo(testPom, goal); + ret.setDatabricksServiceFactory(databricksServiceFactory); + return ret; + } } diff --git a/src/test/java/com/edmunds/tools/databricks/maven/PrepareDbResourcesTest.java b/src/test/java/com/edmunds/tools/databricks/maven/PrepareDbResourcesTest.java index 56072bc..7eacb23 100644 --- a/src/test/java/com/edmunds/tools/databricks/maven/PrepareDbResourcesTest.java +++ b/src/test/java/com/edmunds/tools/databricks/maven/PrepareDbResourcesTest.java @@ -16,53 +16,60 @@ package com.edmunds.tools.databricks.maven; -import com.google.common.collect.Lists; -import java.io.File; -import java.util.List; import org.apache.commons.io.FileUtils; +import org.apache.maven.plugin.MojoExecutionException; +import org.testng.annotations.BeforeClass; import org.testng.annotations.BeforeMethod; import org.testng.annotations.Test; - -import static com.edmunds.tools.databricks.maven.BaseDatabricksJobMojo.DEFAULT_JOB_JSON; import static org.hamcrest.CoreMatchers.containsString; import static org.hamcrest.MatcherAssert.assertThat; -import static org.testng.Assert.assertTrue; -public class PrepareDbResourcesTest extends BaseDatabricksMojoTest { +public class PrepareDbResourcesTest extends DatabricksMavenPluginTestHarness { - private PrepareDbResources underTest = new PrepareDbResources(); + private final String GOAL = "prepare-db-resources"; - private ClassLoader classLoader = PrepareDbResources.class.getClassLoader(); - private String outputBuildDir; + @BeforeClass + public void initClass() throws Exception { + super.setUp(); + } @BeforeMethod - public void init() throws Exception { - - super.init(); - - underTest.setDatabricksServiceFactory(databricksServiceFactory); - underTest.setProject(project); - underTest.setFailOnDuplicateJobName(true); - outputBuildDir = build.getOutputDirectory(); + public void beforeMethod() throws Exception { + super.beforeMethod(); } @Test - public void testGetJobSettingsDTOs() throws Exception { - File outputFile = new File(outputBuildDir + PrepareDbResources.MODEL_FILE_NAME); - underTest.setDbJobFile(new File(classLoader.getResource(DEFAULT_JOB_JSON).getFile())); - underTest.setJobTemplateModelFile(outputFile); - underTest.prepareJobTemplateModel(); - String lines = FileUtils.readFileToString(outputFile); + public void testExecuteJobTemplateFile_default() throws Exception { + PrepareDbResources underTest = getNoOverridesMojo(GOAL); + // MUST be done otherwise invocations will read from an existing file instead. + underTest.jobTemplateModelFile.delete(); + underTest.execute(); - assertThat(lines, containsString(" \"groupId\" : \"com.edmunds.test\",")); - assertThat(lines, containsString(" \"artifactId\" : \"mycoolartifact\",")); - assertThat(lines, containsString(" \"version\" : \"1.0\",")); + String key = "unit-test-group/unit-test-artifact/1.0.0-SNAPSHOT/unit-test-artifact-1.0.0-SNAPSHOT" + + ".jar"; + + String lines = FileUtils.readFileToString(underTest.jobTemplateModelFile); + assertThat(lines, containsString(" \"groupId\" : \"unit-test-group\",")); + assertThat(lines, containsString(" \"artifactId\" : \"unit-test-artifact\",")); + assertThat(lines, containsString(" \"version\" : \"1.0.0-SNAPSHOT\",")); assertThat(lines, containsString(" \"environment\" : null,")); - assertThat(lines, containsString(" \"groupWithoutCompany\" : \"test\"")); + assertThat(lines, containsString(" \"groupWithoutCompany\" : \"unit-test-group\"")); + assertThat(lines, containsString(" \"databricks.repo\" : \"my-bucket/artifacts\"")); + assertThat(lines, containsString(" \"databricksRepo\" : \"my-bucket/artifacts\"")); + assertThat(lines, containsString(" \"databricks.repo.key\" : \"" + key +"\"")); + assertThat(lines, containsString(" \"databricksRepoKey\" : \""+key+"\"")); } + @Test(expectedExceptions = MojoExecutionException.class, expectedExceptionsMessageRegExp = ".*databricksRepo.*") + public void testExecuteJobTemplateFile_missingThrowsException() throws Exception { + PrepareDbResources underTest = getMissingMandatoryMojo(GOAL); + // MUST be done otherwise invocations will read from an existing file instead. + underTest.jobTemplateModelFile.delete(); + underTest.execute(); + } +/* //Currently I can't get this test to work for the life of me on jenkins... It fails during the copy dir stage. @Test(enabled = false) public void testNotebookCopy() throws Exception { @@ -83,5 +90,5 @@ public void testNotebookCopy() throws Exception { for (String expectedFile : expectedFiles) { assertThat(expectedFile, new File(expectedFile).exists()); } - } + }*/ } \ No newline at end of file diff --git a/src/test/java/com/edmunds/tools/databricks/maven/UploadMojoTest.java b/src/test/java/com/edmunds/tools/databricks/maven/UploadMojoTest.java index eb54e4e..daa3ed3 100644 --- a/src/test/java/com/edmunds/tools/databricks/maven/UploadMojoTest.java +++ b/src/test/java/com/edmunds/tools/databricks/maven/UploadMojoTest.java @@ -40,8 +40,6 @@ public class UploadMojoTest extends DatabricksMavenPluginTestHarness { private UploadMojo underTest; - private ClassLoader classLoader = UploadMojoTest.class.getClassLoader(); - @Mock AmazonS3Client s3Client; @@ -53,11 +51,11 @@ public void initClass() throws Exception { @BeforeMethod public void beforeMethod() throws Exception { super.beforeMethod(); - underTest = getNoOverridesMojo(GOAL); } @Test - public void testDefaultExecute() throws MojoExecutionException { + public void testDefaultExecute() throws Exception { + underTest = getNoOverridesMojo(GOAL); underTest.s3Client = s3Client; underTest.execute(); } diff --git a/src/test/java/com/edmunds/tools/databricks/maven/UpsertJobMojoNoProjectTest.java b/src/test/java/com/edmunds/tools/databricks/maven/UpsertJobMojoNoProjectTest.java new file mode 100644 index 0000000..3dd634d --- /dev/null +++ b/src/test/java/com/edmunds/tools/databricks/maven/UpsertJobMojoNoProjectTest.java @@ -0,0 +1,136 @@ +/* + * Copyright 2018 Edmunds.com, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.edmunds.tools.databricks.maven; + +import com.edmunds.rest.databricks.DTO.JobDTO; +import com.edmunds.rest.databricks.DTO.JobSettingsDTO; +import com.edmunds.rest.databricks.DTO.JobsDTO; +import com.edmunds.rest.databricks.DTO.NewClusterDTO; +import com.edmunds.tools.databricks.maven.validation.ValidationUtil; +import org.apache.maven.plugin.MojoExecutionException; +import org.mockito.ArgumentCaptor; +import org.mockito.Mockito; +import org.testng.annotations.BeforeClass; +import org.testng.annotations.BeforeMethod; +import org.testng.annotations.Test; +import org.testng.collections.Maps; + +import java.io.File; +import java.util.Map; + +import static com.edmunds.tools.databricks.maven.BaseDatabricksJobMojo.DELTA_TAG; +import static com.edmunds.tools.databricks.maven.BaseDatabricksJobMojo.TEAM_TAG; +import static org.hamcrest.CoreMatchers.nullValue; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.is; +import static org.mockito.Matchers.anyBoolean; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Tests for @{@link UpsertJobMojo}. + *

+ * For these tests, the regex as part of the expected exceptions no longer works. + */ +public class UpsertJobMojoNoProjectTest extends DatabricksMavenPluginTestHarness { + + private final String GOAL = "upsert-job-np"; + + private ClassLoader classLoader = UpsertJobMojoNoProjectTest.class.getClassLoader(); + + @BeforeClass + public void initClass() throws Exception { + super.setUp(); + } + + @BeforeMethod + public void beforeMethod() throws Exception { + super.beforeMethod(); + } + + @Test + public void executeWithDefault() throws Exception{ + UpsertJobMojoNoProject underTest = getNoOverridesMojo(GOAL); + Mockito.when(jobService.getJobByName("unit-test-group/unit-test-artifact", true)).thenReturn(createJobDTO + ("unit-test-group/unit-test-artifact", 1)); + underTest.execute(); + + JobSettingsDTO[] jobSettingsDTOS = underTest.buildJobSettingsDTOsWithDefault(); + assert(jobSettingsDTOS.length == 0); + } + + @Test(expectedExceptions = MojoExecutionException.class, expectedExceptionsMessageRegExp = "" + + ".*databricksRepo must be set.*") + public void executeWithMissingProperties() throws Exception{ + UpsertJobMojoNoProject underTest = getMissingMandatoryMojo(GOAL); + underTest.execute(); + } + + @Test(expectedExceptions = MojoExecutionException.class, expectedExceptionsMessageRegExp = ".*databricksRepo.*") + public void whenJobFileExistsButNoTemplate_fails() throws Exception { + UpsertJobMojoNoProject underTest = getOverridesMojo(GOAL); + + underTest.execute(); + } + + /** + * This is testing the case where a job file is going to be upserted outside of the context of a project. + * A use case for this is when a company does not want to use CD and instead wants to be able to deploy specific + * artifacts deterministically after the fact. This requires that the artifact contains everything that is needed + * to do that. In this case, given a serialized json with project information, a job template and an environment can + * we upsert the job? + * + * @throws Exception + */ + @Test + public void whenJobFileAndTemplateExists_succeeds() throws Exception { + UpsertJobMojoNoProject underTest = getOverridesMojo(GOAL, "2"); + Mockito.when(jobService.getJobByName("dwh/inventory-databricks", true)).thenReturn(createJobDTO + ("dwh/inventory-databricks", 1)); + + underTest.execute(); + + JobSettingsDTO[] jobSettingsDTOS = underTest.buildJobSettingsDTOsWithDefault(); + assertThat(jobSettingsDTOS.length, is(1)); + assertThat(jobSettingsDTOS[0].getName(), is("dwh/inventory-databricks")); + assertThat(jobSettingsDTOS[0].getLibraries()[0].getJar(), is + ("s3://edmunds-repos/artifacts/com.edmunds" + + ".dwh/inventory-databricks/1.1.182-SNAPSHOT/inventory-databricks-1.1.182-SNAPSHOT" + + ".jar")); + + assert(jobSettingsDTOS.length == 1); + ArgumentCaptor jobCaptor = ArgumentCaptor.forClass(JobSettingsDTO.class); + verify(jobService, Mockito.times(1)).upsertJob(jobCaptor.capture(), anyBoolean()); + assertEquals(jobSettingsDTOS[0], jobCaptor.getValue()); + } + + private JobsDTO createJobsDTO(JobDTO... jobDTOs) { + JobsDTO jobsDTO = new JobsDTO(); + jobsDTO.setJobs(jobDTOs); + return jobsDTO; + } + + private JobDTO createJobDTO(String jobName, long jobId) { + JobDTO jobDTO = new JobDTO(); + JobSettingsDTO jobSettingsDTO = new JobSettingsDTO(); + jobSettingsDTO.setName(jobName); + jobDTO.setSettings(jobSettingsDTO); + jobDTO.setJobId(jobId); + return jobDTO; + } +} \ No newline at end of file diff --git a/src/test/java/com/edmunds/tools/databricks/maven/UpsertJobMojoTest.java b/src/test/java/com/edmunds/tools/databricks/maven/UpsertJobMojoTest.java index 1d6a27d..de8e12a 100644 --- a/src/test/java/com/edmunds/tools/databricks/maven/UpsertJobMojoTest.java +++ b/src/test/java/com/edmunds/tools/databricks/maven/UpsertJobMojoTest.java @@ -84,6 +84,38 @@ public void test_executeWithDefault() throws Exception{ assertEquals(jobSettingsDTOS[0], jobCaptor.getValue()); } + @Test + public void test_executeWithProjectProperties() throws Exception { + underTest = getOverridesMojo(GOAL, "_viaProperties"); + Mockito.when(jobService.getJobByName("unit-test-group/unit-test-artifact", true)).thenReturn(createJobDTO + ("unit-test-group/unit-test-artifact", 1)); + JobSettingsDTO[] jobSettingsDTOs = underTest.buildJobSettingsDTOsWithDefault(); + underTest.execute(); + assertThat(jobSettingsDTOs.length, is(1)); + assertThat(jobSettingsDTOs[0].getLibraries()[0].getJar(), is + ("s3://projectProperty/unit-test-group/unit-test-artifact/1.0.0-SNAPSHOT/unit-test-artifact" + + "-1.0.0-SNAPSHOT.jar")); + ArgumentCaptor jobCaptor = ArgumentCaptor.forClass(JobSettingsDTO.class); + verify(jobService, Mockito.times(1)).upsertJob(jobCaptor.capture(), anyBoolean()); + assertEquals(jobSettingsDTOs[0], jobCaptor.getValue()); + } + + @Test + public void test_executeWithProjectPropertiesAndConfig() throws Exception { + underTest = getOverridesMojo(GOAL, "_viaBothSettings"); + Mockito.when(jobService.getJobByName("unit-test-group/unit-test-artifact", true)).thenReturn(createJobDTO + ("unit-test-group/unit-test-artifact", 1)); + JobSettingsDTO[] jobSettingsDTOs = underTest.buildJobSettingsDTOsWithDefault(); + underTest.execute(); + assertThat(jobSettingsDTOs.length, is(1)); + assertThat(jobSettingsDTOs[0].getLibraries()[0].getJar(), is + ("s3://configProperty/unit-test-group/unit-test-artifact/1.0.0-SNAPSHOT/unit-test-artifact" + + "-1.0.0-SNAPSHOT.jar")); + ArgumentCaptor jobCaptor = ArgumentCaptor.forClass(JobSettingsDTO.class); + verify(jobService, Mockito.times(1)).upsertJob(jobCaptor.capture(), anyBoolean()); + assertEquals(jobSettingsDTOs[0], jobCaptor.getValue()); + } + @Test public void test_executeWithMissingProperties() throws Exception{ underTest = getMissingMandatoryMojo(GOAL); diff --git a/src/test/resources/unit/basic-test/prepare-db-resources/test-missing-mandatory-plugin-config.xml b/src/test/resources/unit/basic-test/prepare-db-resources/test-missing-mandatory-plugin-config.xml new file mode 100644 index 0000000..8ba32e4 --- /dev/null +++ b/src/test/resources/unit/basic-test/prepare-db-resources/test-missing-mandatory-plugin-config.xml @@ -0,0 +1,44 @@ + + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + + + + + prepare-db-resources + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/prepare-db-resources/test-no-overrides-plugin-config.xml b/src/test/resources/unit/basic-test/prepare-db-resources/test-no-overrides-plugin-config.xml new file mode 100644 index 0000000..0d3d493 --- /dev/null +++ b/src/test/resources/unit/basic-test/prepare-db-resources/test-no-overrides-plugin-config.xml @@ -0,0 +1,45 @@ + + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + my-bucket/artifacts + + + + + prepare-db-resources + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/prepare-db-resources/test-overrides-plugin-config.xml b/src/test/resources/unit/basic-test/prepare-db-resources/test-overrides-plugin-config.xml new file mode 100644 index 0000000..9375171 --- /dev/null +++ b/src/test/resources/unit/basic-test/prepare-db-resources/test-overrides-plugin-config.xml @@ -0,0 +1,46 @@ + + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + my-bucket/artifacts + my-destination + + + + + prepare-db-resources + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job-np/databricks-job-settings.json b/src/test/resources/unit/basic-test/upsert-job-np/databricks-job-settings.json new file mode 100644 index 0000000..446aba1 --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job-np/databricks-job-settings.json @@ -0,0 +1,18 @@ +[ + { + "email_notifications": { + "on_failure": ["myfakeemail.com"] + }, + "spark_jar_task": { + "main_class_name": "", + "parameters": [] + }, + "timeout_seconds": 1800, + // If streaming job, timeout_seconds should override to 0 + "retry_on_timeout": false, + "max_retries": 0, + //0 : never retry, -1: indefinitely + "min_retry_interval_millis": 120000, + "max_concurrent_runs": 1 + } +] \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job-np/mySerializedProperties.json b/src/test/resources/unit/basic-test/upsert-job-np/mySerializedProperties.json new file mode 100644 index 0000000..a006845 --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job-np/mySerializedProperties.json @@ -0,0 +1,16 @@ +{ + "projectProperties" : { + "databricks.repo" : "edmunds-repos/artifacts", + "databricks.repo.key" : "com.edmunds.dwh/inventory-databricks/1.1.182-SNAPSHOT/inventory-databricks-1.1.182-SNAPSHOT.jar" + }, + "systemProperties" : { + + }, + "groupId" : "com.edmunds.dwh", + "artifactId" : "inventory-databricks", + "version" : "1.1.182-SNAPSHOT", + "environment" : "QA", + "groupWithoutCompany" : "dwh", + "databricksRepo" : "edmunds-repos/artifacts", + "databricksRepoKey" : "com.edmunds.dwh/inventory-databricks/1.1.182-SNAPSHOT/inventory-databricks-1.1.182-SNAPSHOT.jar" +} \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job-np/test-missing-mandatory-plugin-config.xml b/src/test/resources/unit/basic-test/upsert-job-np/test-missing-mandatory-plugin-config.xml new file mode 100644 index 0000000..d769380 --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job-np/test-missing-mandatory-plugin-config.xml @@ -0,0 +1,45 @@ + + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + + + + + + upsert-job-np + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job-np/test-no-overrides-plugin-config.xml b/src/test/resources/unit/basic-test/upsert-job-np/test-no-overrides-plugin-config.xml new file mode 100644 index 0000000..ef3a535 --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job-np/test-no-overrides-plugin-config.xml @@ -0,0 +1,47 @@ + + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + QA + my-bucket + + + + + upsert-job-np + + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job-np/test-overrides-plugin-config.xml b/src/test/resources/unit/basic-test/upsert-job-np/test-overrides-plugin-config.xml new file mode 100644 index 0000000..cc2b36a --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job-np/test-overrides-plugin-config.xml @@ -0,0 +1,48 @@ + + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + QA + databricks-job-settings.json + false + + + + + upsert-job-np + + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job-np/test-overrides-plugin-config2.xml b/src/test/resources/unit/basic-test/upsert-job-np/test-overrides-plugin-config2.xml new file mode 100644 index 0000000..1f0cf13 --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job-np/test-overrides-plugin-config2.xml @@ -0,0 +1,49 @@ + + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + QA + databricks-job-settings.json + mySerializedProperties.json + false + + + + + upsert-job-np + + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job/test-overrides-plugin-config_viaBothSettings.xml b/src/test/resources/unit/basic-test/upsert-job/test-overrides-plugin-config_viaBothSettings.xml new file mode 100644 index 0000000..4021dba --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job/test-overrides-plugin-config_viaBothSettings.xml @@ -0,0 +1,51 @@ + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + projectProperty + + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + QA + false + configProperty + + + + + upsert-job-np + + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/unit/basic-test/upsert-job/test-overrides-plugin-config_viaProperties.xml b/src/test/resources/unit/basic-test/upsert-job/test-overrides-plugin-config_viaProperties.xml new file mode 100644 index 0000000..2be0d69 --- /dev/null +++ b/src/test/resources/unit/basic-test/upsert-job/test-overrides-plugin-config_viaProperties.xml @@ -0,0 +1,50 @@ + + + + 4.0.0 + + unit-test-group + unit-test-artifact + 1.0.0-SNAPSHOT + + + projectProperty + + + + + + com.edmunds + databricks-maven-plugin + ${project.version} + + QA + false + + + + + upsert-job-np + + + + + + + + + \ No newline at end of file