Merge pull request #25 from edmunds/bug-fix-no-project

Bug fix no project

README.md

@@ -9,51 +9,45 @@ _This is the maven databricks plugin, which uses the databricks rest api._
 
 [![Javadocs](http://www.javadoc.io/badge/com.edmunds/databricks-maven-plugin.svg)](http://www.javadoc.io/doc/com.edmunds/databricks-maven-plugin)
 
+## Prerequisites
 
-### Building, Installing and Running
+For Users:
+- You have a databricks account
+- You are somewhat familiar with Maven and have maven installed
+- You have an s3 bucket (we will call databricksRepo) that you will use to store your artifacts. 
+- You have AWS keys that can write to this s3 bucket
+- Databricks has access to an IAM Role that can read from this bucket.
 
-How to build the project locally:
-```mvn clean install```
-
-- Not required! Because you can build and develop without it, but you will likely want Lombok configured with your IDEA:
-https://projectlombok.org/setup/intellij
-
-How to run the project locally (if applicable):
-
-
-## Running the tests
+For Contributors:
+- You need to be able execute an integration test that will actually do things on your databricks account.
 
-```mvn clean test```
-
-
-### End-to-End testing
+## Configuring
 
-Please have these set in your .bash_profile
-```bash
-export DB_USER=myuser
-export DB_PASSWORD=mypassword
-export DB_URL=my-test-db-instance
-export DB_TOKEN=my-db-token
-export DB_REPO=my-s3-bucket/my-artifact-location
-export INSTANCE_PROFILE_ARN=arn:aws:iam::123456789:instance-profile/MyDatabricksRole
-```
+### System Properties
+For databricks specific properties we also support system properties. This can be useful for when you don't want tokens or passwords
+stored in a pom or a script and instead want it to be available on a build server.
+Currently the following environment properties are supported:
+DB_URL -> my-databrics.cloud.databricks.com
+DB_TOKEN -> dapiceexampletoken
+DB_USER -> my_user
+DB_PASSWORD -> my_password
 
-```bash
-mvn clean -P run-its install
-```
+We can continue to support more system properties in the future if users have a compelling reason for it.
 
-Please note, this will:
-1. create a job, if it does not exist, delete it if it does
-2. start the job (e.g. run it once)
-3. wait for the job to finish and ensure it's success
+### AWS Credentials
+For the upload mojo that uploads your artifact to s3, the default aws provider chain is used. As long as you have appropriate permissions on that chain
+it should just work.
 
-## Deployment
+### All other properties
 
-Since this code is a library, you do not need to deploy it anywhere, once it passes build, you can just use it in another project.
+For all other properties we support configuration in the following ways:
+1. via configuration in the mojo
+2. via property configuration on the command line or in the <properties> section
 
-## Configuring
+### Examples
 
-It is recommended that you use maven profiles to allow for credentials per an environment to be defined.
+If you would like to setup default profiles for users, you can take the following approach.
+NOTE: if you define like below, you cannot override via CLI args unless you use project properties as well.
 ```xml
          <!-- Databricks QA Credentials -->
          <profile>
@@ -78,6 +72,15 @@ It is recommended that you use maven profiles to allow for credentials per an en
          </profile>
 ```
 
+Yet another option is to provide all of your credentials when you call the plugin.
+You can even rely on System Properties or the default aws provider chain 
+for the host/user/password OR token for databricks rest client. Please see End to End testing section or the
+BaseDatabricksMojo for information on these system properties. 
+
+```sh
+mvn databricks:upload-to-s3 -Ddatabricks.repo=my-repo -Denvironment=QA
+```
+
 ## Instructions
 ### Use Case 1 - Uploading an Artifact to s3, for Databricks
 ```bash
@@ -213,6 +216,49 @@ mvn databricks:job -Djob.command=RESTART
 ```
 
 
+
+## Building, Installing and Running
+
+How to build the project locally:
+```mvn clean install```
+
+- Not required! Because you can build and develop without it, but you will likely want Lombok configured with your IDEA:
+https://projectlombok.org/setup/intellij
+
+How to run the project locally (if applicable):
+
+
+### Running the tests
+
+```mvn clean test```
+
+
+### End-to-End testing
+
+Please have these set in your .bash_profile.
+
+```bash
+export DB_USER=myuser
+export DB_PASSWORD=mypassword
+export DB_URL=my-test-db-instance
+export DB_TOKEN=my-db-token
+export DB_REPO=my-s3-bucket/my-artifact-location
+export INSTANCE_PROFILE_ARN=arn:aws:iam::123456789:instance-profile/MyDatabricksRole
+```
+
+```bash
+mvn clean -P run-its install
+```
+
+Please note, this will:
+1. create a job, if it does not exist, delete it if it does
+2. start the job (e.g. run it once)
+3. wait for the job to finish and ensure it's success
+
+## Releasing
+
+Please see the contributing section on how to RELEASE.
+
 ## Contributing
 
 Please read [CONTRIBUTING.md](CONTRIBUTING.md) for the process for merging code into master.

src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksJobMojo.java

@@ -129,25 +129,20 @@ String getJobSettingsFromTemplate(JobTemplateModel jobTemplateModel) throws Mojo
     JobTemplateModel getJobTemplateModel() throws MojoExecutionException {
         try {
             JobTemplateModel jobTemplateModel;
+            // TODO this if/else should be done with polymorphism. It isn't needed in local builds
             if (jobTemplateModelFile.exists()) {
                 String jobTemplateModelJson = FileUtils.readFileToString(jobTemplateModelFile);
                 jobTemplateModel = ObjectMapperUtils.deserialize(jobTemplateModelJson, JobTemplateModel.class);
             } else {
                 if (isLocalBuild) {
-                    jobTemplateModel = new JobTemplateModel(project);
+                    if (databricksRepo == null) {
+                        throw new MojoExecutionException("databricksRepo must be set!");
+                    }
+                    jobTemplateModel = new JobTemplateModel(project, environment, databricksRepo, databricksRepoKey);
                 } else {
                     throw new MojoExecutionException(String.format("[%s] file was not found in the build. Please ensure prepare-package was ran during build.", MODEL_FILE_NAME));
                 }
             }
-
-            // [BDD-3114] - we want the current environment, to honor what was passed into the build, and not what was serialized [SAE]
-            jobTemplateModel.setEnvironment(environment);
-            if (StringUtils.isBlank(databricksRepo)) {
-                throw new MojoExecutionException("missing property: databricks.repo");
-            }
-            jobTemplateModel.getProjectProperties().setProperty("databricks.repo", databricksRepo);
-            jobTemplateModel.getProjectProperties().setProperty("databricks.repo.key", databricksRepoKey);
-
             return jobTemplateModel;
         } catch (IOException e) {
             throw new MojoExecutionException(e.getMessage(), e);

src/main/java/com/edmunds/tools/databricks/maven/BaseDatabricksMojo.java

@@ -50,13 +50,17 @@ public abstract class BaseDatabricksMojo extends AbstractMojo {
      * for example:
      * "my-bucket/artifacts"
      *
-     * For some reason, I couldn't use the "." syntax for the name.
+     * This property is not required due to the no project option.
+     *
+     * If both project property and mojo configuration is set, mojo configuration wins.
      */
-    @Parameter(name = "databricksRepo", property = "databricks.repo", required = true)
+    @Parameter(name = "databricksRepo", property = "databricks.repo")
     protected String databricksRepo;
 
     /**
      * The prefix to load to. This is appended to the databricksRepo property.
+     * This is an artifact specific key and will by default be the maven style qualifier:
+     * groupId/artifactId/version/artifact-version.jar
      */
     @Parameter(name= "databricksRepoKey", property = "databricks.repo.key",
         defaultValue = "${project.groupId}/${project.artifactId}/${project.version}/${project.build.finalName}" +
@@ -72,28 +76,42 @@ public abstract class BaseDatabricksMojo extends AbstractMojo {
     /**
      * The environment name. Is used in freemarker templating for conditional job settings.
      */
-    @Parameter(property = "environment")
+    @Parameter(name = "environment", property = "environment")
     protected String environment;
 
-    @Parameter(property = "host")
+    /**
+     * This property can be picked up via an environment property!
+     * DB_URL
+     */
+    @Parameter(name = "host", property = "host")
     protected String host;
 
     /**
+     * This property can be picked up via an environment property!
+     * DB_TOKEN
      * NOTE: user+password authentication will take precedence over token based authentication if both are provided.
      */
-    @Parameter(property = "token")
+    @Parameter(name = "token", property = "token")
     protected String token;
 
-    @Parameter(property = "user")
+    /**
+     * This property can be picked up via an environment property!
+     * DB_USER
+     */
+    @Parameter(name = "user", property = "user")
     protected String user;
 
-    @Parameter(property = "password")
+    /**
+     * This property can be picked up via an environment property!
+     * DB_PASSWORD
+     */
+    @Parameter(name = "password", property = "password")
     protected String password;
 
     /**
      * Whether or not you want to validate the databricks job settings file.
      */
-    @Parameter(defaultValue = "true", property = "validate")
+    @Parameter(name = "validate", defaultValue = "true", property = "validate")
     protected boolean validate;
 
 
@@ -174,6 +192,7 @@ protected void validateRepoProperties() throws MojoExecutionException {
     }
 
     String createArtifactPath() throws MojoExecutionException {
+        //TODO if we want databricksRepo to be specified via system properties, this is where it could happen.
         validateRepoProperties();
         String modifiedDatabricksRepo = databricksRepo;
         String modifiedDatabricksRepoKey = databricksRepoKey;

src/main/java/com/edmunds/tools/databricks/maven/PrepareDbResources.java

@@ -46,7 +46,7 @@ public void execute() throws MojoExecutionException {
 
     void prepareJobTemplateModel() throws MojoExecutionException {
         try {
-            FileUtils.writeStringToFile(jobTemplateModelFile, ObjectMapperUtils.serialize(new JobTemplateModel(project)));
+            FileUtils.writeStringToFile(jobTemplateModelFile, ObjectMapperUtils.serialize(getJobTemplateModel()));
         } catch (IOException e) {
             throw new MojoExecutionException(e.getMessage(), e);
         }

src/main/java/com/edmunds/tools/databricks/maven/model/JobTemplateModel.java

@@ -36,22 +36,34 @@ public class JobTemplateModel {
     private String groupId;
     private String artifactId;
     private String version;
+    // TODO This property should not be persisted with the template model
     private String environment;
     private String groupWithoutCompany;
+    // The rationale for persisting these properties is because a deployed artifact will already have been deployed
+    // to a specific place. You cannot change that after the fact!
+    private String databricksRepo;
+    private String databricksRepoKey;
 
     /**
      * Don't use this - it's for jackson deserialization only!
      */
     public JobTemplateModel() {
     }
 
-    public JobTemplateModel(MavenProject project) {
+    public JobTemplateModel(MavenProject project,
+                            String environment, String databricksRepo, String databricksRepoKey) {
         this.groupId = project.getGroupId();
         this.artifactId = project.getArtifactId();
         this.projectProperties = project.getProperties();
         this.systemProperties = System.getProperties();
         this.version = defaultString(systemProperties.getProperty(DEPLOY_VERSION), project.getVersion());
         this.groupWithoutCompany = stripCompanyPackage(project.getGroupId());
+        this.databricksRepo = databricksRepo;
+        this.databricksRepoKey = databricksRepoKey;
+        this.environment = environment;
+        //TODO NEED TO GET RID OF this once we are ready. This is for backwards compatibility
+        projectProperties.setProperty("databricks.repo", databricksRepo);
+        projectProperties.setProperty("databricks.repo.key", databricksRepoKey);
     }
 
     public static String stripCompanyPackage(String path) {
@@ -93,4 +105,20 @@ public String getGroupWithoutCompany() {
     public String toString() {
         return ReflectionToStringBuilder.toString(this);
     }
+
+    public String getDatabricksRepo() {
+        return databricksRepo;
+    }
+
+    public void setDatabricksRepo(String databricksRepo) {
+        this.databricksRepo = databricksRepo;
+    }
+
+    public String getDatabricksRepoKey() {
+        return databricksRepoKey;
+    }
+
+    public void setDatabricksRepoKey(String databricksRepoKey) {
+        this.databricksRepoKey = databricksRepoKey;
+    }
 }

src/test/java/com/edmunds/tools/databricks/maven/DatabricksMavenPluginTestHarness.java

@@ -91,4 +91,13 @@ public <T extends BaseDatabricksMojo> T getOverridesMojo(String goal) throws Exc
         ret.setDatabricksServiceFactory(databricksServiceFactory);
         return ret;
     }
+
+    public <T extends BaseDatabricksMojo> T getOverridesMojo(String goal, String variation) throws Exception {
+        File testPom = new File(getBasedir(),
+            String.format("src/test/resources/unit/basic-test/%s/test-overrides-plugin-config%s" +
+                ".xml", goal, variation));
+        T ret = (T) lookupConfiguredMojo(testPom, goal);
+        ret.setDatabricksServiceFactory(databricksServiceFactory);
+        return ret;
+    }
 }