From c01d78dc511317f190226efa903e422d2a46ab4e Mon Sep 17 00:00:00 2001 From: Anastasios Zouzias Date: Mon, 30 Dec 2019 21:12:42 +0100 Subject: [PATCH] Prepare for release 0.3.8 (#215) * Remove unused code (#141) * README: update to 0.3.3 * README: fix javadoc badge * remove unused param * [sbt] version updates * [sbt] disable build for scala 2.12 * [conf] allow not_analyzed string fields (#145) * [not-analyzed-fields] do not analyzed fields ending with _notanalyzed * [sbt] version updates * [sbt] disable build for scala 2.12 * [conf] allow not_analyzed string fields (#145) * [not-analyzed-fields] do not analyzed fields ending with _notanalyzed * [sbt] version updates * [sbt] disable build for scala 2.12 * [conf] allow not_analyzed string fields (#145) * [not-analyzed-fields] do not analyzed fields ending with _notanalyzed * Revert "Revert "Setting version to 0.3.5-SNAPSHOT"" This reverts commit a6da0af34c5e8b80609b806df9cf04c0e6c8452e. * [build] update Lucene to 7.7.0 * Hotfix: issue 150 (#151) * Remove unused code (#141) * Revert "Setting version to 0.3.4-SNAPSHOT" This reverts commit 2f1d7beed90de725cfac55d7f29b31f9e13d17d7. * README: update to 0.3.3 * README: fix javadoc badge * remove unused param * [sbt] version updates * [conf] allow not_analyzed string fields (#145) * [not-analyzed-fields] do not analyzed fields ending with _notanalyzed * [hotfix] fixes issue 150 * [tests] issue 150 * fix typo * [blockEntityLinkage] drop queryPartColumns * [sbt] version updates * [scripts] fix shell * Block linkage: allow a block linker with Row to Query (#154) * [linkage] block linker with => Query * [linkage] block linker is Row => Query * remove Query analyzer on methods * [sbt] set version to 0.3.6-SNAPSHOT * Feature: allow custom analyzers during compile time (#160) * [analyzers] custom analyzer * test return null * [travis] travis_wait 1 min * Revert "[travis] travis_wait 1 min" This reverts commit c79456ef4ced306b44c2067433afcebf4b794473. * use lucene examples * custom analyzer return null * fix java reflection * add docs * Update to Lucene 8 (#161) * [lucene] upgrade to version 8.0.0 * [lucene] remove ngram analyzer * delete ngram analyzer * minor fix * add scaladoc * LuceneRDDResponseSpec.collect() should work when no results are found - Issue #166 (#168) * [sbt] update scalatest 3.0.7 * [sbt] update spark 2.4.1 * [build.sbt] add credentials file * [plugins] update versions * [sbt] update to 0.13.18 * Allow Lucene Analyzers per field (#164) * [issue_163] per field analysis * [sbt] update scalatest to 3.0.7 * [issue_163] fix docs; order of arguments * fixes on ShapeLuceneRDD * [issue_163] fix test * issue_163: minor fix * introduce LuceneRDDParams case class * fix apply in LuceneRDDParams * [issue_163] remove duplicate apply defn * add extra LuceneRDD.apply * [issue_165] throw runtime exception; use traversable trait (#170) [issue_165] throw runtime exception; handle multi-valued fields in DataFrames * [config] refactor; add environment variables in config (#173) * [refactor] configuration loading * [travis] code hygiene * Make LuceneRDDResponse to extend RDD[Row] (#175) * WIP * fix tests * remove SparkDoc class * make test compile * use GenericRowWithSchema * tests: getDouble score * score is a float * fix casting issue with Seq[String] * tests: LuceneDocToSparkRowpec * tests: LuceneDocToSparkRowpec * more tests * LuceneDocToSparkRowpec: more tests * LuceneDocToSparkRowpec: fix tests * LuceneDocToSparkRow: fix Number type inference * LuceneDocToSparkRowpec: fix tests * implicits: remove StoredField for Numeric types * implicits: revert remove StoredField for Numeric types * fix more tests * fix more tests * [tests] fix LuceneRDDResponse .toDF() * fix multivalued fields * fix score type issue * minor * stored fields for numerics * hotfix: TextField must be stored using StoredField * hotfix: stringToDocument implicit * link issue 179 * fix tests * remove _.toRow() calls * fix compile issue * [sbt] update to spark 2.4.2 * [travis] use spark 2.4.2 * [build] minor updates * Remove sbt-spark-package plugin (#181) * [sbt] remove sbt-spark-package * WIP * [sbt] add spark-mllib * [sbt] make spark provided * update to sbt to 1.X.X (#182) * [wip] update to sbt 1.X.X * [travis] fix script * [sbt] update to 1.2.8 * [sbt] update all plugins * [sbt] spark update v2.4.3 (#183) * [sbt] spark update v2.4.3 * minor update joda-time * [sbt] update spark-testing * [sbt] lucene 8.1.0 update (#184) * [sbt] lucene update 8.1.1 (#185) * [scalatest] update to 3.0.8 * [sbt] joda-time patch update * [release-info] add sonatype credentials * [sbt] lucene 8.2.0 update (#187) * [sbt] update plugins * [sbt] update spark 2.4.4 (#188) * [sbt] update joda to 2.10.4 * [sbt] update joda / typesafe config (#189) * [sbt] update Lucene 8.3.0 (#191) * [sbt] version updates (#194) * Update Lucene to version 8.3.1 * Update Twitter algebird to version 0.13.6 * Update scalatest/scalactic to version 3.1.0 * [github-actions] add scala.yml (#193) * [github-actions] add scala.yml * [sbt] update to version 1.3.3 (#195) * [plugins] update sbt plugins (#196) * [lucene] update version 8.4.0 (#197) * fix version to SNAPSHOT * code hygiene --- .github/workflows/scala.yml | 17 +++++++++++++++++ .travis.yml | 11 +++++------ build.sbt | 35 ++++++++++++----------------------- deployToSonartype.md | 25 +++++++++++++++++++++++++ deployToSonartype.sh | 13 ------------- project/build.properties | 2 +- project/plugins.sbt | 16 +++++++--------- spark-shell.sh | 2 +- 8 files changed, 68 insertions(+), 53 deletions(-) create mode 100644 .github/workflows/scala.yml create mode 100644 deployToSonartype.md delete mode 100644 deployToSonartype.sh diff --git a/.github/workflows/scala.yml b/.github/workflows/scala.yml new file mode 100644 index 00000000..fbeee571 --- /dev/null +++ b/.github/workflows/scala.yml @@ -0,0 +1,17 @@ +name: Scala CI + +on: [push] + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v1 + - name: Set up JDK 1.8 + uses: actions/setup-java@v1 + with: + java-version: 1.8 + - name: Run tests + run: sbt test diff --git a/.travis.yml b/.travis.yml index 2a043c46..25972266 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,16 +17,15 @@ before_cache: matrix: include: - jdk: oraclejdk8 - env: TEST_SPARK_VERSION="2.4.2" LUCENERDD_ANALYZER_NAME="en" LUCENERDD_LINKER_METHOD="cartesian" + env: LUCENERDD_ANALYZER_NAME="en" LUCENERDD_LINKER_METHOD="cartesian" - jdk: openjdk8 - env: TEST_SPARK_VERSION="2.4.2" LUCENERDD_ANALYZER_NAME="en" LUCENERDD_LINKER_METHOD="collectbroadcast" + env: LUCENERDD_ANALYZER_NAME="en" LUCENERDD_LINKER_METHOD="collectbroadcast" - jdk: openjdk8 - env: TEST_SPARK_VERSION="2.4.2" LUCENERDD_ANALYZER_NAME="whitespace" LUCENERDD_LINKER_METHOD="cartesian" + env: LUCENERDD_ANALYZER_NAME="whitespace" LUCENERDD_LINKER_METHOD="cartesian" - jdk: oraclejdk8 - env: TEST_SPARK_VERSION="2.4.2" LUCENERDD_ANALYZER_NAME="whitespace" LUCENERDD_LINKER_METHOD="collectbroadcast" + env: LUCENERDD_ANALYZER_NAME="whitespace" LUCENERDD_LINKER_METHOD="collectbroadcast" script: - - sbt ++$TRAVIS_SCALA_VERSION clean update - -Dlucenerdd.spatial.linker.method=${LUCENE_SPATIAL_LINKER_METHOD} -test + - sbt ++$TRAVIS_SCALA_VERSION -Dlucenerdd.spatial.linker.method=${LUCENE_SPATIAL_LINKER_METHOD} clean update test - sbt ++$TRAVIS_SCALA_VERSION scalastyle - sbt ++$TRAVIS_SCALA_VERSION assembly - travis_wait 30 sbt ++$TRAVIS_SCALA_VERSION clean coverage test coverageReport diff --git a/build.sbt b/build.sbt index c7a7f74b..4050268c 100644 --- a/build.sbt +++ b/build.sbt @@ -79,32 +79,20 @@ pomExtra := credentials += Credentials(Path.userHome / ".sbt" / ".credentials") -val luceneV = "8.0.0" - -spName := "zouzias/spark-lucenerdd" -sparkVersion := "2.4.2" -spShortDescription := "Spark RDD with Lucene's query capabilities" -sparkComponents ++= Seq("core", "sql", "mllib") -spAppendScalaVersion := true -// This is necessary because of how we explicitly specify Spark dependencies -// for tests rather than using the sbt-spark-package plugin to provide them. -spIgnoreProvided := true - -val testSparkVersion = settingKey[String]("The version of Spark to test against.") - -testSparkVersion := sys.props.get("spark.testVersion").getOrElse(sparkVersion.value) +val luceneV = "8.4.0" +val sparkVersion = "2.4.4" // scalastyle:off -val scalactic = "org.scalactic" %% "scalactic" % "3.0.7" -val scalatest = "org.scalatest" %% "scalatest" % "3.0.7" % "test" +val scalactic = "org.scalactic" %% "scalactic" % "3.1.0" +val scalatest = "org.scalatest" %% "scalatest" % "3.1.0" % "test" -val joda_time = "joda-time" % "joda-time" % "2.10.1" -val algebird = "com.twitter" %% "algebird-core" % "0.13.5" -val joda_convert = "org.joda" % "joda-convert" % "2.2.0" +val joda_time = "joda-time" % "joda-time" % "2.10.5" +val algebird = "com.twitter" %% "algebird-core" % "0.13.6" +val joda_convert = "org.joda" % "joda-convert" % "2.2.1" val spatial4j = "org.locationtech.spatial4j" % "spatial4j" % "0.7" -val typesafe_config = "com.typesafe" % "config" % "1.3.3" +val typesafe_config = "com.typesafe" % "config" % "1.3.4" val lucene_facet = "org.apache.lucene" % "lucene-facet" % luceneV val lucene_analyzers = "org.apache.lucene" % "lucene-analyzers-common" % luceneV @@ -135,9 +123,10 @@ libraryDependencies ++= Seq( ) libraryDependencies ++= Seq( - "org.apache.spark" %% "spark-core" % testSparkVersion.value % "test" force(), - "org.apache.spark" %% "spark-sql" % testSparkVersion.value % "test" force(), - "com.holdenkarau" %% "spark-testing-base" % s"2.4.0_0.11.0" % "test" intransitive(), + "org.apache.spark" %% "spark-core" % sparkVersion % "provided", + "org.apache.spark" %% "spark-sql" % sparkVersion % "provided", + "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided", + "com.holdenkarau" %% "spark-testing-base" % s"2.4.3_0.12.0" % "test" intransitive(), "org.scala-lang" % "scala-library" % scalaVersion.value % "compile" ) diff --git a/deployToSonartype.md b/deployToSonartype.md new file mode 100644 index 00000000..edb371bf --- /dev/null +++ b/deployToSonartype.md @@ -0,0 +1,25 @@ +## Setup + +# Add `sonatype.sbt` file under `~/.sbt/1.0/` folder with contents + +``` +credentials += Credentials("Sonatype Nexus Repository Manager", + "oss.sonatype.org", + "zouzias", + "PASSWORD_HERE") +``` + +## Run sbt release to release signed both 2.10 and 2.11 + +``` +sbt release +``` + +## Then, git checkout v0.X.X to the release tag first, and then type + +``` +sbt sonatypeRelease +``` + +## This will allow sonatype to release the artifacts to maven central. +## An alternative is to browse to https://oss.sonatype.org and do it manually diff --git a/deployToSonartype.sh b/deployToSonartype.sh deleted file mode 100644 index 8ea75f35..00000000 --- a/deployToSonartype.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - - -# Run sbt release to release signed both 2.10 and 2.11 -sbt release - - -# Then, git checkout v0.X.X to the release tag first, and then type - -sbt sonatypeRelease - -# This will allow sonatype to release the artifacts to maven central. -# An alternative is to browse to https://oss.sonatype.org and do it manually diff --git a/project/build.properties b/project/build.properties index 8e682c52..00b48d97 100644 --- a/project/build.properties +++ b/project/build.properties @@ -1 +1 @@ -sbt.version=0.13.18 +sbt.version=1.3.6 diff --git a/project/plugins.sbt b/project/plugins.sbt index e3447af7..50dd37ab 100644 --- a/project/plugins.sbt +++ b/project/plugins.sbt @@ -17,22 +17,20 @@ resolvers += "bintray-spark-packages" at "https://dl.bintray.com/spark-packages/maven/" -addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.7.0") +addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.9.0") -addSbtPlugin("com.timushev.sbt" % "sbt-updates" % "0.4.0") +addSbtPlugin("com.timushev.sbt" % "sbt-updates" % "0.5.0") -addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9") +addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10") -addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.11") +addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.12") addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "1.0.0") addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.5.1") -addSbtPlugin("org.scoverage" % "sbt-coveralls" % "1.2.6") +addSbtPlugin("org.scoverage" % "sbt-coveralls" % "1.2.7") -addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "1.1") +addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.5") -addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.1.1") - -addSbtPlugin("org.spark-packages" % "sbt-spark-package" % "0.2.6") +addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.1.2") diff --git a/spark-shell.sh b/spark-shell.sh index 82789320..d4b74db8 100755 --- a/spark-shell.sh +++ b/spark-shell.sh @@ -6,7 +6,7 @@ CURRENT_DIR=`pwd` SPARK_LUCENERDD_VERSION=`cat version.sbt | awk '{print $5}' | xargs` # You should have downloaded this spark version under your ${HOME} -SPARK_VERSION="2.4.0" +SPARK_VERSION="2.4.4" echo "===============================================" echo "Loading LuceneRDD with version ${SPARK_LUCENERDD_VERSION}"