Skip to content

Commit

Permalink
[KYUUBI apache#6197] Revise dependency management of Spark authZ plugin
Browse files Browse the repository at this point in the history 
# 🔍 Description
## Issue References 🔗

The POM of `kyuubi-spark-authz-shaded` is redundant, just pull `kyuubi-spark-authz` is necessary.

The current dependency management does not work on Ranger 2.1.0, this patch cleans up the POM definition and fixes the compatibility with Ranger 2.1.0

## Describe Your Solution 🔧

Carefully revise the dependency list and exclusion.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

perform packing kyuubi-spark-authz-shaded module.
```
build/mvn clean install -pl extensions/spark/kyuubi-spark-authz-shaded -am -DskipTests
```

before
```
[INFO] --- maven-shade-plugin:3.5.2:shade (default)  kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including org.apache.kyuubi:kyuubi-spark-authz_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util-scala_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common🫙2.4.0 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs🫙1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl🫙1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl🫙1.9.13 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred🫙2.4.0 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client🫙1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core🫙1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j🫙1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna🫙5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform🫙5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit🫙2.4.0 in the shaded jar.
```

after

```
[INFO] --- maven-shade-plugin:3.5.2:shade (default)  kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including org.apache.kyuubi:kyuubi-spark-authz_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util-scala_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common🫙2.4.0 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs🫙1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl🫙1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl🫙1.9.13 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred🫙2.4.0 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client🫙1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core🫙1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j🫙1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna🫙5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform🫙5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugin-classloader🫙2.4.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit🫙2.4.0 in the shaded jar.
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6197 from pan3793/authz-dep.

Closes apache#6197

d0becab [Cheng Pan] 2.4
47e3850 [Cheng Pan] ranger 2.4
af01f7e [Cheng Pan] test ranger 2.1
203aff3 [Cheng Pan] ranger-plugins-cred
974d76b [Cheng Pan] Resive dependency management of authz
e5154f3 [Cheng Pan] improve authz deps

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
  • Loading branch information
@pan3793
pan3793 committed Mar 22, 2024
1 parent 67f099a commit 3b9f25b
Show file tree
Hide file tree
Showing 3 changed files with 60 additions and 236 deletions.
210 changes: 5 additions & 205 deletions extensions/spark/kyuubi-spark-authz-shaded/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,197 +30,12 @@
<name>Kyuubi Dev Spark Authorization Extension Shaded</name>
<url>https://kyuubi.apache.org/</url>

<properties>
<!-- the following components' version may need to tune to align w/ the ranger.version-->
<gethostname4j.version>1.0.0</gethostname4j.version>
<jersey.client.version>1.19.4</jersey.client.version>
<jna.version>5.7.0</jna.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-authz_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-util-scala_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-common</artifactId>
<version>${ranger.version}</version>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-bundle</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugin-classloader</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.kstruct</groupId>
<artifactId>gethostname4j</artifactId>
</exclusion>
<exclusion>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
</exclusion>
<exclusion>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<version>${jersey.client.version}</version>
<exclusions>
<exclusion>
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>com.kstruct</groupId>
<artifactId>gethostname4j</artifactId>
<version>${gethostname4j.version}</version>
</dependency>

<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version>
</dependency>

<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
<version>${jna.version}</version>
</dependency>

<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
<version>${ranger.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-cred</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch.client</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.lucene</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
<exclusion>
<groupId>com.carrotsearch</groupId>
<artifactId>hppc</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
<exclusion>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>

<build>
Expand All @@ -232,18 +47,12 @@
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
<include>org.apache.kyuubi:kyuubi-util-scala_${scala.binary.version}</include>
<include>org.apache.kyuubi:kyuubi-spark-authz_${scala.binary.version}</include>
<include>org.apache.kyuubi:kyuubi-util</include>
<include>org.apache.ranger:ranger-plugins-common</include>
<include>org.apache.ranger:ranger-plugins-audit</include>
<include>org.apache.ranger:ranger-plugins-cred</include>
<include>org.codehaus.jackson:jackson-jaxrs</include>
<include>org.codehaus.jackson:jackson-core-asl</include>
<include>org.codehaus.jackson:jackson-mapper-asl</include>
<include>com.sun.jersey:jersey-client</include>
<include>com.sun.jersey:jersey-core</include>
<include>org.apache.kyuubi:*</include>
<include>org.apache.ranger:*</include>
<include>org.codehaus.jackson:*</include>
<include>com.sun.jersey:*</include>
<include>com.kstruct:gethostname4j</include>
<!-- JNA is the transitive dependency of gethostname4j -->
<include>net.java.dev.jna:jna</include>
<include>net.java.dev.jna:jna-platform</include>
</includes>
Expand Down Expand Up @@ -287,15 +96,6 @@
<pattern>com.kstruct.gethostname4j</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.kstruct.gethostname4j</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.hadoop.security</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.apache.hadoop.security</shadedPattern>
<includes>
<include>org.apache.hadoop.security.KrbPasswordSaverLoginModule</include>
<include>org.apache.hadoop.security.SecureClientLogin</include>
<include>org.apache.hadoop.security.SecureClientLoginConfiguration</include>
</includes>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"></transformer>
Expand Down
85 changes: 55 additions & 30 deletions extensions/spark/kyuubi-spark-authz/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
<url>https://kyuubi.apache.org/</url>

<properties>
<ranger.version>2.4.0</ranger.version>
<!-- the following components' version may need to tune to align w/ the ranger.version-->
<gethostname4j.version>1.0.0</gethostname4j.version>
<jersey.client.version>1.19.4</jersey.client.version>
Expand All @@ -49,10 +50,6 @@
<artifactId>ranger-plugins-common</artifactId>
<version>${ranger.version}</version>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-bundle</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugin-classloader</artifactId>
Expand All @@ -61,6 +58,11 @@
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
</exclusion>
<!-- this is going to be replaced with jersey-client -->
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-bundle</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
Expand All @@ -70,17 +72,13 @@
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<groupId>org.apache.commons</groupId>
<artifactId>commons-configuration2</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
Expand All @@ -89,14 +87,6 @@
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
<exclusion>
<groupId>com.kstruct</groupId>
<artifactId>gethostname4j</artifactId>
Expand Down Expand Up @@ -142,6 +132,12 @@
<version>${jna.version}</version>
</dependency>

<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugin-classloader</artifactId>
<version>${ranger.version}</version>
</dependency>

<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
Expand Down Expand Up @@ -226,6 +222,35 @@
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-cred</artifactId>
<version>${ranger.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-configuration2</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<!-- they were removed in RANGER-3184 (2.2.0) -->
<exclusion>
<groupId>org.eclipse.persistence</groupId>
<artifactId>javax.persistence</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.persistence</groupId>
<artifactId>eclipselink</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
Expand Down Expand Up @@ -256,18 +281,6 @@
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
Expand All @@ -294,6 +307,18 @@
<!-- for hive related test only -->
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
Expand Down
1 change: 0 additions & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,6 @@
<prometheus.version>0.16.0</prometheus.version>
<protobuf.version>3.21.7</protobuf.version>
<py4j.version>0.10.7</py4j.version>
<ranger.version>2.4.0</ranger.version>
<scalatest.version>3.2.16</scalatest.version>
<scalatestplus.version>3.2.16.0</scalatestplus.version>
<scopt.version>4.1.0</scopt.version>
Expand Down

0 comments on commit 3b9f25b

Please sign in to comment.