Skip to content

Commit 7b9b6fe

Browse files
alldatafounderalldatafounder
authored
Merge pull request alldatacenter#330 from alldatacenter/alldatadc
[master] 引入ranger建设数据安全管理 && 引入shifu建设云原生IOT开发引擎
2 parents 55796f5 + 6230dbc commit 7b9b6fe

File tree

3,200 files changed

+934251
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

3,200 files changed

+934251
-0
lines changed

‎govern/data-security/DockerApp/Dockerfile

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
FROM java:8
2+
# 在主机/var/lib/docker创建临时文件链接到容器的/tmp
3+
VOLUME /tmp
4+
# 将jar包添加到容器中并更名为real-app.jar
5+
ADD docker-app.jar real-app.jar
6+
# 运行jar包
7+
RUN bash -c 'touch /real-app.jar'
8+
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/real-app.jar"]
9+

‎govern/data-security/DockerApp/build.sh

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
docker build -t docker-app .
2+
docker run -d -p 8080:8080 docker-app

‎govern/data-security/InstallGuide.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# INSTALL GUIDE FOR ALL DATA
2+
3+
```markdown
4+
5+
> 安装指南
6+
7+
```

‎govern/data-security/README.md

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# DATA GOVERN FOR ALL DATA PLATFORM 数据治理引擎
2+
3+
## 基于Apache Ranger建设AllData统一数据安全管理
4+
5+
> 数据全生命周期的安全管控
6+
>
7+
> 安全预警机制，让数据使用安全、合规
8+
>
9+
> 1、数据权限管理
10+
>
11+
> 2、敏感数据发现
12+
>
13+
> 3、隐私保护
14+
>
15+
> 4、风险预警

‎govern/data-security/ranger-2.3.0/.gitattributes

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Auto detect text files and perform LF normalization
2+
* text=auto
3+
4+
*.java text diff=java eol=lf
5+
*.html text diff=html eol=lf
6+
*.py text diff=python eol=lf
7+
*.css text eol=lf
8+
*.js text eol=lf
9+
*.sql text eol=lf
10+
*.sh text eol=lf
11+
*.xml text eol=lf
12+
*.txt text eol=lf
13+
*.json text eol=lf
14+
*.jsp text eol=lf
15+
*.csv text eol=lf
16+
*.properties text eol=lf

‎govern/data-security/ranger-2.3.0/.gitignore

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
*.class
2+
*.iml
3+
.pydevproject
4+
.settings/
5+
.metadata
6+
.classpath
7+
.project
8+
/target/
9+
/venv/
10+
winpkg/target
11+
.DS_Store
12+
.idea
13+
14+
#Python
15+
*.pyc
16+
**/build
17+
**/dist
18+
**/apache_ranger.egg-info
19+
.python-version

‎govern/data-security/ranger-2.3.0/.travis.yml

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
17+
sudo: false
18+
dist: bionic
19+
language: generic
20+
21+
cache:
22+
directories:
23+
- $HOME/.m2
24+
25+
env:
26+
global:
27+
- KEY=default VALUE=default
28+
29+
# Environment to testing with different versions, disabled because ranger is not compatible
30+
# - KEY=hadoop.version VALUE=2.8.0
31+
32+
jobs:
33+
include:
34+
- name: Linux AMD64 OpenJDK8
35+
env: JDK_VERSION=8
36+
- name: Linux AMD64 OpenJDK11
37+
env: JDK_VERSION=11
38+
- name: Linux ARM64 OpenJDK11
39+
arch: arm64-graviton2
40+
dist: focal
41+
virt: lxd
42+
group: edge
43+
env:
44+
- JDK_VERSION=11
45+
- MAVEN_ARGS="-DskipJSTests=true -P!all"
46+
47+
before_install:
48+
- export MAVEN_OPTS="-Xmx1200M -XX:MaxPermSize=768m -Xms512m"
49+
- sudo apt update -y
50+
- sudo apt install openjdk-${JDK_VERSION}-jdk maven
51+
52+
install:
53+
- mvn install $MAVEN_ARGS -D$KEY=$VALUE -DskipTests -Dmaven.javadoc.skip=true --no-transfer-progress -B -V
54+
55+
# KafkaRangerAuthorizerGSSTest is a failing test, TestLdapUserGroup needs too much memory for travis
56+
script:
57+
- mvn test $MAVEN_ARGS -D$KEY=$VALUE -Dmaven.javadoc.skip=true -B -V -pl !plugin-kafka,!ugsync,!hive-agent
58+
- mvn test $MAVEN_ARGS -D$KEY=$VALUE -Dmaven.javadoc.skip=true -B -V -pl plugin-kafka -Dtest="*,!KafkaRangerAuthorizerGSSTest"
59+
- mvn test $MAVEN_ARGS -D$KEY=$VALUE -Dmaven.javadoc.skip=true -B -V -pl ugsync -Dtest="*,!TestLdapUserGroup"
60+
- if [[ "$JDK_VERSION" != "8" ]]; then mvn test $MAVEN_ARGS -D$KEY=$VALUE -Dmaven.javadoc.skip=true -B -V -pl hive-agent -Dtest="*,!HIVERangerAuthorizerTest" -DfailIfNoTests=false ; fi
61+
- if [[ "$JDK_VERSION" == "8" ]]; then mvn test $MAVEN_ARGS -D$KEY=$VALUE -Dmaven.javadoc.skip=true -B -V -pl hive-agent ; fi

‎govern/data-security/ranger-2.3.0/DISCLAIMER.txt

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Apache Ranger is a Top Level Project (TLP) at the Apache Software Foundation (ASF).
2+
3+
This product includes software developed at The Apache Software
4+
Foundation (http://www.apache.org/).
5+
6+
http://ranger.apache.org

‎govern/data-security/ranger-2.3.0/LICENSE.txt

+323
Large diffs are not rendered by default.

‎govern/data-security/ranger-2.3.0/NOTICE.txt

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Apache Ranger
2+
Copyright 2014-2022 The Apache Software Foundation
3+
4+
This product includes software developed at The Apache Software Foundation (http://www.apache.org/).
5+
This product includes software developed by Spring Security Project (http://www.springframework.org/security)
6+
This product includes Azure/azure-sdk-for-java (https://github.com/Azure/azure-sdk-for-java/blob/master/LICENSE.txt ), Copyright (c) 2015 Microsoft.

‎govern/data-security/ranger-2.3.0/README.txt

+111
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
Check Apache Ranger Wiki for up to date instruction:
17+
https://cwiki.apache.org/confluence/display/RANGER/Index
18+
19+
NOTE about using sending pull request on github:
20+
================================================
21+
Apache Ranger is currently NOT setup to use pull requests to take in the changes for commit.
22+
Please use the apache review board to submit your code changes for review and commit. https://reviews.apache.org
23+
Also create a jira to go along with the review and mention it in the review board review. https://issues.apache.org/jira/browse/RANGER
24+
25+
Build Process
26+
=============
27+
28+
1. Check out the code from GIT repository
29+
30+
2. On the root folder, please execute the following Maven command:
31+
32+
$ mvn clean compile package install
33+
$ mvn eclipse:eclipse
34+
35+
(Ranger Admin UI tests depend on PhantomJS. If the build fails with npm or Karma errors you can either
36+
i. install PhantomJS dependencies for your platform (bzip2 and fontconfig)
37+
ii. skip JavaScript test execution: mvn -DskipJSTests ...)
38+
39+
3. After the above build command execution, you should see the following TAR files in the target folder:
40+
41+
42+
ranger-<version>-admin.tar.gz
43+
ranger-<version>-atlas-plugin.tar.gz
44+
ranger-<version>-hbase-plugin.tar.gz
45+
ranger-<version>-hdfs-plugin.tar.gz
46+
ranger-<version>-hive-plugin.tar.gz
47+
ranger-<version>-kafka-plugin.tar.gz
48+
ranger-<version>-kms.tar.gz
49+
ranger-<version>-knox-plugin.tar.gz
50+
ranger-<version>-migration-util.tar.gz
51+
ranger-<version>-ranger-tools.tar.gz
52+
ranger-<version>-solr-plugin.tar.gz
53+
ranger-<version>-sqoop-plugin.tar.gz
54+
ranger-<version>-src.tar.gz
55+
ranger-<version>-storm-plugin.tar.gz
56+
ranger-<version>-tagsync.tar.gz
57+
ranger-<version>-usersync.tar.gz
58+
ranger-<version>-yarn-plugin.tar.gz
59+
ranger-<version>-kylin-plugin.tar.gz
60+
ranger-<version>-elasticsearch-plugin.tar.gz
61+
ranger-<version>-ozone-plugin.tar.gz
62+
ranger-<version>-presto-plugin.tar.gz
63+
ranger-<version>-schema-registry-plugin.tar.gz
64+
65+
Importing Apache Ranger Project into Eclipse
66+
============================================
67+
68+
1. Create a Eclipse workspace called 'ranger'
69+
70+
2. Import maven project from the root directory where ranger source code is downloaded (and build)
71+
72+
73+
Deployment Process
74+
==================
75+
76+
Installation Host Information
77+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
78+
1. Ranger Admin Tool Component (ranger-<version-number>-admin.tar.gz) should be installed on a host where Policy Admin Tool web application runs on port 6080 (default).
79+
2. Ranger User Synchronization Component (ranger-<version-number>-usersync.tar.gz) should be installed on a host to synchronize the external user/group information into Ranger database via Ranger Admin Tool.
80+
3. Ranger Component plugin should be installed on the component boxes:
81+
(a) HDFS Plugin needs to be installed on Name Node hosts
82+
(b) Hive Plugin needs to be installed on HiveServer2 hosts
83+
(c) HBase Plugin needs to be installed on both Master and Regional Server nodes.
84+
(d) Knox Plugin needs to be installed on Knox gateway host.
85+
(e) Storm Plugin needs to be installed on Storm hosts.
86+
(f) Kafka/Solr Plugin needs to be installed on their respective component hosts.
87+
(g) YARN plugin needs to be installed on YARN Resource Manager hosts
88+
(h) Sqoop plugin needs to be installed on Sqoop2 hosts
89+
(i) Kylin plugin needs to be installed on Kylin hosts
90+
(j) Elasticsearch plugin needs to be installed on Elasticsearch hosts
91+
(k) Ozone plugin needs to be installed on Ozone hosts
92+
(l) Presto plugin needs to be installed on Presto hosts
93+
(m) Schema Registry plugin needs to be installed on Schema Registry hosts
94+
95+
Installation Process
96+
~~~~~~~~~~~~~~~~~~~~
97+
98+
1. Download the tar.gz file into a temporary folder in the box where it needs to be installed.
99+
100+
2. Expand the tar.gz file into /usr/lib/ranger/ folder
101+
102+
3. Go to the component name under the expanded folder (e.g. /usr/lib/ranger/ranger-<version-number>-admin/)
103+
104+
4. Modify the install.properties file with appropriate variables
105+
106+
5. If the module has setup.sh,
107+
Execute ./setup.sh
108+
109+
If the install.sh file does not exists,
110+
Execute ./enable-<component>-plugin.sh
111+

‎govern/data-security/ranger-2.3.0/agents-audit/.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
/target/
2+
/bin/
3+
/target
4+
.settings/

0 commit comments

Comments
 (0)