Skip to content

Commit

Permalink
Initial Commit
Browse files Browse the repository at this point in the history
  • Loading branch information
asenf committed Feb 18, 2018
0 parents commit fe53583
Show file tree
Hide file tree
Showing 17 changed files with 1,202 additions and 0 deletions.
66 changes: 66 additions & 0 deletions Overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# EGA Data API v3 -- Overview

The EGA Data API is a portable set of microservices providing a REST API to access to EGA data. There are currently several clients available for this API, including a command line download client and a FUSE layer for direct random access to archived files.

The API is written in Java 1.8 using the Spring Framework and Spring Boot 1.4. As a prerequisite two Spring Servers are expected to be running at all times:
* (Netflix) Eureka Server
* (Netflix) Config Server

All sercives register themselves with the Eureka instance, using the Spring application name `{app.name}` specified in `'bootstrap.configuration'`:`"spring.application.name={app.name}"`. Contact between microservices then utilises `{app.name}` instead of the absolute URL. Eureka automatically resolves the application name to a URL; if multiple instances with the same application name are present, Eureka automatically performs Ribbon load balancing between the instances. This greatly simplifies deployment and allows for easy and dynamic scaling to meet demand.

The Configuration server serves all `'application.configuration'` files to the respective microservices. The location (URL) of the config server must be specified in the `'bootstrap.configuration'`:`"spring.cloud.config.uri="` file of each service. This entry requires an absolute URL to the configration server. This is necessary because the configuration file must be loaded before the application startup. A config file named `'{app.name}.configuration'` must be in the config directory referenced by the configuration server before a service with `"spring.application.name={app.name}"` can be started.

Without these two services the remaining microservices will not function properly.

There are two microservices interfacing with a database:
* FILEDATABASE
* PERMISSIONSDATABASE [used only at Central EGA]

The PERMISSIONSDATABASE service reads (read-only) permissions-related information (user-dataset associations; etc.) while the FILEDATABASE service serves archive-related information (file-dataset associations; file paths; etc.) and keeps some logs (read/write).

Two microservices provide publicly accessible Edge services:
* CENTRAL [used only at Central EGA]
* DATAEDGE

DATAEDGE provides access to the data. It provides access to archived files directly; it streams downloas. This is the primary back end for the FUSE layer. All DATAEDGE endpoints require a valid EGA (ELIXIR) OAuth2 Bearer Token. Security for CENTRAL is different because it is meant as a service-facing API. CENTRAL serves user-dataset associations, and general dataset-file information.

One microservice serves as front door to all requests. It is a reverse proxy/filter and performs functions such as rate limitations, permissions-injection for ELIXIR access tokens. It also forwards all requests to an available back end service, based on availability provided by EUREKA, thus tying all services behind a single URL and load-balancing multiple back end services.

At EGA Central these services are deployed behind an SSL terminating load balancer; therefore https is not initially implemented with these services.

The CPU heavy cryptographic work is performed by the RES service, supported by a Key provider service and an H2 database (to enable this service to be properly load balanced). The key provider service produces the archive decryption key for a specified file, and is only used from within RES. The H2 database stores MD5 values of data transfers, to enable verification of completed downloads.
* KEY
* H2 DB
* RES

### Summary
* EUREKA
* Requires: CONFIG
* Required By: DATAEDGE, CENTRAL PERMISSIONSDATABASE, FILEDATABASE, RES, KEY, ZUUL
* CONFIG
* Requires:
* Required By: DATAEDGE, CENTRAL PERMISSIONSDATABASE, FILEDATABASE, RES, KEY
* H2
* Requires:
* Required By: RES, DATAEDGE
* KEY
* Requires: EUREKA, CONFIG
* Required By: RES
* PERMISSIONSDATABASE
* Requires: EUREKA, CONFIG
* Required By: CENTRAL
* FILEDATABASE
* Requires: EUREKA, CONFIG
* Required By: DATAEDGE
* DATAEDGE
* Requires: EUREKA, CONFIG, FILEDATABASE
* Required By: ZUUL
* CENTRAL
* Requires: EUREKA, CONFIG, PERMISSIONSDATABASE
* Required By: ZUUL
* RES
* Requires: EUREKA, CONFIG, KEY, H2
* Required By: FILEDATABASE
* ZUUL
* Requires: EUREKA, CONFIG
* Required By:
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Zuul Server

Run this app as a normal Spring Boot app. If you run from this project
it will be on port 8765 (per the `application.yml`).
Also run : [eureka](https://github.com/EGA-archive/ega-eureka-service)

Zuul has several functions:
* Integration with ELIXIR. The Zuul filter detects ELIXIR tokens, and injects EGA permissions in the REST call as a signed "X-Permissions" header.
* Traffic Shaping to protect the underlying resources
* Load balancing: automatically (via EUREKA) detect and add/remove available back end API services, and load balance between them. Also perform atomatic retries of REST calls.
* Integration: Make several back end microservices availabe on a single URL and Port
23 changes: 23 additions & 0 deletions docker/Dockerfile_Deploy
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Use phusion/baseimage as base image.
FROM phusion/baseimage:latest

# Use baseimage-docker's init system.
CMD ["/sbin/my_init"]

# custon build instructions here...
# Java: OpenJDK8
RUN apt-get update
RUN apt-get install -y software-properties-common python-software-properties
RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt-get update
RUN apt-get -y install openjdk-8-jdk
ADD zuul-server-1.0.0.BUILD-SNAPSHOT.jar /zuul-server-1.0.0.BUILD-SNAPSHOT.jar
RUN mkdir /etc/service/ega_v3_zuul
ADD zuuld.sh /etc/service/ega_v3_zuul/run
RUN chmod +x /etc/service/ega_v3_zuul/run

# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Port
EXPOSE 8051
30 changes: 30 additions & 0 deletions docker/Dockerfile_EGABuild
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Build Ubuntu Base Image
FROM ubuntu:latest

# For now...
MAINTAINER Alexander Senf <[email protected]>

# ROOT to set up the image
USER root

# Add a user and group first to make sure their IDs get assigned consistently, regardless of whatever dependencies get added
#RUN groupadd -r ega && useradd -r -g ega ega

# Dirs in Docker FS
#RUN mkdir /docker-entrypoint-initdb.d

# Suppress unnecesary warning messages
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections

# Enable addition of Repositories in Ubuntu
RUN apt-get update -y
RUN apt-get install -y software-properties-common python-software-properties

# Update Image, and Install Java 8 & Monit
RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt-get -y update
RUN apt-get -y install openjdk-8-jdk
RUN apt-get -y install git
RUN apt-get -y install maven
RUN apt -y upgrade

6 changes: 6 additions & 0 deletions docker/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/bash
git clone https://github.com/EbiEga/Ega_ZUUL_Server.git
mvn -f /Ega_ZUUL_Server/pom.xml install
mv /Ega_ZUUL_Server/target/zuul-server-1.0.0.BUILD-SNAPSHOT.jar /EGA_build
mv /Ega_ZUUL_Server/docker/eurekad.sh /EGA_build
mv /Ega_ZUUL_Server/docker/Dockerfile_Deploy /EGA_build
2 changes: 2 additions & 0 deletions docker/runfromimage.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/bash
sudo docker run -d -p 8051:8051 alexandersenf/ega_zuul
8 changes: 8 additions & 0 deletions docker/runfromsource.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
sudo docker run --rm --name build -v $DIR:/EGA_build -it alexandersenf/ega_zuul sh -c 'exec /EGA_build/build.sh'
sudo docker build -t ega_zuul -f Dockerfile_Deploy .
sudo rm zuul-server-1.0.0.BUILD-SNAPSHOT.jar
sudo rm Dockerfile_Deploy
sudo rm zuuld.sh
sudo docker run -d -p 8051:8051 ega_zuul
12 changes: 12 additions & 0 deletions docker/zuuld.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash
SERVICE_NAME=ZuulService
PATH_TO_JAR=/zuul-server-1.0.0.BUILD-SNAPSHOT.jar
PROCESSCNT=$(ps x | grep -v grep | grep -c "zuul-server-1.0.0.BUILD-SNAPSHOT.jar")
#PID=$(ps aux | grep "zuul-server-1.0.0.BUILD-SNAPSHOT.jar" | grep -v grep | awk '{print $2}')
if [ $PROCESSCNT == 0 ]; then
echo "Starting $SERVICE_NAME ..."
nohup java -jar $PATH_TO_JAR 2>> /dev/null >> /dev/null &
echo "$SERVICE_NAME started ..."
#else
# echo "$SERVICE_NAME is already running ..."
fi
231 changes: 231 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,231 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>org.demo</groupId>
<artifactId>zuul-server</artifactId>
<packaging>jar</packaging>
<name>Spring Cloud Netflix Zuul Server</name>
<url>http://projects.spring.io/spring-cloud/</url>
<version>1.0.0.BUILD-SNAPSHOT</version>

<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.10.BUILD-SNAPSHOT</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<!--skip deploy (this is just a test module) -->
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<!--
<plugin>
<groupId>com.spotify</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.4.11</version>
<configuration>
<imageName>ega-data-api/zuul-server</imageName>
<baseImage>java:8</baseImage>
<entryPoint>["java", "-jar", "/${project.build.finalName}.jar"]</entryPoint>
<resources>
<resource>
<targetPath>/</targetPath>
<directory>${project.build.directory}</directory>
<include>${project.build.finalName}.jar</include>
</resource>
</resources>
</configuration>
<executions>
<execution>
<id>build-image</id>
<phase>package</phase>
<goals>
<goal>build</goal>
</goals>
</execution>
</executions>
</plugin>
-->

</plugins>
</build>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Camden.SR7</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-eureka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<!--
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-oauth2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zuul</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-eureka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
-->
<!-- <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
-->
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>3.4.2</version>
</dependency>
<dependency>
<groupId>com.squareup.moshi</groupId>
<artifactId>moshi</artifactId>
<version>1.3.1</version>
</dependency>

<dependency>
<groupId>com.github</groupId>
<artifactId>bucket4j</artifactId>
<version>1.3.0</version>
</dependency>

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.16.10</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>com.marcosbarbero.cloud</groupId>
<artifactId>spring-cloud-zuul-ratelimit</artifactId>
<version>1.0.7.RELEASE</version>
</dependency>
<!--
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.8.2</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.8.2</version>
</dependency>
-->
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-jwt</artifactId>
<version>1.0.9.RELEASE</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.springframework.security.oauth</groupId>
<artifactId>spring-security-oauth2</artifactId>
<version>2.0.14.RELEASE</version>
<type>jar</type>
</dependency>
</dependencies>

<repositories>
<repository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>http://repo.spring.io/libs-snapshot-local</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>http://repo.spring.io/libs-miletone-local</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>spring-releases</id>
<name>Spring Releases</name>
<url>http://repo.spring.io/libs-release-local</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>

<repository>
<id>jcenter</id>
<url>http://jcenter.bintray.com</url>
</repository>

</repositories>
<pluginRepositories>
<pluginRepository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>http://repo.spring.io/libs-snapshot-local</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
<pluginRepository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>http://repo.spring.io/libs-milestone-local</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>

</project>
Loading

0 comments on commit fe53583

Please sign in to comment.