Skip to content

Commit

Permalink
Add multiple datasources to Jdbc example (#180)
Browse files Browse the repository at this point in the history 
* Add multiple datasources to Jdbc example

* excluded .gitignore and added license

* Changed README.adoc

* Updated examples.json

* skipping test containers-tests
  • Loading branch information
shravani committed Nov 23, 2023
1 parent 173c1dc commit 1d47146
Show file tree
Hide file tree
Showing 13 changed files with 451 additions and 141 deletions.
10 changes: 5 additions & 5 deletions docs/modules/ROOT/attachments/examples.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,6 @@
"description": "Shows how to use Camel CXF SOAP component.",
"link": "https://github.com/apache/camel-quarkus-examples/tree/main/cxf-soap"
},
{
"title": "Connecting to a JDBC DataSource",
"description": "Shows how to connect to a Database using Datastores.",
"link": "https://github.com/apache/camel-quarkus-examples/tree/main/jdbc-datasource"
},
{
"title": "Custom `main()`",
"description": "Shows how to start Camel from a custom `main()` method",
Expand All @@ -19,6 +14,11 @@
"description": "Shows how to deploy a Camel Quarkus route as an AWS Lambda function",
"link": "https://github.com/apache/camel-quarkus-examples/tree/main/aws-lambda"
},
{
"title": "Extract, Transform and Load between two databases",
"description": "Shows how to extract, transform and load between two databases",
"link": "https://github.com/apache/camel-quarkus-examples/tree/main/jdbc-datasource"
},
{
"title": "File consumer with Bindy \u0026 FTP",
"description": "Shows how to consume CSV files, marshal \u0026 unmarshal the data and send it onwards via FTP",
Expand Down
115 changes: 82 additions & 33 deletions jdbc-datasource/README.adoc
View file
Original file line number Diff line number Diff line change
@@ -1,69 +1,118 @@
= Connecting to a JDBC DataSource: A Camel Quarkus example
:cq-example-description: An example that shows how to connect to a Database using Datastores.
= Extract, Transform and Load between two databases: A Camel Quarkus example
:cq-example-description: An example that shows how to extract, transform and load between two databases

{cq-description}

In particular, it demonstrates the following:
TIP: Check the https://camel.apache.org/camel-quarkus/latest/first-steps.html[Camel Quarkus User guide] for prerequisites
and other general information.

1. Defining a DataSource
2. Querying the Database defined in the previous `DataSource`
3. Usage of properties defined in `application.properties`
4. No Java code required or used, the route defined in XML can still be compiled to native code.
== Start the source and target databases

This example will connect to an H2 database with the connection details defined in `application.properties`.
If the example is run on Development mode and no database exists, Quarkus will create a matching database
https://quarkus.io/guides/datasource#dev-services[as described here].
All the commands in this example are expected to be run from the example directory, at the same level than the `pom.xml` file.

TIP: Check the https://camel.apache.org/camel-quarkus/latest/first-steps.html[Camel Quarkus User guide] for prerequisites
and other general information.
In a first terminal, let's start the source database by executing the command below:

== Start in the Development mode
[source,shell]
----
docker run -p 5432:5432 \
-e POSTGRES_USER=ETL_source_user \
-e POSTGRES_PASSWORD=1234567@8_source \
-e POSTGRES_DB=source_db \
-v ${PWD}/src/test/resources/init-source-db.sql:/docker-entrypoint-initdb.d/init-source-db.sql \
docker.io/postgres:15.0
----

In a second terminal, let's start the target database:

[source,shell]
----
$ mvn clean compile quarkus:dev
docker run -p 5433:5432 \
-e POSTGRES_USER=ETL_target_user \
-e POSTGRES_PASSWORD=1234567@8_target \
-e POSTGRES_DB=target_db \
-v ${PWD}/src/test/resources/init-target-db.sql:/docker-entrypoint-initdb.d/init-target-db.sql \
docker.io/postgres:15.0
----

The above command compiles the project, starts the application and lets the Quarkus tooling watch for changes in your
workspace. Any modifications in your project will automatically take effect in the running application.
== Running the application in dev mode

You can run your application in dev mode that enables live coding using:

[source,shell]
----
mvn compile quarkus:dev
----

[NOTE]
====
Quarkus now ships with a Dev UI, which is available in dev mode only at http://localhost:8080/q/dev/.
====

TIP: Please refer to the Development mode section of
https://camel.apache.org/camel-quarkus/latest/first-steps.html#_development_mode[Camel Quarkus User guide] for more details.

=== Package and run the application
Extract, Transform and Load related logs should be output as below:

[source,shell]
----
2023-11-14 15:12:55,878 INFO [route17] (Camel (camel-9) thread #9 - timer://insertCamel) Extracting data from source database
2023-11-14 15:12:55,881 INFO [route17] (Camel (camel-9) thread #9 - timer://insertCamel) -> Transforming review for hotel 'Grand Hotel'
2023-11-14 15:12:55,886 INFO [route17] (Camel (camel-9) thread #9 - timer://insertCamel) -> Loading transformed data in target database
2023-11-14 15:12:55,893 INFO [route17] (Camel (camel-9) thread #9 - timer://insertCamel) -> Transforming review for hotel 'Middle Hotel'
2023-11-14 15:12:55,897 INFO [route17] (Camel (camel-9) thread #9 - timer://insertCamel) -> Loading transformed data in target database
2023-11-14 15:12:55,904 INFO [route17] (Camel (camel-9) thread #9 - timer://insertCamel) -> Transforming review for hotel 'Small Hotel'
2023-11-14 15:12:55,909 INFO [route17] (Camel (camel-9) thread #9 - timer://insertCamel) -> Loading transformed data in target database
----

=== Packaging and running the application

Once you are done with developing you may want to package and run the application.

TIP: Find more details about the JVM mode and Native mode in the Package and run section of
https://camel.apache.org/camel-quarkus/latest/first-steps.html#_package_and_run_the_application[Camel Quarkus User guide]

==== JVM mode
The application can be packaged using:

[source,shell]
----
$ mvn clean package
$ java -jar target/quarkus-app/quarkus-run.jar
...
mvn package
----

It produces the `quarkus-run.jar` file in the `target/quarkus-app/` directory.
Be aware that it’s not an _über-jar_ as the dependencies are copied into the `target/quarkus-app/lib/` directory.

The application is now runnable executing the command below:

[io.quarkus] (main) camel-quarkus-examples-... started in 0.570s.
[source,shell]
----
java -jar target/quarkus-app/quarkus-run.jar
----

==== Native mode
The application should output the same logs than in previous section.

==== Creating a native executable

Finally, the application can be compiled to native with the following command:

IMPORTANT: Native mode requires having GraalVM and other tools installed. Please check the Prerequisites section
of https://camel.apache.org/camel-quarkus/latest/first-steps.html#_prerequisites[Camel Quarkus User guide].
[source,shell]
----
mvn package -Dnative
----

Or, if you don't have GraalVM installed, you can run the native executable build in a container using:

[source,shell]
----
mvn package -Dnative -Dquarkus.native.container-build=true
----

To prepare a native executable using GraalVM, run the following command:
Either way, the resulting native executable could be start as below:

[source,shell]
----
$ mvn clean package -Pnative
$ ./target/*-runner
...
[io.quarkus] (main) camel-quarkus-examples-... started in 0.011s.
...
./target/*-runner
----

== Feedback
The application should output the same logs than in previous section.

Please report bugs and propose improvements via https://github.com/apache/camel-quarkus/issues[GitHub issues of Camel Quarkus] project.
If you want to learn more about building native executables, please consult https://quarkus.io/guides/maven-tooling.
109 changes: 65 additions & 44 deletions jdbc-datasource/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,50 +66,59 @@
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-microprofile-health</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-xml-io-dsl</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-log</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-timer</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-jdbc</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-jdbc-h2</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-agroal</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit5</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-test-h2</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-bean</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-timer</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-platform-http</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-agroal</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-jdbc-postgresql</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-arc</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit5</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
Expand Down Expand Up @@ -184,6 +193,7 @@
<exclude>**/NOTICE</exclude>
<exclude>**/README</exclude>
<exclude>**/pom.xml.versionsBackup</exclude>
<exclude>**/quarkus.log*</exclude>
</excludes>
<mapping>
<java>SLASHSTAR_STYLE</java>
Expand Down Expand Up @@ -273,5 +283,16 @@
<quarkus.package.type>native</quarkus.package.type>
</properties>
</profile>
<profile>
<id>skip-testcontainers-tests</id>
<activation>
<property>
<name>skip-testcontainers-tests</name>
</property>
</activation>
<properties>
<skipTests>true</skipTests>
</properties>
</profile>
</profiles>
</project>
36 changes: 27 additions & 9 deletions jdbc-datasource/src/main/java/org/acme/jdbc/JdbcRoutes.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,37 @@
*/
package org.acme.jdbc;

import java.util.HashMap;
import java.util.Map;

import org.apache.camel.builder.RouteBuilder;

public class JdbcRoutes extends RouteBuilder {
@Override
public void configure() throws Exception {
from("timer://insertCamel?period=1000")
.log("Inserting Camel ${messageTimestamp}")
.setBody().simple("INSERT INTO Camel (timestamp) VALUES (${messageTimestamp})")
.to("jdbc:camel-ds")
.log("Inserted Camel ${messageTimestamp}")
.setBody().simple("SELECT * FROM Camel")
.to("jdbc:camel-ds")
.log("We have ${header[CamelJdbcRowCount]} camels in the database.")
.log("Camels found: ${body}");
// Define a mapping for the review values
HashMap<String, Integer> reviewMapping = new HashMap<>();
reviewMapping.put("best", 1);
reviewMapping.put("good", 0);
reviewMapping.put("worst", -1);

from("timer://insertCamel?period=1000&repeatCount={{etl.timer.repeatcount}}")
.setBody().simple("DELETE FROM Target")
.to("jdbc:target_db")
.setBody().simple("SELECT * FROM Source")
.to("jdbc:source_db")
.log("Extracting data from source database")
.split(body())
.process(exchange -> {
Map<String, Object> sourceData = exchange.getIn().getBody(Map.class);
String review = (String) sourceData.get("review");
int mappedReview = reviewMapping.getOrDefault(review, 0);
sourceData.put("review", mappedReview);
})
.log("-> Transforming review for hotel '${body[hotel_name]}'")
.setBody()
.simple("INSERT INTO Target (id, hotel_name, price, review) VALUES(${body[id]}, '${body[hotel_name]}', ${body[price]}, ${body[review]})")
.to("jdbc:target_db")
.log("-> Loading transformed data in target database");
}
}
Loading

0 comments on commit 1d47146

Please sign in to comment.