Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "how to write dev services" documentation #42535

Merged
merged 1 commit into from
Aug 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/cache-redis-reference.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ include::{includes}/extension-status.adoc[]

When using Redis as the backend for Quarkus cache, each cached item will be stored in Redis:

- The backend uses the _<default>_ Redis client (if not configured otherwise), so make sure it's configured (or use the xref:redis-dev-services.adoc[redis dev service])
- The backend uses the _<default>_ Redis client (if not configured otherwise), so make sure it's configured (or use the xref:redis-dev-services.adoc[Redis Dev Service])
- the Redis key is built as follows: `cache:$cache-name:$cache-key`, where `cache-key` is the key the application uses.
- the value is encoded to JSON if needed

Expand Down
4 changes: 2 additions & 2 deletions docs/src/main/asciidoc/databases-dev-services.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ quarkus.datasource.db-kind=mysql
quarkus.datasource.devservices.volumes."/local/test/data"=/var/lib/mysql
----

When starting Dev Services (for example, in tests or in dev mode), you will see that the folder "/local/test/data" will be created at your file sytem and that will contain all the database data. When rerunning again the same dev services, this data will contain all the data you might have created beforehand.
When starting Dev Services (for example, in tests or in dev mode), you will see that the folder "/local/test/data" will be created at your file sytem and that will contain all the database data. When rerunning again the same Dev Services, this data will contain all the data you might have created beforehand.

[IMPORTANT]
====
Expand All @@ -190,7 +190,7 @@ Overriding the MariaDB/MySQL configuration would be done as follows:
quarkus.datasource.devservices.container-properties.TC_MY_CNF=testcontainers/mysql-conf
----

This support is database specific and needs to be implemented in each dev service specifically.
This support is database specific and needs to be implemented in each Dev Service specifically.

== Connect To Database Run as a Dev Service

Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/dev-services.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
= Dev Services Overview
include::_attributes.adoc[]
:categories: core
:summary: An introduction to dev services and a list of all extensions that support Dev Services and their configuration options.
:summary: An introduction to Dev Services and a list of all extensions that support Dev Services and their configuration options.
:topics: dev-services,dev-mode,testing

== What Are Dev Services?
Expand Down
123 changes: 123 additions & 0 deletions docs/src/main/asciidoc/extension-writing-dev-service.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
////
This document is maintained in the main Quarkus repository
and pull requests should be submitted there:
https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
////
[id="extension-writing-dev-service"]
= Writing a Dev Service
include::_attributes.adoc[]
:categories: writing-extensions
:diataxis-type: howto
:topics: extensions
////
////


== Prerequisites

- You should already have an xref:building-my-first-extension.adoc[extension structure] in place
- You should have a containerised version of your external service (not all Dev Services rely on containers, but most do)

== Creating a Dev Service

If your extension provides APIs for connecting to an external service, it's a good idea to provide a xref:dev-services.adoc[Dev Service] implementation.

To create a Dev Service, add a new build step into the extension processor class that returns a `DevServicesResultBuildItem`.
Here, the link:https://hub.docker.com/_/hello-world`hello-world` image is used, but you should set up the right image for your service.

[source%nowrap,java]
----
@BuildStep(onlyIfNot = IsNormal.class, onlyIf = GlobalDevServicesConfig.Enabled.class) {
public DevServicesResultBuildItem createContainer() {
DockerImageName dockerImageName = DockerImageName.parse("hello-world");
GenericContainer container = new GenericContainer<>(dockerImageName)
.withExposedPorts(SERVICE_PORT, OTHER_SERVICE_PORT)
.waitingFor(Wait.forLogMessage(".*" + "Started" + ".*", 1))
.withReuse(true);
container.start();
String newUrl = "http://" + container.getHost() + ":" + container.getMappedPort(SERVICE_PORT);
Map<String, String> configOverrides = Map.of("some-service.base-url", newUrl);
return new DevServicesResultBuildItem.RunningDevService(FEATURE, container.getContainerId(),
container::close, configOverrides)
.toBuildItem();
}
----

With this code, you should be able to see your container starting if you add your extension to a test application and run `quarkus dev`.
However, the application will not be able to connect to it, because no ports are exposed. To expose ports, add `withExposedPorts` to the container construction.
For example,

[source%nowrap,java]
----
GenericContainer container = new GenericContainer<>(dockerImageName)
.withExposedPorts(SERVICE_PORT, OTHER_SERVICE_PORT);
----

Testcontainers will map these ports to random ports on the host. This avoids port conflicts, but presents a new problem – how do applications connect to the service in the container?

To allow applications to connect, the extension should override the default configuration for the service with the mapped ports.
This must be done after starting the container.
For example,
Comment on lines +61 to +63
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this is not as simple when dealing with the shared network config. But this thing is complex and a bit in motion as I'm not very satisfied with what we have so probably better to keep it simple for now.

Copy link
Contributor Author

@holly-cummins holly-cummins Aug 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to say, I looked at a lot of our core dev services as examples and the complexity was a bit overwhelming. I had a hard time deciding where the right balance was between a naive 4-line dev service like the ones we live-code in workshops, and the 60-LoC monsters we have for a lot of our services. We don't want to give people instructions that don't work, but we also don't want to make people decide that writing a dev service is the last thing they want to attempt. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think we need to find a way to hide the complexity better.

I'm not entirely sure it's going to be easy but we really need to.


[source%nowrap,java]
----
container.start();
Map<String, String> configOverrides = Map.of("some-service.base-url",
"http://" + container.getHost() + ":" + container.getMappedPort(SERVICE_PORT));
----

Other configuration overrides may be included in the same map.

== Waiting for the container to start

You should add a `.waitingFor` call to the container construction, to wait for the container to start. For example

[source%nowrap,java]
----
.waitingFor(Wait.forLogMessage(".*" + "Started" + ".*", 1))
----

Waiting for a port to be open is another option. See the link:https://java.testcontainers.org/features/startup_and_waits/[Testcontainers documentation] for a full discussion of wait strategies.

== Configuring the Dev Service

To configure the Dev Service launch process, your build step can accept a `ConfigPhase.BUILD_TIME` config class in its constructor.
For example,

[source%nowrap,java]
----
@BuildStep(onlyIfNot = IsNormal.class, onlyIf = GlobalDevServicesConfig.Enabled.class) {
public DevServicesResultBuildItem createContainer(MyConfig config) {
----

You may wish to use this config to set a fixed port, or set an image name, for example.

[source%nowrap,java]
----
if (config.port.isPresent()) {
container.setPortBindings(List.of(config.port.get() + ":" + SERVICE_PORT));
}
----

== Controlling re-use

In dev mode, with live reload, Quarkus may restart frequently. By default, this will also restart test containers.
Quarkus restarts are usually very fast, but containers may take much longer to restart.
To prevent containers restarting on every code change, you can mark the container as reusable:

[source%nowrap,java]
----
.withReuse(true)
----

Some Dev Services implement sophisticated reuse logic in which they track the state of the container in the processor itself.
You may need this if your service has more complex requirements, or needs sharing across instances.


== References

- xref:dev-services.adoc[Dev services overview]
- xref:writing-extensions.adoc[Guide to writing extensions]
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/getting-started-dev-services.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ But what about production? You won't want to use Dev Services in production.
In fact, Quarkus only starts Dev Services in dev and test modes.

Wouldn't it be nice to configure an external database,
but have it *only* used in production, so you could still use dev services the rest of the time?
but have it *only* used in production, so you could still use Dev Services the rest of the time?

Add a `%prod.`
prefix to the database configuration. This means the configuration
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/getting-started.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -489,10 +489,10 @@

== What's next?

This guide covered the creation of an application using Quarkus.

Check warning on line 492 in docs/src/main/asciidoc/getting-started.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.TermsSuggestions] Depending on the context, consider using 'by using' or 'that uses' rather than 'using'. Raw Output: {"message": "[Quarkus.TermsSuggestions] Depending on the context, consider using 'by using' or 'that uses' rather than 'using'.", "location": {"path": "docs/src/main/asciidoc/getting-started.adoc", "range": {"start": {"line": 492, "column": 4}}}, "severity": "INFO"}

Check warning on line 492 in docs/src/main/asciidoc/getting-started.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Fluff] Depending on the context, consider using 'using more direct instructions' rather than 'recommend'. Raw Output: {"message": "[Quarkus.Fluff] Depending on the context, consider using 'using more direct instructions' rather than 'recommend'.", "location": {"path": "docs/src/main/asciidoc/getting-started.adoc", "range": {"start": {"line": 492, "column": 52}}}, "severity": "INFO"}
However, there is much more.
We recommend continuing the journey by creating xref:getting-started-dev-services.adoc[your second Quarkus application], with dev services and persistence.
We recommend continuing the journey by creating xref:getting-started-dev-services.adoc[your second Quarkus application], with Dev Services and persistence.
You can learn about creating a native executable and packaging it in a container with the xref:building-native-image.adoc[building a native executable guide].

Check warning on line 495 in docs/src/main/asciidoc/getting-started.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Fluff] Depending on the context, consider using 'using more direct instructions' rather than 'recommend'. Raw Output: {"message": "[Quarkus.Fluff] Depending on the context, consider using 'using more direct instructions' rather than 'recommend'.", "location": {"path": "docs/src/main/asciidoc/getting-started.adoc", "range": {"start": {"line": 495, "column": 152}}}, "severity": "INFO"}
If you are interested in reactive, we recommend the xref:getting-started-reactive.adoc[getting started with reactive guide], where you can see how to implement reactive applications with Quarkus.

In addition, the xref:tooling.adoc[tooling guide] document explains how to:
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/infinispan-dev-services.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@

== Multiple Dev Services for named connections
The Infinispan Client extension supports connecting to more than one Infinispan Cluster with
the named connections. If you need to spin an additional dev service for a connection name, configure
the named connections. If you need to spin an additional Dev Service for a connection name, configure

Check warning on line 111 in docs/src/main/asciidoc/infinispan-dev-services.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Fluff] Depending on the context, consider using 'Rewrite the sentence, or use 'must', instead of' rather than 'need to'. Raw Output: {"message": "[Quarkus.Fluff] Depending on the context, consider using 'Rewrite the sentence, or use 'must', instead of' rather than 'need to'.", "location": {"path": "docs/src/main/asciidoc/infinispan-dev-services.adoc", "range": {"start": {"line": 111, "column": 31}}}, "severity": "INFO"}
at least on property in the application properties:

[source,properties]
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/kafka-dev-services.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Dev Services for Kafka supports https://redpanda.com[Redpanda], https://github/o
and https://strimzi.io[Strimzi] (in https://github.com/apache/kafka/blob/trunk/config/kraft/README.md[Kraft] mode) images.

**Redpanda** is a Kafka compatible event streaming platform.
Because it provides a fast startup times, dev services defaults to Redpanda images from `vectorized/redpanda`.
Because it provides a fast startup times, Dev Services defaults to Redpanda images from `vectorized/redpanda`.
You can select any version from https://hub.docker.com/r/vectorized/redpanda.

**kafka-native** provides images of standard Apache Kafka distribution compiled to native binary using Quarkus and GraalVM.
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/kafka-schema-registry-avro.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -620,7 +620,7 @@ The `quarkus-apicurio-registry-avro` extension depends on recent versions of Api
and most versions of Apicurio Registry server and client are backwards compatible.
For some you need to make sure that the client used by Serdes is compatible with the server.

For example, with Apicurio dev service if you set the image name to use version `2.1.5.Final`:
For example, with Apicurio Dev Service if you set the image name to use version `2.1.5.Final`:

[source,properties]
----
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -648,7 +648,7 @@ The `quarkus-apicurio-registry-json-schema` extension depends on recent versions
and most versions of Apicurio Registry server and client are backwards compatible.
For some you need to make sure that the client used by Serdes is compatible with the server.

For example, with Apicurio dev service if you set the image name to use version `2.1.5.Final`:
For example, with Apicurio Dev Service if you set the image name to use version `2.1.5.Final`:

[source,properties]
----
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/observability-devservices.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@
:topics: observability,grafana,lgtm,prometheus,victoriametrics,jaeger,otel,otlp
:extensions: io.quarkus:quarkus-observability-devservices

We are already familiar with xref:dev-services.adoc[Dev Service] concept, but in the case of Observability we need a way to orchestrate and connect more than a single dev service, usually a whole stack of them; e.g. a metrics agent periodically scraping application for metrics, pushing them into timeseries database, and Grafana feeding graphs of this timeseries data.
We are already familiar with xref:dev-services.adoc[Dev Service] concept, but in the case of Observability we need a way to orchestrate and connect more than a single Dev Service, usually a whole stack of them; e.g. a metrics agent periodically scraping application for metrics, pushing them into timeseries database, and Grafana feeding graphs of this timeseries data.

Check warning on line 13 in docs/src/main/asciidoc/observability-devservices.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer. Raw Output: {"message": "[Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer.", "location": {"path": "docs/src/main/asciidoc/observability-devservices.adoc", "range": {"start": {"line": 13, "column": 1}}}, "severity": "INFO"}

Check warning on line 13 in docs/src/main/asciidoc/observability-devservices.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.TermsWarnings] Consider using 'for example' rather than 'e.g.' unless updating existing content that uses the term. Raw Output: {"message": "[Quarkus.TermsWarnings] Consider using 'for example' rather than 'e.g.' unless updating existing content that uses the term.", "location": {"path": "docs/src/main/asciidoc/observability-devservices.adoc", "range": {"start": {"line": 13, "column": 212}}}, "severity": "WARNING"}

Check warning on line 13 in docs/src/main/asciidoc/observability-devservices.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Spelling] Use correct American English spelling. Did you really mean 'timeseries'? Raw Output: {"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'timeseries'?", "location": {"path": "docs/src/main/asciidoc/observability-devservices.adoc", "range": {"start": {"line": 13, "column": 298}}}, "severity": "WARNING"}

Check warning on line 13 in docs/src/main/asciidoc/observability-devservices.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Spelling] Use correct American English spelling. Did you really mean 'timeseries'? Raw Output: {"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'timeseries'?", "location": {"path": "docs/src/main/asciidoc/observability-devservices.adoc", "range": {"start": {"line": 13, "column": 354}}}, "severity": "WARNING"}

With this in mind, we added a new concept of Dev Resource, an adapter between Dev Service concept and https://testcontainers.com/[Testcontainers]. And since we now have fine-grained services - with the Dev Resource per container, we can take this even further, allowing the user to choose the way to use this new Dev Resource concept:

Check warning on line 15 in docs/src/main/asciidoc/observability-devservices.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer. Raw Output: {"message": "[Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer.", "location": {"path": "docs/src/main/asciidoc/observability-devservices.adoc", "range": {"start": {"line": 15, "column": 148}}}, "severity": "INFO"}

NOTE: Each Dev Resource implementation is an `@QuarkusTestResourceLifecycleManager` implementation as well

Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/opentelemetry-metrics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,9 @@

First we need to start a system to visualise the OpenTelemetry data.

=== See the data

Check warning on line 154 in docs/src/main/asciidoc/opentelemetry-metrics.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale

[vale] reported by reviewdog 🐶 [Quarkus.Headings] Use sentence-style capitalization in 'Grafana-OTel-LGTM Dev Service'. Raw Output: {"message": "[Quarkus.Headings] Use sentence-style capitalization in 'Grafana-OTel-LGTM Dev Service'.", "location": {"path": "docs/src/main/asciidoc/opentelemetry-metrics.adoc", "range": {"start": {"line": 154, "column": 13}}}, "severity": "INFO"}

==== Grafana-OTel-LGTM dev service
==== Grafana-OTel-LGTM Dev Service
You can use the xref:observability-devservices-lgtm.adoc[Grafana-OTel-LGTM] devservice.

This Dev service includes a Grafana for visualizing data, Loki to store logs, Tempo to store traces and Prometheus to store metrics.
Expand Down
6 changes: 6 additions & 0 deletions docs/src/main/asciidoc/writing-extensions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -1978,6 +1978,12 @@ in your runtime module, and add a `META-INF/services/io.quarkus.dev.spi.HotRepla
On startup the `setupHotDeployment` method will be called, and you can use the provided `io.quarkus.dev.spi.HotReplacementContext`
to initiate a scan for changed files.

==== Dev Services

Where extensions use an external service, adding a Dev Service can improve the user experience in development and test modes.
See xref:extension-writing-dev-service.adoc[how to write a Dev Service] for more details.


=== Testing Extensions

Testing of Quarkus extensions should be done with the `io.quarkus.test.QuarkusUnitTest` JUnit 5 extension.
Expand Down
Loading