From 1fcafea2a5930e5a34ec78ddc43571461a150d9f Mon Sep 17 00:00:00 2001 From: Jilles van Gurp Date: Thu, 28 Nov 2024 14:10:15 +0100 Subject: [PATCH 1/3] tweak readme --- .../test/kotlin/documentation/projectreadme/readme-intro.md | 6 +++--- .../test/kotlin/documentation/projectreadme/readme-outro.md | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/src/test/kotlin/documentation/projectreadme/readme-intro.md b/docs/src/test/kotlin/documentation/projectreadme/readme-intro.md index 30284d50..f354b21a 100644 --- a/docs/src/test/kotlin/documentation/projectreadme/readme-intro.md +++ b/docs/src/test/kotlin/documentation/projectreadme/readme-intro.md @@ -1,19 +1,19 @@ ## Why Kt-search? -If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or wasm and using that to run lambda functions. +If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or WASM and using that to run lambda functions. Kt-search has you covered for all of those. The official Elastic or Opensearch clients are Java clients. You can use them from Kotlin but only on the JVM. And they are not source compatible with each other. The Opensearch client is based on a fork of the old Java client which after the fork was deprecated. On top of that, it uses opensearch specific package names. Kt-search solves a few important problems here: - It's Kotlin! You don't have to deal with all the Java idiomatic stuff that comes with the three Java libraries. You can write pure Kotlin code, use co-routines, and use Kotlin DSLs for everything. Simpler code, easier to debug, etc. -- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Wasm support is coming soon and there is also native and mobile support. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those. +- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Targets for native, IOS, WASM, etc. are also there. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those. - It doesn't force you to choose between Elasticsearch or Opensearch. Some features are specific to those products and will only work for those platforms but most of the baseline functionality is exactly the same for both. - It's future proof. Everything is extensible (DSLs) and modular. Even supporting custom plugins that add new features is pretty easy with the `json-dsl` library that is part of kt-search. ## License -This project is [licensed](LICENSE) under the MIT license. +This project is [licensed](LICENSE) under the MIT license and will always be. ## Learn more diff --git a/docs/src/test/kotlin/documentation/projectreadme/readme-outro.md b/docs/src/test/kotlin/documentation/projectreadme/readme-outro.md index 947da014..8afcb803 100644 --- a/docs/src/test/kotlin/documentation/projectreadme/readme-outro.md +++ b/docs/src/test/kotlin/documentation/projectreadme/readme-outro.md @@ -56,19 +56,20 @@ The search client module is the main module of this library. I extracted the jso ## Contributing -Pull requests are very welcome! Please consider communicating your intentions in advance to avoid conflicts, or redundant work. +Pull requests are very welcome! This project runs on community contributions. If you don't like something, suggest changes. Is a feature you need missing from the DSL? Add it. To avoid conflicts or double work, please reach out via the issue tracker for bigger things. I try to be as responsive as I can Some suggestions of things you could work on: - Extend the mapping or query DSLs. Our goal is to have coverage of all the common things we and other users need. The extensibility of `JsonDsl` always gives you the option to add whatever is not directly supported by manipulating the underlying map. But creating extension functions that do this properly is not har. - Add more API support for things in Opensearch/Elasticsearch that are not currently supported. The REST api has dozens of end point other than search. Like the DSL, adding extension functions is easy and using the underlying rest client allows you to customize any requests. - Work on one of the issues or suggest some new ones. +- Refine the documentation. Add examples. Document missing things. ## Support and Community Please file issues if you find any or have any suggestions for changes. -Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, trainings, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your search setup and your product roadmap. +Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, training, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your Elasticsearch/Opensearch setup and your product roadmap. The best way to reach me is via email if you wish to use my services professionally. Please refer to my [website](https://www.jillesvangurp.com) for that. From 4d1989f0814faa72245511950b0b77a40de574b6 Mon Sep 17 00:00:00 2001 From: Jilles van Gurp Date: Thu, 28 Nov 2024 14:10:34 +0100 Subject: [PATCH 2/3] add some features to the Terms aggregation --- .../querydsl/aggregation-queries.kt | 86 ++++++++++++++++--- 1 file changed, 73 insertions(+), 13 deletions(-) diff --git a/search-dsls/src/commonMain/kotlin/com/jillesvangurp/searchdsls/querydsl/aggregation-queries.kt b/search-dsls/src/commonMain/kotlin/com/jillesvangurp/searchdsls/querydsl/aggregation-queries.kt index 02cb5746..94beb9a8 100644 --- a/search-dsls/src/commonMain/kotlin/com/jillesvangurp/searchdsls/querydsl/aggregation-queries.kt +++ b/search-dsls/src/commonMain/kotlin/com/jillesvangurp/searchdsls/querydsl/aggregation-queries.kt @@ -3,9 +3,11 @@ package com.jillesvangurp.searchdsls.querydsl import com.jillesvangurp.jsondsl.JsonDsl +import com.jillesvangurp.jsondsl.withJsonDsl import kotlin.reflect.KProperty open class AggQuery(name: String) : ESQuery(name) + inline fun JsonDsl.add(name: String, aggQuery: T) { this[name] = aggQuery } @@ -22,18 +24,21 @@ private fun AggQuery.aggs(): JsonDsl { } } -fun SearchDSL.agg(name: String, aggQuery: AggQuery, block: (AggQuery.() -> Unit)?=null) { +fun SearchDSL.agg(name: String, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) { aggs().add(name, aggQuery) block?.invoke(aggQuery) } -fun AggQuery.agg(name: String, aggQuery: AggQuery,block: (AggQuery.() -> Unit)?=null) { +fun AggQuery.agg(name: String, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) { aggs().add(name, aggQuery) block?.invoke(aggQuery) } -fun SearchDSL.agg(name: Enum<*>, aggQuery: AggQuery,block: (AggQuery.() -> Unit)?=null) = agg(name.name, aggQuery,block) -fun AggQuery.agg(name: Enum<*>, aggQuery: AggQuery,block: (AggQuery.() -> Unit)?=null) = agg(name.name, aggQuery,block) +fun SearchDSL.agg(name: Enum<*>, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) = + agg(name.name, aggQuery, block) + +fun AggQuery.agg(name: Enum<*>, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) = + agg(name.name, aggQuery, block) class TermsAggConfig : JsonDsl() { var field by property() @@ -41,6 +46,55 @@ class TermsAggConfig : JsonDsl() { var minDocCount by property("min_doc_count") var shardSize by property("shard_size") var showTermDocCountError by property("show_term_doc_count_error") + + /** include values by regex or exact valye; use the includePartition function for partitions. */ + var include by property>() + + /** exclude values by regex or exact valye */ + var exclude by property>() + + /** + * Partitions the keys into the specified [numPartitions] and only returns keys falling in [partition]. + * Use for large terms aggregations to get results with multiple aggregation requests to avoid stressing with huge + * responses. + * + * Note, cannot be used with an `exclude` clause. + * + * @see https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values_with_partitions + * Elasticsearch 8.x only + */ + fun includePartition(numPartitions: Int, partition: Int) { + this["include"] = withJsonDsl { + this["num_partitions"] = numPartitions + this["partition"] = partition + } + } + + + fun orderByKey(direction: SortOrder) { + getOrCreateMutableList("order").add( + withJsonDsl { + this["_key"] = direction.name.lowercase() + } + ) + } + + fun orderByField(field: String, direction: SortOrder) { + getOrCreateMutableList("order").add( + withJsonDsl { + this[field] = direction.name.lowercase() + } + ) + } + + fun orderByField(field: KProperty<*>, direction: SortOrder) { + getOrCreateMutableList("order").add( + withJsonDsl { + this[field.name] = direction.name.lowercase() + } + ) + } + } class TermsAgg(val field: String, block: (TermsAggConfig.() -> Unit)? = null) : AggQuery("terms") { @@ -54,19 +108,21 @@ class TermsAgg(val field: String, block: (TermsAggConfig.() -> Unit)? = null) : } } -class AggRange: JsonDsl() { +class AggRange : JsonDsl() { var key by property() + /** * Aggregation includes the `from` value */ var from by property() + /** * Aggregation excludes the `to` value */ var to by property() companion object { - fun create(block: AggRange.() -> Unit) = AggRange().apply(block) + fun create(block: AggRange.() -> Unit) = AggRange().apply(block) } } @@ -86,15 +142,17 @@ class RangesAgg(val field: String, block: (RangesAggConfig.() -> Unit)? = null) } } -class AggDateRange: JsonDsl() { +class AggDateRange : JsonDsl() { /** * Customizes the key for each range */ var key by property() + /** * Aggregation includes the `from` value */ var from by property() + /** * Aggregation excludes the `to` value */ @@ -286,6 +344,7 @@ class TopHitsAggConfig : JsonDsl() { this["sort"] = builder.sortFields } } + class TopHitsAgg(block: (TopHitsAggConfig.() -> Unit)? = null) : AggQuery("top_hits") { init { val config = TopHitsAggConfig() @@ -306,24 +365,25 @@ class FilterConfig : JsonDsl() { set(value) { this["query"] = value.wrapWithName() } - fun namedFilter(name: String, query:ESQuery) { - val filters = this["filters"]?.let { it as JsonDsl} ?: JsonDsl() + + fun namedFilter(name: String, query: ESQuery) { + val filters = this["filters"]?.let { it as JsonDsl } ?: JsonDsl() this["filters"] = filters - filters[name]=query.wrapWithName() + filters[name] = query.wrapWithName() } } class FiltersAgg( - block: FilterConfig.()->Unit + block: FilterConfig.() -> Unit ) : AggQuery("filters") { init { this[name] = FilterConfig().apply(block) } } -class FilterAgg(filter: ESQuery): AggQuery("filter") { +class FilterAgg(filter: ESQuery) : AggQuery("filter") { init { - this[name]=filter.wrapWithName() + this[name] = filter.wrapWithName() } } From ae0fa201faa07fa5b864b7b02ccde50b76ee4fac Mon Sep 17 00:00:00 2001 From: Jilles van Gurp Date: Thu, 28 Nov 2024 14:11:58 +0100 Subject: [PATCH 3/3] regenerate README --- README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index e3d02498..fac6aabb 100644 --- a/README.md +++ b/README.md @@ -6,20 +6,20 @@ Kt-search is a Kotlin Multi Platform library to search across the Opensearch and ## Why Kt-search? -If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or wasm and using that to run lambda functions. +If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or WASM and using that to run lambda functions. Kt-search has you covered for all of those. The official Elastic or Opensearch clients are Java clients. You can use them from Kotlin but only on the JVM. And they are not source compatible with each other. The Opensearch client is based on a fork of the old Java client which after the fork was deprecated. On top of that, it uses opensearch specific package names. Kt-search solves a few important problems here: - It's Kotlin! You don't have to deal with all the Java idiomatic stuff that comes with the three Java libraries. You can write pure Kotlin code, use co-routines, and use Kotlin DSLs for everything. Simpler code, easier to debug, etc. -- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Wasm support is coming soon and there is also native and mobile support. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those. +- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Targets for native, IOS, WASM, etc. are also there. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those. - It doesn't force you to choose between Elasticsearch or Opensearch. Some features are specific to those products and will only work for those platforms but most of the baseline functionality is exactly the same for both. - It's future proof. Everything is extensible (DSLs) and modular. Even supporting custom plugins that add new features is pretty easy with the `json-dsl` library that is part of kt-search. ## License -This project is [licensed](LICENSE) under the MIT license. +This project is [licensed](LICENSE) under the MIT license and will always be. ## Learn more @@ -443,19 +443,20 @@ The search client module is the main module of this library. I extracted the jso ## Contributing -Pull requests are very welcome! Please consider communicating your intentions in advance to avoid conflicts, or redundant work. +Pull requests are very welcome! This project runs on community contributions. If you don't like something, suggest changes. Is a feature you need missing from the DSL? Add it. To avoid conflicts or double work, please reach out via the issue tracker for bigger things. I try to be as responsive as I can Some suggestions of things you could work on: - Extend the mapping or query DSLs. Our goal is to have coverage of all the common things we and other users need. The extensibility of `JsonDsl` always gives you the option to add whatever is not directly supported by manipulating the underlying map. But creating extension functions that do this properly is not har. - Add more API support for things in Opensearch/Elasticsearch that are not currently supported. The REST api has dozens of end point other than search. Like the DSL, adding extension functions is easy and using the underlying rest client allows you to customize any requests. - Work on one of the issues or suggest some new ones. +- Refine the documentation. Add examples. Document missing things. ## Support and Community Please file issues if you find any or have any suggestions for changes. -Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, trainings, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your search setup and your product roadmap. +Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, training, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your Elasticsearch/Opensearch setup and your product roadmap. The best way to reach me is via email if you wish to use my services professionally. Please refer to my [website](https://www.jillesvangurp.com) for that.