Merge pull request #156 from jillesvangurp/improve-terms-agg

improve terms agg

README.md

@@ -6,20 +6,20 @@ Kt-search is a Kotlin Multi Platform library to search across the Opensearch and
 
 ## Why Kt-search?
 
-If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or wasm and using that to run lambda functions. 
+If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or WASM and using that to run lambda functions. 
 
 Kt-search has you covered for all of those. The official Elastic or Opensearch clients are Java clients. You can use them from Kotlin but only on the JVM. And they are not source compatible with each other. The Opensearch client is based on a fork of the old Java client which after the fork was deprecated. On top of that, it uses opensearch specific package names.
 
 Kt-search solves a few important problems here:
 
 - It's Kotlin! You don't have to deal with all the Java idiomatic stuff that comes with the three Java libraries. You can write pure Kotlin code, use co-routines, and use Kotlin DSLs for everything. Simpler code, easier to debug, etc.
-- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Wasm support is coming soon and there is also native and mobile support. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those.
+- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Targets for native, IOS, WASM, etc. are also there. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those.
 - It doesn't force you to choose between Elasticsearch or Opensearch. Some features are specific to those products and will only work for those platforms but most of the baseline functionality is exactly the same for both.
 - It's future proof. Everything is extensible (DSLs) and modular. Even supporting custom plugins that add new features is pretty easy with the `json-dsl` library that is part of kt-search.
 
 ## License
 
-This project is [licensed](LICENSE) under the MIT license.
+This project is [licensed](LICENSE) under the MIT license and will always be. 
 
 ## Learn more
 
@@ -443,19 +443,20 @@ The search client module is the main module of this library. I extracted the jso
 
 ## Contributing
 
-Pull requests are very welcome! Please consider communicating your intentions in advance to avoid conflicts, or redundant work.
+Pull requests are very welcome! This project runs on community contributions. If you don't like something, suggest changes. Is a feature you need missing from the DSL? Add it. To avoid conflicts or double work, please reach out via the issue tracker for bigger things. I try to be as responsive as I can
 
 Some suggestions of things you could work on:
 
 - Extend the mapping or query DSLs. Our goal is to have coverage of all the common things we and other users need. The extensibility of `JsonDsl` always gives you the option to add whatever is not directly supported by manipulating the underlying map. But creating extension functions that do this properly is not har.
 - Add more API support for things in Opensearch/Elasticsearch that are not currently supported. The REST api has dozens of end point other than search. Like the DSL, adding extension functions is easy and using the underlying rest client allows you to customize any requests.
 - Work on one of the issues or suggest some new ones.
+- Refine the documentation. Add examples. Document missing things.
 
 ## Support and Community
 
 Please file issues if you find any or have any suggestions for changes.
 
-Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, trainings, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your search setup and your product roadmap.
+Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, training, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your Elasticsearch/Opensearch setup and your product roadmap.
 
 The best way to reach me is via email if you wish to use my services professionally. Please refer to my [website](https://www.jillesvangurp.com) for that.
 

docs/src/test/kotlin/documentation/projectreadme/readme-intro.md

@@ -1,19 +1,19 @@
 ## Why Kt-search?
 
-If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or wasm and using that to run lambda functions. 
+If you develop software in Kotlin and would like to use Opensearch or Elasticsearch, you have a few choices to make. There are multiple clients to choose from and not all of them work for each version. And then there is Kotlin multi platform to consider as well. Maybe you are running spring boot on the jvm. Or maybe you are using ktor compiled to native or WASM and using that to run lambda functions. 
 
 Kt-search has you covered for all of those. The official Elastic or Opensearch clients are Java clients. You can use them from Kotlin but only on the JVM. And they are not source compatible with each other. The Opensearch client is based on a fork of the old Java client which after the fork was deprecated. On top of that, it uses opensearch specific package names.
 
 Kt-search solves a few important problems here:
 
 - It's Kotlin! You don't have to deal with all the Java idiomatic stuff that comes with the three Java libraries. You can write pure Kotlin code, use co-routines, and use Kotlin DSLs for everything. Simpler code, easier to debug, etc.
-- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Wasm support is coming soon and there is also native and mobile support. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those.
+- It's a multiplatform library. We use it on the jvm and in the browser (javascript). Targets for native, IOS, WASM, etc. are also there. So, your Kotlin code should be extremely portable. So, whether you are doing backend development, doing lambda functions, command line tools, mobile apps, or web apps, you can embed kt-search in each of those.
 - It doesn't force you to choose between Elasticsearch or Opensearch. Some features are specific to those products and will only work for those platforms but most of the baseline functionality is exactly the same for both.
 - It's future proof. Everything is extensible (DSLs) and modular. Even supporting custom plugins that add new features is pretty easy with the `json-dsl` library that is part of kt-search.
 
 ## License
 
-This project is [licensed](LICENSE) under the MIT license.
+This project is [licensed](LICENSE) under the MIT license and will always be. 
 
 ## Learn more
 

docs/src/test/kotlin/documentation/projectreadme/readme-outro.md

@@ -56,19 +56,20 @@ The search client module is the main module of this library. I extracted the jso
 
 ## Contributing
 
-Pull requests are very welcome! Please consider communicating your intentions in advance to avoid conflicts, or redundant work.
+Pull requests are very welcome! This project runs on community contributions. If you don't like something, suggest changes. Is a feature you need missing from the DSL? Add it. To avoid conflicts or double work, please reach out via the issue tracker for bigger things. I try to be as responsive as I can
 
 Some suggestions of things you could work on:
 
 - Extend the mapping or query DSLs. Our goal is to have coverage of all the common things we and other users need. The extensibility of `JsonDsl` always gives you the option to add whatever is not directly supported by manipulating the underlying map. But creating extension functions that do this properly is not har.
 - Add more API support for things in Opensearch/Elasticsearch that are not currently supported. The REST api has dozens of end point other than search. Like the DSL, adding extension functions is easy and using the underlying rest client allows you to customize any requests.
 - Work on one of the issues or suggest some new ones.
+- Refine the documentation. Add examples. Document missing things.
 
 ## Support and Community
 
 Please file issues if you find any or have any suggestions for changes.
 
-Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, trainings, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your search setup and your product roadmap.
+Within reason, I can help with simple issues. Beyond that, I offer my services as a consultant as well if you need some more help with getting started or just using Elasticsearch/Opensearch in general with just about any tech stack. I can help with discovery projects, training, architecture analysis, query and mapping optimizations, or just generally help you get the most out of your Elasticsearch/Opensearch setup and your product roadmap.
 
 The best way to reach me is via email if you wish to use my services professionally. Please refer to my [website](https://www.jillesvangurp.com) for that.
 

search-dsls/src/commonMain/kotlin/com/jillesvangurp/searchdsls/querydsl/aggregation-queries.kt

@@ -3,9 +3,11 @@
 package com.jillesvangurp.searchdsls.querydsl
 
 import com.jillesvangurp.jsondsl.JsonDsl
+import com.jillesvangurp.jsondsl.withJsonDsl
 import kotlin.reflect.KProperty
 
 open class AggQuery(name: String) : ESQuery(name)
+
 inline fun <reified T : AggQuery> JsonDsl.add(name: String, aggQuery: T) {
     this[name] = aggQuery
 }
@@ -22,25 +24,77 @@ private fun AggQuery.aggs(): JsonDsl {
     }
 }
 
-fun SearchDSL.agg(name: String, aggQuery: AggQuery, block: (AggQuery.() -> Unit)?=null) {
+fun SearchDSL.agg(name: String, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) {
     aggs().add(name, aggQuery)
     block?.invoke(aggQuery)
 }
 
-fun AggQuery.agg(name: String, aggQuery: AggQuery,block: (AggQuery.() -> Unit)?=null) {
+fun AggQuery.agg(name: String, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) {
     aggs().add(name, aggQuery)
     block?.invoke(aggQuery)
 }
 
-fun SearchDSL.agg(name: Enum<*>, aggQuery: AggQuery,block: (AggQuery.() -> Unit)?=null) = agg(name.name, aggQuery,block)
-fun AggQuery.agg(name: Enum<*>, aggQuery: AggQuery,block: (AggQuery.() -> Unit)?=null) = agg(name.name, aggQuery,block)
+fun SearchDSL.agg(name: Enum<*>, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) =
+    agg(name.name, aggQuery, block)
+
+fun AggQuery.agg(name: Enum<*>, aggQuery: AggQuery, block: (AggQuery.() -> Unit)? = null) =
+    agg(name.name, aggQuery, block)
 
 class TermsAggConfig : JsonDsl() {
     var field by property<String>()
     var aggSize by property<Long>("size") // can't redefine Map.size sadly
     var minDocCount by property<Long>("min_doc_count")
     var shardSize by property<Long>("shard_size")
     var showTermDocCountError by property<Long>("show_term_doc_count_error")
+
+    /** include values by regex or exact valye; use the includePartition function for partitions. */
+    var include by property<List<String>>()
+
+    /** exclude values by regex or exact valye */
+    var exclude by property<List<String>>()
+
+    /**
+     * Partitions the keys into the specified [numPartitions] and only returns keys falling in [partition].
+     * Use for large terms aggregations to get results with multiple aggregation requests to avoid stressing with huge
+     * responses.
+     *
+     * Note, cannot be used with an `exclude` clause.
+     *
+     * @see https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values_with_partitions
+     * Elasticsearch 8.x only
+     */
+    fun includePartition(numPartitions: Int, partition: Int) {
+        this["include"] = withJsonDsl {
+            this["num_partitions"] = numPartitions
+            this["partition"] = partition
+        }
+    }
+
+
+    fun orderByKey(direction: SortOrder) {
+        getOrCreateMutableList("order").add(
+            withJsonDsl {
+                this["_key"] = direction.name.lowercase()
+            }
+        )
+    }
+
+    fun orderByField(field: String, direction: SortOrder) {
+        getOrCreateMutableList("order").add(
+            withJsonDsl {
+                this[field] = direction.name.lowercase()
+            }
+        )
+    }
+
+    fun orderByField(field: KProperty<*>, direction: SortOrder) {
+        getOrCreateMutableList("order").add(
+            withJsonDsl {
+                this[field.name] = direction.name.lowercase()
+            }
+        )
+    }
+
 }
 
 class TermsAgg(val field: String, block: (TermsAggConfig.() -> Unit)? = null) : AggQuery("terms") {
@@ -54,19 +108,21 @@ class TermsAgg(val field: String, block: (TermsAggConfig.() -> Unit)? = null) :
     }
 }
 
-class AggRange: JsonDsl() {
+class AggRange : JsonDsl() {
     var key by property<String>()
+
     /**
      * Aggregation includes the `from` value
      */
     var from by property<Double>()
+
     /**
      * Aggregation excludes the `to` value
      */
     var to by property<Double>()
 
     companion object {
-      fun create(block: AggRange.() -> Unit) = AggRange().apply(block)
+        fun create(block: AggRange.() -> Unit) = AggRange().apply(block)
     }
 }
 
@@ -86,15 +142,17 @@ class RangesAgg(val field: String, block: (RangesAggConfig.() -> Unit)? = null)
     }
 }
 
-class AggDateRange: JsonDsl() {
+class AggDateRange : JsonDsl() {
     /**
      * Customizes the key for each range
      */
     var key by property<String>()
+
     /**
      * Aggregation includes the `from` value
      */
     var from by property<String>()
+
     /**
      * Aggregation excludes the `to` value
      */
@@ -286,6 +344,7 @@ class TopHitsAggConfig : JsonDsl() {
         this["sort"] = builder.sortFields
     }
 }
+
 class TopHitsAgg(block: (TopHitsAggConfig.() -> Unit)? = null) : AggQuery("top_hits") {
     init {
         val config = TopHitsAggConfig()
@@ -306,24 +365,25 @@ class FilterConfig : JsonDsl() {
         set(value) {
             this["query"] = value.wrapWithName()
         }
-    fun namedFilter(name: String, query:ESQuery) {
-        val filters = this["filters"]?.let { it as JsonDsl} ?: JsonDsl()
+
+    fun namedFilter(name: String, query: ESQuery) {
+        val filters = this["filters"]?.let { it as JsonDsl } ?: JsonDsl()
         this["filters"] = filters
-        filters[name]=query.wrapWithName()
+        filters[name] = query.wrapWithName()
     }
 }
 
 class FiltersAgg(
-    block: FilterConfig.()->Unit
+    block: FilterConfig.() -> Unit
 ) : AggQuery("filters") {
     init {
         this[name] = FilterConfig().apply(block)
     }
 }
 
-class FilterAgg(filter: ESQuery): AggQuery("filter") {
+class FilterAgg(filter: ESQuery) : AggQuery("filter") {
     init {
-        this[name]=filter.wrapWithName()
+        this[name] = filter.wrapWithName()
     }
 }