-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
0fac341
commit 50f0f6c
Showing
67 changed files
with
3,964 additions
and
355 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# class Keyword [](#class-Keyword) [](#top) | ||
The `Keyword` class in the `Stretchy::Attributes::Type` module defines a keyword attribute for the model. | ||
|
||
### Options[¶](#class-Keyword-label-Options) [↑](#top) | ||
|
||
The attribute can be refined using a hash of options: | ||
|
||
- `:doc_values` (Boolean, default: `true`): Indicates if the field should be stored on disk in a column-stride fashion. | ||
|
||
- `:eager_global_ordinals` (Boolean, default: `false`): Indicates if global ordinals should be loaded eagerly on refresh. | ||
|
||
- `:fields` (Hash): Defines multi-fields for the same string value to be indexed in multiple ways. | ||
|
||
- `:ignore_above` (Integer, default: `2147483647`): Sets a limit for the length of the string. Strings longer than this limit will not be indexed. | ||
|
||
- `:index` (Boolean, default: `true`): Indicates if the field should be quickly searchable. | ||
|
||
- `:index_options` (String, default: `'docs'`): Indicates what information should be stored in the index for scoring purposes. | ||
|
||
- `:meta` (Hash): Provides metadata about the field. | ||
|
||
- `:norms` (Boolean, default: `false`): Indicates if field-length should be taken into account when scoring queries. | ||
|
||
- `:null_value` (String, default: `null`): Sets a value to be substituted for any explicit null values. | ||
|
||
- `:on_script_error` (String): Defines what to do if the script defined by the `:script` parameter throws an error at indexing time. Can be `'fail'` or `'continue'`. | ||
|
||
- `:script` (String): Defines a script that will index values generated by this script, rather than reading the values directly from the source. | ||
|
||
- `:store` (Boolean, default: `false`): Indicates if the field value should be stored and retrievable separately from the `_source` field. | ||
|
||
- `:similarity` (String, default: `'BM25'`): Sets the scoring algorithm or similarity to be used. | ||
|
||
- `:normalizer` (String, default: `null`): Defines a pre-processor for the keyword prior to indexing. | ||
|
||
- `:split_queries_on_whitespace` (Boolean, default: `false`): Indicates if full text queries should split the input on whitespace. | ||
|
||
- `:time_series_dimension` (Boolean, default: `false`): Indicates if the field is a time series dimension. | ||
|
||
### Example[¶](#class-Keyword-label-Example) [↑](#top) | ||
|
||
``` | ||
classMyModelincludeStretchyModelattribute:tag,:keyword,ignore\_above:256,time\_series\_dimension:trueend | ||
``` | ||
|
||
Returns nothing. | ||
|
||
## Constants | ||
| Name | Description | | ||
| ---- | ----------- | | ||
| **OPTIONS[](#OPTIONS)** | Not documented | | ||
## Public Instance Methods | ||
### type() [](#method-i-type) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# class Object [](#class-Object) [](#top) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,12 +1,193 @@ | ||
# module Stretchy [](#module-Stretchy) [](#top) | ||
Defines a binary attribute for the model. | ||
|
||
<dl class="rdoc-list note-list"> | ||
<dt>name | ||
</dt> | ||
<dd> | ||
<p>The Symbol name of the attribute.</p> | ||
</dd> | ||
<dt>opts | ||
</dt> | ||
<dd> | ||
<p>The Hash options used to refine the attribute (default: {}):</p> | ||
<dl class="rdoc-list note-list"> | ||
<dt>:doc_values | ||
</dt> | ||
<dd> | ||
<p>The Boolean indicating if the field should be stored on disk in a column-stride fashion. This allows it to be used later for sorting, aggregations, or scripting. Defaults to false.</p> | ||
</dd> | ||
<dt>:store | ||
</dt> | ||
<dd> | ||
<p>The Boolean indicating if the field value should be stored and retrievable separately from the _source field. Defaults to false.</p> | ||
</dd> | ||
</dl> | ||
</dd> | ||
</dl> | ||
### Examples[¶](#module-Stretchy-label-Examples) [↑](#top) | ||
|
||
``` | ||
classMyModelincludeStretchyModelattribute:name,:binary,doc\_values:true,store:trueend | ||
``` | ||
|
||
### Returns[¶](#module-Stretchy-label-Returns) [↑](#top) | ||
|
||
Returns nothing. | ||
|
||
Defines a boolean attribute for the model. | ||
|
||
<dl class="rdoc-list note-list"> | ||
<dt>name | ||
</dt> | ||
<dd> | ||
<p>The Symbol name of the attribute.</p> | ||
</dd> | ||
<dt>opts | ||
</dt> | ||
<dd> | ||
<p>The Hash options used to refine the attribute (default: {}):</p> | ||
<dl class="rdoc-list note-list"> | ||
<dt>:doc_values | ||
</dt> | ||
<dd> | ||
<p>The Boolean indicating if the field should be stored on disk in a column-stride fashion. This allows it to be used later for sorting, aggregations, or scripting. Defaults to true.</p> | ||
</dd> | ||
<dt>:index | ||
</dt> | ||
<dd> | ||
<p>The Boolean indicating if the field should be quickly searchable. Defaults to true.</p> | ||
</dd> | ||
<dt>:ignore_malformed | ||
</dt> | ||
<dd> | ||
<p>The Boolean indicating if exceptions thrown when trying to index the wrong data type into a field should be ignored. Defaults to false.</p> | ||
</dd> | ||
<dt>:null_value | ||
</dt> | ||
<dd> | ||
<p>The Boolean value to be substituted for any explicit null values. Defaults to null.</p> | ||
</dd> | ||
<dt>:on_script_error | ||
</dt> | ||
<dd> | ||
<p>The String defining what to do if the script defined by the :script parameter throws an error at indexing time. Can be ‘fail’ or ‘continue’.</p> | ||
</dd> | ||
<dt>:script | ||
</dt> | ||
<dd> | ||
<p>The String script that will index values generated by this script, rather than reading the values directly from the source.</p> | ||
</dd> | ||
<dt>:store | ||
</dt> | ||
<dd> | ||
<p>The Boolean indicating if the field value should be stored and retrievable separately from the _source field. Defaults to false.</p> | ||
</dd> | ||
<dt>:meta | ||
</dt> | ||
<dd> | ||
<p>The Hash metadata about the field.</p> | ||
</dd> | ||
</dl> | ||
</dd> | ||
</dl> | ||
### Examples[¶](#module-Stretchy-label-Examples) [↑](#top) | ||
|
||
``` | ||
classMyModelincludeStretchyModelattribute:name,:boolean,doc\_values:true,store:trueend | ||
``` | ||
|
||
### Returns[¶](#module-Stretchy-label-Returns) [↑](#top) | ||
|
||
Returns nothing. | ||
|
||
Defines a dense vector attribute for the model. | ||
|
||
<dl class="rdoc-list note-list"> | ||
<dt>name | ||
</dt> | ||
<dd> | ||
<p>The Symbol name of the attribute.</p> | ||
</dd> | ||
<dt>opts | ||
</dt> | ||
<dd> | ||
<p>The Hash options used to refine the attribute (default: {}):</p> | ||
<dl class="rdoc-list note-list"> | ||
<dt>:element_type | ||
</dt> | ||
<dd> | ||
<p>The String data type used to encode vectors. Can be ‘float’ or ‘byte’. ‘float’ indexes a 4-byte floating-point value per dimension. ‘byte’ indexes a 1-byte integer value per dimension. Vectors using ‘byte’ require dimensions with integer values between -128 to 127.</p> | ||
</dd> | ||
<dt>:dims | ||
</dt> | ||
<dd> | ||
<p>The Integer number of vector dimensions. Can’t exceed 4096. If not specified, it will be set to the length of the first vector added to the field.</p> | ||
</dd> | ||
<dt>:index | ||
</dt> | ||
<dd> | ||
<p>The Boolean indicating if you can search this field using the kNN search API (default: true).</p> | ||
</dd> | ||
<dt>:similarity | ||
</dt> | ||
<dd> | ||
<p>The String vector similarity metric to use in kNN search. Can be ‘l2_norm’, ‘dot_product’, ‘cosine’, or ‘max_inner_product’. This parameter can only be specified when :index is true.</p> | ||
</dd> | ||
<dt>:index_options | ||
</dt> | ||
<dd> | ||
<p>The Hash that configures the kNN indexing algorithm. Can only be specified when :index is true.</p> | ||
<dl class="rdoc-list note-list"> | ||
<dt>:type | ||
</dt> | ||
<dd> | ||
<p>The String type of kNN algorithm to use. Can be either ‘hnsw’ or ‘int8_hnsw’.</p> | ||
</dd> | ||
<dt>:m | ||
</dt> | ||
<dd> | ||
<p>The Integer number of neighbors each node will be connected to in the HNSW graph.</p> | ||
</dd> | ||
<dt>:ef_construction | ||
</dt> | ||
<dd> | ||
<p>The Integer number of candidates to track while assembling the list of nearest neighbors for each new node.</p> | ||
</dd> | ||
<dt>:confidence_interval | ||
</dt> | ||
<dd> | ||
<p>The Float confidence interval to use when quantizing the vectors. Only applicable to ‘int8_hnsw’ index types.</p> | ||
</dd> | ||
</dl> | ||
</dd> | ||
</dl> | ||
</dd> | ||
</dl> | ||
### Examples[¶](#module-Stretchy-label-Examples) [↑](#top) | ||
|
||
``` | ||
classMyModelincludeStretchyModelattribute:vector\_embeddings,:dense\_vector,element\_type:'float',dims:3end | ||
``` | ||
|
||
### Returns[¶](#module-Stretchy-label-Returns) [↑](#top) | ||
|
||
Returns nothing. | ||
|
||
attribute :ml, :sparse\_vector | ||
|
||
{ “mappings”: { “properties”: { “ml.tokens”: { “type”: “sparse\_vector” } } } } | ||
|
||
TODO: Break out modules into separate files | ||
|
||
## Constants | ||
| Name | Description | | ||
| ---- | ----------- | | ||
| **VERSION[](#VERSION)** | Not documented | | ||
## Public Class Methods | ||
### boot!() [](#method-c-boot-21) | ||
### configuration() [](#method-c-configuration) | ||
### configure() { |configuration| ... } [](#method-c-configure) | ||
### loader() [](#method-c-loader) | ||
### logger() [](#method-c-logger) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
# module Stretchy::Attributes [](#module-Stretchy::Attributes) [](#top) | ||
## Public Class Methods | ||
### register!() [](#method-c-register-21) | ||
## Public Instance Methods | ||
### [](attribute) [](#method-i-5B-5D) | ||
### []=(attribute, value) [](#method-i-5B-5D-3D) | ||
### attribute_mappings() [](#method-i-attribute_mappings) | ||
### inspect() [](#method-i-inspect) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# module Stretchy::Attributes::Transformers [](#module-Stretchy::Attributes::Transformers) [](#top) | ||
|
18 changes: 18 additions & 0 deletions
18
docs/doc/Stretchy/Attributes/Transformers/KeywordTransformer.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
# class Stretchy::Attributes::Transformers::KeywordTransformer [](#class-Stretchy::Attributes::Transformers::KeywordTransformer) [](#top) | ||
## Constants | ||
| Name | Description | | ||
| ---- | ----------- | | ||
| **KEYWORD_AGGREGATION_KEYS[](#KEYWORD_AGGREGATION_KEYS)** | Not documented | | ||
## Attributes | ||
### attribute_types[R] [](#attribute-i-attribute_types) | ||
## Public Class Methods | ||
### new(attribute_types) [](#method-c-new) | ||
## Public Instance Methods | ||
### assume_keyword_field(args={}, parent_match=false) [](#method-i-assume_keyword_field) | ||
If terms are used, we assume that the field is a keyword field and append .keyword to the field name {terms: {field: ‘gender’}} or nested aggs {terms: {field: ‘gender’}, aggs: {name: {terms: {field: ‘position.name’}}}} should be converted to {terms: {field: ‘gender.keyword’}, aggs: {name: {terms: {field: ‘position.name.keyword’}}}} {date\_histogram: {field: ‘created\_at’, interval: ‘day’}} TODO: There may be cases where we don’t want to add .keyword to the field and there should be a way to override this | ||
|
||
### cast_value_keys() [](#method-i-cast_value_keys) | ||
### keyword?(arg) [](#method-i-keyword-3F) | ||
### protected?(arg) [](#method-i-protected-3F) | ||
### transform(item, *ignore) [](#method-i-transform) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# module Stretchy::Attributes::Type [](#module-Stretchy::Attributes::Type) [](#top) | ||
a space-optimized variant of text that disables scoring and performs slower on queries that need positions. It is best suited for indexing log messages. | ||
|
||
# Public | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# class Stretchy::Attributes::Type::Base [](#class-Stretchy::Attributes::Type::Base) [](#top) | ||
## Constants | ||
| Name | Description | | ||
| ---- | ----------- | | ||
| **OPTIONS[](#OPTIONS)** | Not documented | | ||
## Public Class Methods | ||
### new(**args) [](#method-c-new) | ||
## Public Instance Methods | ||
### mappings(name) [](#method-i-mappings) | ||
### type_for_database() [](#method-i-type_for_database) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# class Stretchy::Attributes::Type::Binary [](#class-Stretchy::Attributes::Type::Binary) [](#top) | ||
## Constants | ||
| Name | Description | | ||
| ---- | ----------- | | ||
| **OPTIONS[](#OPTIONS)** | Not documented | | ||
## Public Class Methods | ||
### new(**args) [](#method-c-new) | ||
## Public Instance Methods | ||
### mappings(name) [](#method-i-mappings) | ||
### type() [](#method-i-type) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
# class Stretchy::Attributes::Type::Boolean [](#class-Stretchy::Attributes::Type::Boolean) [](#top) | ||
## Constants | ||
| Name | Description | | ||
| ---- | ----------- | | ||
| **OPTIONS[](#OPTIONS)** | Not documented | | ||
## Public Class Methods | ||
### new(**args) [](#method-c-new) | ||
## Public Instance Methods | ||
### type() [](#method-i-type) | ||
|
Oops, something went wrong.