wip

Author: tobyhede

### Troubleshooting Guide for Potential .md

@@ -0,0 +1,56 @@
+### Troubleshooting Guide for Potential Constraint Failures
+
+
+
+This guide provides information on potential constraint failures related to the encrypted json payload of EQL.
+It explains what the current errors mean and what you can do to fix them.
+
+
+The database constraint cs_configuration_data_v1_check
+
+1. _cs_config_check_v(VALUE)
+Error Message:
+
+Meaning: This error indicates that the version value in the configuration is not valid.
+
+Fix: Ensure that the version value in the configuration JSON is correct and matches the expected format. The version should be a valid integer or string that the function _cs_config_check_v can recognize.
+
+2. _cs_config_check_tables(VALUE)
+Error Message:
+
+Meaning: This error indicates that the table structure in the configuration JSON is not valid.
+
+Fix: Verify that the table structure in the configuration JSON is correctly defined. Ensure that all required fields are present and that the structure adheres to the expected schema. The function _cs_config_check_tables checks for the presence and correctness of table definitions.
+
+3. _cs_config_check_cast(VALUE)
+Error Message:
+
+Meaning: This error indicates that the cast value in the configuration JSON is not valid.
+
+Fix: Check the cast values in the configuration JSON to ensure they are valid. The function _cs_config_check_cast validates the cast values against a predefined list of acceptable values. Ensure that the cast values are correctly specified and match the expected types.
+
+4. _cs_config_check_indexes(VALUE)
+Error Message:
+
+Meaning: This error indicates that the index structure in the configuration JSON is not valid.
+
+Fix: Review the index structure in the configuration JSON to ensure it is correctly defined. The function _cs_config_check_indexes checks for the presence and correctness of index definitions. Ensure that all indexes are either empty or contain only keys from a list of valid values ({match, ore, unique, ste_vec}).
+
+General Steps to Fix Configuration Errors
+Validate JSON Structure:
+
+Ensure that the JSON structure is well-formed and adheres to the expected schema.
+Use JSON validation tools to check for syntax errors.
+Check Required Fields:
+
+Verify that all required fields are present in the configuration JSON.
+Ensure that the field names and types match the expected values.
+Review Function Definitions:
+
+Review the definitions of the functions _cs_config_check_v, _cs_config_check_tables, _cs_config_check_cast, and _cs_config_check_indexes to understand the validation logic.
+Ensure that the configuration JSON meets the criteria defined in these functions.
+Test with Sample Data:
+
+Test the configuration JSON with sample data to identify and fix errors.
+Use known valid configurations as a reference to compare and correct the current configuration.
+By following this troubleshooting guide, you can identify and fix potential constraint failures related to the configuration checks in the 020-config-schema.sql file.

.gitignore

@@ -1,7 +1,17 @@
+
+
+.DS_Store
+.mise.*
+
+deps.txt
+deps-ordered.txt
+
 # Based on https://raw.githubusercontent.com/github/gitignore/main/Node.gitignore
 
 sql/000-version.sql
 
+
+
 # Logs
 
 logs
@@ -186,7 +196,7 @@ cipherstash-proxy.toml
 # build artifacts
 release/
 
-.mise.*
+
 
 # jupyter notebook
 .ipynb_checkpoints

.vscode/settings.json

@@ -0,0 +1,5 @@
+{
+    "conventionalCommits.scopes": [
+        "ore"
+    ]
+}

DEVELOPMENT.md

@@ -0,0 +1,128 @@
+
+
+
+
+====
+
+
+###
+
+EQL is installed into the `eql_v1` schema.
+
+
+
+
+
+## Types
+
+
+eql_v1_encrypted - Column Type
+
+eql_v1.unique_index;
+eql_v1.match;
+eql_v1.ore_64_8_v1;
+eql_v1.ore_64_8_v1_term;
+
+
+## Operators
+
+Operators are provided for the `eql_v1_encrypted` column type and `jsonb`.
+
+```
+eql_v1_encrypted - eql_v1_encrypted
+jsonb - eql_v1_encrypted
+eql_v1_encrypted - jsonb
+```
+
+The index types and functions are internal implementation details and should not need to be exposed as operators on the `eql_v1_encrypted` type.
+
+
+--      eql_v1_encrypted = eql_v1_encrypted
+--      eql_v1_encrypted = jsonb
+--      jsonb = eql_v1_encrypted
+--      ore_64_8_v1 = ore_64_8_v1
+The jsonb comparison is handy as it automates casting.
+Comparing ore_64_8_v1 index values requires that sides are functionalated:
+eql_v1.ore_64_8_v1(...) = eql_v1.ore_64_8_v1(...)
+In the spirit of aggressive simplification, however, I am not going to add operators to compare eql_v1_encrypted with the ore_64_8_v1 type.
+In an operator world,  the index types and functions are internal implementation details.
+Customers should never need to think about the internals.
+I can't think of a reason to need it that isn't a version of "holding it wrong". (edited)
+
+
+
+
+## Working without operators
+
+
+### Equality
+
+```sql
+eql_v1.eq(a eql_v1_encrypted, b eql_v1_encrypted);
+```
+
+
+
+
+
+## Organisation
+
+Break SQL into small modules, aligned with the core domains and types where possible
+
+ - types.sql
+ - casts.sql
+ - constraints.sql
+ - functions.sql
+ - operators.sql
+
+Operators are also functions, so some judgement is required.
+The intent is to reduce file size and cognitive load.
+
+In general, operator functions should be thin wrappers around a larger function that does the work.
+Put the wrapper functions in `operators.sql` and the "heavy lifting" functions in `functions.sql`.
+
+Tests should follow a similar pattern.
+
+
+
+### Dependencies
+
+SQL sources are split into smaller files.
+Dependencies are resolved at build time to construct a single SQL file with the correct ordering.
+
+Dependencies between files are declared in a comment at the top of the file.
+All SQL files should `REQUIRE` the source file of any other object they reference.
+
+All files must have at least one declaration, and the default is to reference the schema
+
+```
+-- REQUIRE: src/schema.sql
+```
+
+
+
+### Tables
+
+### Configuration
+
+
+`public.eql_v1_configuration`
+
+
+
+EQL Design Note
+Experimenting with using a Composite type instead of a Domain type for the encrypted column.
+Composite types are a bit more capable. Domain types are more like an alias for the underlying type (in this case jsonb)
+The consequence of using a Composite type is that the data is stored in the column as a Tuple - effectively the data is wrapped in ()
+This means
+on insert/update the data needs to be cast to eql_v1_encrypted (proxy mapping will handle)
+on read the data needs to be cast back to jsonb if a customer needs the raw json (for data lake transfer etc etc)
+Already built cast helpers so syntax is something like
+    INSERT INTO encrypted (e) VALUES (
+        eql_v1.to_encrypted('{}')
+    );
+
+    INSERT INTO encrypted (e) VALUES (
+        '{}'::jsonb::eql_v1_encrypted
+    );
+

REFERENCE.md

@@ -0,0 +1,18 @@
+
+
+
+
+### cs_encrypt_v1
+
+Marks the currently `pending` configuration as `encrypting`.
+
+Validates the database schema and returns an error if the configured columns are not of `jsonb` or `cs_encrypted_v1` type.
+
+
+
+
+
+
+```
+SELECT cs_encrypt_v1(true);
+```