docs: json datatype rfc

docs/rfcs/2024-08-06-json-datatype.md

@@ -0,0 +1,104 @@
+---
+Feature Name: Json Datatype
+Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/4230
+Date: 2024-8-6
+Author: "Yuhan Wang <[email protected]>"
+---
+
+# Summary
+This RFC proposes a method for storing and querying JSON data in the database.
+
+# Motivation
+JSON is widely used across various scenarios. Direct support for writing and querying JSON can significantly enhance the database's flexibility.
+
+# Details
+
+## User Interface
+The feature introduces a new data type for the database, similar to the common JSON type. Data is written as JSON strings and can be queried using functions.
+
+For example:
+```SQL
+CREATE TABLE IF NOT EXISTS test (
+    ts TIMESTAMP TIME INDEX,
+    a INT,
+    b JSON
+);
+
+INSERT INTO test VALUES(
+    0,
+    0,
+    '{
+        "name": "jHl2oDDnPc1i2OzlP5Y",
+        "timestamp": "2024-07-25T04:33:11.369386Z",
+        "attributes": { "event_attributes": 48.28667 }
+    }'
+);
+
+SELECT json_get(b, 'name') FROM test;
++---------------------+
+| b.name              |
++---------------------+
+| jHl2oDDnPc1i2OzlP5Y |
++---------------------+
+
+SELECT json_get_by_paths_int(b, 'attributes', 'event_attributes') + 1 FROM test;
++-------------------------------+
+| b.attributes.event_attributes |
++-------------------------------+
+|                      49.28667 |
++-------------------------------+
+
+```
+
+## Storage and Query
+
+Data of JSON type is stored as JSONB format in the database. For storage layer and query engine, data is represented as a binary array and can be queried through pre-defined JSON functions. For clients, data is shown as strings.
+
+Insertions of JSON goes through following steps:
+
+1. Client gets JSON strings and sends it to the frontend.
+2. Frontend encode JSON strings to JSONB format and sends it to the datanode.
+3. Datanode stores binary data in the database.
+
+```
+Insertion:
+                          Encode                    Store
+         JSON Strings ┌────────────┐ JSONB Data ┌────────────┐
+ client ------------->│  Frontend  │----------->│  Datanode  │--> Storage
+                      └────────────┘            └────────────┘
+```
+
+The data of JSON type is represented by `Binary` data type in arrow. There are 2 types of JSON queries: get json elements through keys and compute over json elements.
+
+For the former, the query engine performs queries directly over binary data. We provide functions like `json_get` and `json_get_by_paths` to extract json elements through keys.
+
+For the latter, users need to manually specify the data type of the json elements for computing. Before computing, the query engine will decode the binary data in JSONB format into the specified data type. We provide functions like `json_get_int` and `json_get_by_paths_double` to extract json elements and convert them for further computation.
+
+Queries of JSON goes through following steps:
+
+1. Client sends query to frontend, and frontend sends it to datafusion, which is the query engine of GreptimeDB.
+2. Datafusion performs query over JSON data, and returns binary data to frontend.
+3. If no computation is needed, frontend directly decodes it to JSON strings and return it to clients.
+4. If computation is needed, the binary data is decoded and converted to the specified data type to perform computation. There's no need for further decoding in the frontend.
+
+```
+Queries without computation:
+                          Decode                     Query
+         JSON Strings ┌────────────┐ JSONB Data ┌──────────────┐
+ client <-------------│  Frontend  │<-----------│  Datafusion  │<-- Storage
+                      └────────────┘            └──────────────┘
+
+Queries with computation:
+                                                                         Query
+         Data of Specified Type ┌────────────┐ Data of Certain Type ┌──────────────┐
+ client <-----------------------│  Frontend  │<---------------------│  Datafusion  │<-- Storage
+                                └────────────┘                      └──────────────┘
+```
+
+# Drawbacks
+
+As a general purpose data type, JSONB may not be as efficient as specialized data types for specific scenarios.
+
+# Alternatives
+
+Extract and flatten JSON schema to store in a structured format through pipeline. For nested data, we can provide nested types like `STRUCT` or `ARRAY`.