Add tutorial, implementation matrix and execution instructions to sha…

…cl2flink directory

Signed-off-by: marcel <[email protected]>

KafkaBridge/config/config.json

@@ -54,8 +54,7 @@
 	"debeziumBridge": {
 		"topic": "iff.ngsild.public.entity",
 		"entityTopicPrefix": "iff.ngsild.entities",
-		"attributesTopic": "iff.ngsild.attributes",
-		"rdfSources": ["../config/knowledge.ttl"]
+		"attributesTopic": "iff.ngsild.attributes"
 	},
 	"bridgeCommon": {
 		"kafkaSyncOnAttribute": "https://industry-fusion.com/types/v0.9/metadata/kafkaSyncOn"

semantic-model/shacl2flink/Makefile

@@ -114,7 +114,6 @@ flink-deploy: clean
 	# make build || make enable-strimzi
 	kubectl -n $(NAMESPACE) delete -f output/ngsild-kafka.yaml --ignore-not-found || make enable-strimzi
 	kubectl -n $(NAMESPACE) delete -f output/rdf-kafka.yaml --ignore-not-found || make enable-strimzi
-	kubectl -n $(NAMESPACE) delete -f output/knowledge-configmap.yaml --ignore-not-found
 	cd ../../helm && ./helmfile -f helmfile-shacl.yaml apply || make enable-strimzi
 	make enable-strimzi
 	make test-flink-is-deployed

semantic-model/shacl2flink/README.md

@@ -1,10 +1,32 @@
-# Digital Twin Shacl2Flink
+# Digital Twin Shacl to Flink Transformation
+
+This directory contains the translation mechanism from SHACL basee constraints and rules to SQL/Flink.
+There are always three ingredients to such a translation, called KMS (Knowledge, Model-instance, SHACL)
+
+- **K**nowledge contains OWL/RDF data, preferable serialized in Turtle
+- **M**odel-instance describes the actual instances/objects of the setup. These are described in JSON-LD/NGSI-LD.
+- **S**HACL is the W3C standard describing the constraints and rules for the model with respect to the Knowledge.
+
+A first [overview](../datamodel/README.md) and [tutorial](../datamodel/Tutorial.md) can be found in the [datamodel](../datamodel/) directory.
+
+# Table of Contents
+
+1. [Quick Setup](#quick-setup)
+2. [KMS Examples & Tutorial](./docs/examples.md)
+3. [Supported SHACL Features](./docs/supported-features.md)
+4. User Defined Functions
+3. [Build and test KMS](#build-and-test-kms)
+4. [Deploy Flink-Jobs](#deploy-flink-jobs)
+5. [References](#references)
+
+# Quick Setup
 
 ## Requirements
 
-- You need to have Python > 3.8
-- Virtualenv needs to be installed
-- `sqlite3` and `sqlite3-pcre` need to be installed
+- Python > 3.8
+- Virtualenv installed
+
+In addition, install `sqlite3` and `sqlite3-pcre`
 
   ```bash
   sudo  apt install sqlite3 sqlite3-pcre
@@ -23,12 +45,6 @@ source venv/bin/activate
 make setup
 ```
 
-Then run:
-
-```bash
-make build
-```
-
 ## VS Code
 
 Normally VS Code should recognize the virtual environment and ask you if you want to use the virtual environment as you Python interpreter.
@@ -49,12 +65,77 @@ pip install -r requirements-dev.txt
 Run with
 
 ```bash
-python -m pytest
+make test
 ```
 ## Linting
 
 Run with
 
 ```bash
 make lint
-```
+```
+
+
+# Build and Test KMS
+## Build KMS directory
+
+There are three files expected in the `../kms` directory:
+
+- shacl.ttl
+- knowledge.ttl
+- model-instance.ttl
+
+To build:
+
+```bash
+make build
+```
+
+As a result, there must be a new directory `output` with the following files included:
+
+- **core.yaml** - SQL-Tables for Flink (Core tables are used independent of concrete SHACL rules)
+- **core.sqlite** - SQL-Tables for SQLite (Core tables are used independent of concrete SHACL rules)
+- **shacl-validation.yaml** - From SHACL compiled SQL scripts for Flink
+- **shacl-validation.sqlite** - From SHACL compiled SQL scripts for SQLite
+- **shacl-validation-maps.yaml** - Additional SQL scripts when result is too large to store in  **shacl-validation.sqlite** directly
+- **rdf.sqlite** - Knowledge translated to RDF triples for SQLite
+- **rdf.yaml** - Knowledge translated to RDF triples for Flink
+- **ngsild-kafka.yaml** - Kafka topics used by Flink
+- **ngsild-models.sqlite** - translated model-instance.ttl for SQLite (only for SQLite needed)
+- **ngsild.sqlite** - SQL tables for the concrete SHACL rules generated for SQLite
+- **ngsild.yaml** - SQL tables for the concrete SHACL rules generated for Flink
+- **rdf-kafka.yaml** - Kafka topic for rdf data
+- **rdf-maps.yaml** - RDF data add-on when data is too much to fit into **rdf.yaml**
+- **udf.yaml** - User Defined Functions (UDF) for Flink SQL
+
+
+## Test locally with SQLite
+
+```bash
+make test-sqlite
+```
+
+# Deploy Flink Jobs
+
+## Deploy SHACL rules to Flink
+
+```bash
+make flink-deploy
+```
+
+## Undeploy SHACL rules to Flink
+
+```bash
+make flink-undeploy
+```
+
+# References
+
+[RDF] RDF
+[RDFS] RDFS
+[TURTLE] TURTLE
+[OWL] OWL
+[SHACL] SHACL
+[JSONLD] JSON-LD
+[XSD] XSD
+[SPARQL]

semantic-model/shacl2flink/create_rdf_table.py

@@ -166,12 +166,6 @@ def main(knowledgefile, namespace, output_folder='output'):
                                            configs.rdf_topic,
                                            configs.kafka_topic_object_label,
                                            config), fk)
-    with open(os.path.join(output_folder, "knowledge-configmap.yaml"), "w") as fp:
-        qres = g.query(filter_out)
-        class_ttl = {}
-        class_ttl['knowledge.ttl'] = qres.serialize(format='turtle').decode("utf-8")
-        fp.write("---\n")
-        yaml.dump(utils.create_configmap_generic('knowledge', class_ttl), fp)
 
 
 if __name__ == '__main__':