Bump mysql-connector-java from 5.1.33 to 8.0.16 in /knowagedao

.gitignore

@@ -87,3 +87,9 @@ Knowage-Python/pythonwebservice/logs/
 Knowage-Python/pythonwebservice/cert.pem
 
 Knowage-Python/pythonwebservice/key.pem
+
+Knowage-R/configs.R
+
+Knowage-R/.Rhistory
+
+Knowage-R/.RData

CLA.md

@@ -0,0 +1,62 @@
+Contributor License Agreement
+===============
+
+[www.knowage-suite.com](https://www.knowage-suite.com)
+
+Thank you for your interest in [Knowage Project](https://www.knowage-suite.com). In order to clarify both the intellectual property license, granted with Contributions from any person, and the contribution management guidelines, outlined in order to achieve a project systematic growth, the Knowage Project must have a Contributor License Agreement (or Agreement) that has been accepted by each Contributor, indicating his/her agreement to terms and conditions of this Contributor License Agreement.
+
+This Agreement is for your protection as a Contributor as well as the protection of ENGINEERING Ingegneria Informatica S.p.A., which keeps the copyright of Knowage software and documentation. This does not change your rights to use your own Contributions for any other purpose.
+
+This Agreement:
+
+- clarifies that, to your knowledge, the technology you’ve submitted was developed by you and that you have authority to provide it;
+- grants to ENGINEERING Ingegneria Informatica S.p.A. a perpetual, worldwide, no-charge, royalty-free, irrevocable, non-exclusive copyright license to use your Contribution and every derivative work, according to Knowage license.
+
+**Please read this document carefully before accepting its terms and conditions.**
+
+Definitions
+---------------
+
+“Contributor” is the copyright owner that is making this Agreement with ENGINEERING Ingegneria Informatica S.p.A.
+
+“Engineering” (ENGINEERING Ingegneria Informatica S.p.A.) is the entity owner of the copyright of Knowage software and documentation.
+
+“Knowage Project” or “Project” shall mean every organization, procedure, mean, practice, term, condition and license, used by ENGINEERING Ingegneria Informatica S.p.A. for the aim of the free/open source Knowage project development by means of all its software components and documentation which are pointed out on the project site: [www.knowage-suite.com](https://www.knowage-suite.com).
+
+“Contribution” shall mean any original work of authorship, including any modifications or additions to an existing work, which is intentionally submitted by the Contributor to ENGINEERING for inclusion in Knowage software or documentation. For the purpose of this definition, “submitted” means any form of electronic, verbal or written communication sent to ENGINEERING, including but not limited to communication on electronic mailing lists, source code control systems and issue tracking systems that are managed by ENGINEERING for the purpose of discussing and improving Knowage software and documentation, but excluding the communication that is conspicuously marked or otherwise designated in writing by the Contributor as “Not a Contribution”.
+
+Acceptance and Termination procedure
+---------------
+
+The Contributor agrees and commits himself/herself to respect both the Contribution Guidelines (you can find it on [www.knowage-suite.com](https://www.knowage-suite.com)) and the Acceptance and Termination procedure, as follows.
+
+To contribute, the Contributor must accept this Contributor License Agreement and the Contribution Guidelines of this Agreement. Otherwise, ENGINEERING Ingegneria Informatica S.p.A. will not consider his/her Contribution.
+
+A Developer who has sent Contributions in solid, useful source code and/or documentation and who asked to be a Contributor will be add to our Contributor List published on Knowage website. ENGINEERING will notify to the Contributor via email his/her elevation to Contributor status.
+
+ENGINEERING can exclude a Contributor from the Contributors list, by its exclusive right, when this Contributor has not respected the above rules repeatedly or when s/he has not contributed for a long period (more than three years). If any of these events occurs, any assignment under this Agreement for the Contributor’s Contribution shall terminate, by a communication that ENGINEERING will send to the Contributor, starting from the date of this communication.
+
+Contributor’s responsibility
+---------------
+
+The Contributor owns and has the rights to contribute all source code and related material, intended to be compiled or integrated with the source code for Knowage software and documentation which the Contributor has ever delivered, and the Project has accepted, for incorporation into the technology, made available under the Knowage Project.
+
+The Contributor grants to comply with the applicable laws and regulations, the Knowage license terms of use and the guiding principles concerning any use of copyrighted materials.
+
+Copyright assignment by Contributor
+---------------
+
+Subject to the terms and conditions of this Agreement, the Contributor grants to ENGINEERING Ingegneria Informatica S.p.A. a perpetual, worldwide, no-charge, royalty-free, irrevocable, non-exclusive copyright license to use the Contribution and every derivative work, according to Knowage License.
+
+The Contributor represents that s/he is legally entitled to grant the above assignment.
+
+The Contributor represents that his/her Contribution is his/her original creation.
+
+The Contributor represents that his/her Contribution includes complete details of any third-party license or other restriction of which s/he is personally aware and which are associated with any party of his/her Contribution.
+
+Miscellaneous Terms
+---------------
+
+The Agreement is subject to the Italian law and any dispute arising between the Parties relating to this Agreement shall be under the exclusive jurisdiction of the Court of Rome, Italy.
+
+These Terms constitute the entire agreement between you and ENGINEERING relating to the subject matter herein and supersede all prior communications, agreements, understandings and arrangements between you and ENGINEERING, whether oral or written.

Knowage-Python/.dockerignore

@@ -0,0 +1,6 @@
+.git
+.idea
+__pycache__
+*.pyc
+*.pyo
+*.pyd

Knowage-Python/Dockerfile

@@ -0,0 +1,11 @@
+FROM python:3.7
+
+COPY . .
+
+RUN pip install -r requirements.txt
+
+WORKDIR /pythonwebservice
+
+EXPOSE 5000
+
+CMD [ "python3", "knowage-python.py" ]

Knowage-Python/pythonwebservice/__init__.py

@@ -0,0 +1,2 @@
+#!/usr/bin/env python3
+

Knowage-Python/pythonwebservice/app/__init__.py

@@ -0,0 +1,2 @@
+#!/usr/bin/env python3
+

Knowage-Python/pythonwebservice/app/services/__init__.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 from flask import Flask
 from flask_cors import CORS
 

Knowage-Python/pythonwebservice/app/services/dataset_service.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 # Knowage, Open Source Business Intelligence suite
 # Copyright (C) 2016 Engineering Ingegneria Informatica S.p.A.
 #

Knowage-Python/pythonwebservice/app/services/editmode_service.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 # Knowage, Open Source Business Intelligence suite
 # Copyright (C) 2016 Engineering Ingegneria Informatica S.p.A.
 #
@@ -45,7 +47,7 @@ def python_html():
         python_widget.script = python_widget.script.replace("$P{" + d + "}", "drivers_.get(\'" + d + "\')")
     #retrieve dataset
     if python_widget.dataset_name != None:
-        dataset_file = "tmp/" + python_widget.dataset_name + ".pckl"
+        dataset_file = constants.TMP_FOLDER + python_widget.dataset_name + ".pckl"
         df = utils.getDatasetAsDataframe(python_widget)
         df.to_pickle(dataset_file)
         python_widget.script = "import pandas as pd\n" + python_widget.dataset_name + " = pd.read_pickle(\"" + dataset_file + "\")\n" + python_widget.script

Knowage-Python/pythonwebservice/app/services/viewmode_service.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 # Knowage, Open Source Business Intelligence suite
 # Copyright (C) 2016 Engineering Ingegneria Informatica S.p.A.
 #

Knowage-Python/pythonwebservice/app/utilities/__init__.py

@@ -0,0 +1,2 @@
+#!/usr/bin/env python3
+

Knowage-Python/pythonwebservice/app/utilities/constants.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 EDIT_PYTHON_SCRIPTS = "EditPythonScripts"
 TMP_FOLDER = "app/services/tmp/"
 LOG_FILE = "logs/knowage-python.log"

Knowage-Python/pythonwebservice/app/utilities/cuncurrency_manager.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 from threading import Lock
 
 active_servers = {} #{widget_id : bokeh_server}

Knowage-Python/pythonwebservice/app/utilities/objects.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 # Knowage, Open Source Business Intelligence suite
 # Copyright (C) 2016 Engineering Ingegneria Informatica S.p.A.
 #
@@ -63,6 +65,10 @@ def widget_id(self):
     def script(self):
         return self._script
 
+    @property
+    def output_variable(self):
+        return self._output_variable
+
     @script.setter
     def script(self, value):
         self._script = value

Knowage-Python/pythonwebservice/app/utilities/security.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 # Knowage, Open Source Business Intelligence suite
 # Copyright (C) 2016 Engineering Ingegneria Informatica S.p.A.
 #

Knowage-Python/pythonwebservice/app/utilities/utils.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 # Knowage, Open Source Business Intelligence suite
 # Copyright (C) 2016 Engineering Ingegneria Informatica S.p.A.
 #

Knowage-Python/pythonwebservice/knowage-python.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 import sys
 from app.services import create_app
 from app.utilities import constants

Knowage-Python/requirements.txt

@@ -0,0 +1,8 @@
+bokeh==1.3.4
+Flask==1.1.1
+Flask-Cors==3.0.8
+pandas==0.25.1
+pybase64==1.0.1
+PyJWT==1.7.1
+requests==2.22.0
+tornado==6.0.3

Knowage-Python/setup.py

@@ -1,11 +1,18 @@
+#!/usr/bin/env python3
+
 import setuptools
 
+def parse_requirements(filename):
+    """ load requirements from a pip requirements file """
+    lineiter = (line.strip() for line in open(filename))
+    return [line for line in lineiter if line and not line.startswith("#")]
+
 with open("README.md", "r") as f:
     long_description = f.read()
 
 setuptools.setup(
     name="knowage-python",
-    version="0.2.8",
+    version="1.0",
     license='AGPL v3',
     author="Marco Balestri",
     author_email="[email protected]",
@@ -17,16 +24,7 @@
     package_data={
         '': ['*.html'],
     },
-    install_requires=[
-          'flask',
-          'flask_cors',
-          'pybase64',
-          'bokeh',
-          'tornado',
-          'requests',
-          'pandas',
-          'pyjwt',
-      ],
+    install_requires = parse_requirements('requirements.txt'),
     classifiers=[
         "Programming Language :: Python :: 3",
         "License :: OSI Approved :: GNU Affero General Public License v3",

Knowage-R/configs.R

@@ -0,0 +1 @@
+hmac_key <- "abc123"

Knowage-R/constants.R

@@ -0,0 +1,2 @@
+knowage_address <- "0.0.0.0"
+r_webservice_port <- 5001

Knowage-R/knowage-r.R

@@ -0,0 +1,4 @@
+source("constants.R")
+
+pr <- plumber::plumb("webservice.R")
+pr$run(port = r_webservice_port, host = knowage_address, swagger = FALSE)

Knowage-R/utils.R

@@ -0,0 +1,69 @@
+library(openssl)
+library(jose)
+source("configs.R")
+
+get_libraries <- function(){
+  str(allPackages <- installed.packages(.Library, priority = "high"))
+  lib_matrix <- allPackages[, c(1,3:5)]
+  lib_info <- lib_matrix[,c(1,2)]
+  lib_info
+}
+
+resolve_drivers <- function(script, drivers){
+  for(name in names(drivers)) {
+    value <- drivers[[name]]
+    original <- paste0('\\$P\\{', name , '\\}')
+    final <- paste0('drivers_[[\\"', name, '\\"]]')
+    script <- gsub(original,final,script)
+  }
+  script
+}
+
+build_parameters <- function(parameters){
+  to_return <- list()
+  if(length(parameters) > 0) {
+    parameters_df <- as.data.frame(parameters)
+    for(i in 1:nrow(parameters_df)) {
+      row <- parameters_df[i,]
+      name <- row[["name"]]
+      value <- row[["value"]]
+      type <- row[["type"]]
+      if(value == "") {
+        value <- row[["defaultValue"]]
+      }
+      if(type == "Number") {
+        value <- as.numeric(value)
+      }
+      to_return[[name]] <- value
+    }
+  }
+  to_return
+}
+
+resolve_parameters <- function(script, parameters){
+  for(name in names(parameters)) {
+    value <- parameters[[name]]
+    original <- paste0('\\$P\\{', name , '\\}')
+    final <- paste0('parameters_[[\\"', name, '\\"]]')
+    script <- gsub(original,final,script)
+  }
+  script
+}
+
+decode_jwt_token <- function(script){
+  token <- jwt_decode_hmac(script, secret = hmac_key)
+  token
+}
+
+get_script_from_token <- function(token){
+  script <- token[["script"]]
+  script
+}
+
+is_dataset_request_authorized <- function(token){
+  expirationTime <- token[["exp"]]
+  now <- as.numeric(as.POSIXct(Sys.time()))
+  if (now > expirationTime)
+    FALSE
+  TRUE
+}

Knowage-R/webservice.R

@@ -0,0 +1,58 @@
+library("jsonlite")
+library("base64enc")
+source("utils.R")
+
+#' @post /img
+function(dataset, dataset_name=NULL, script, drivers, output_variable){
+  env <- new.env()
+  analytical_drivers <- fromJSON(drivers)
+  env$drivers_ <- analytical_drivers
+  script <- resolve_drivers(script, analytical_drivers)
+  if (!is.null(dataset_name)) {
+    script <- gsub(dataset_name,"df_",script)
+    env$df_ <- as.data.frame(fromJSON(dataset))
+  }
+  eval(parse(text=script), envir = env)
+  enc_img <- base64encode(output_variable)
+  if (file.exists(output_variable))
+    file.remove(output_variable)
+  enc_img
+}
+
+#' @post /html
+function(dataset, dataset_name=NULL, script, drivers, output_variable){
+  env <- new.env()
+  analytical_drivers <- fromJSON(drivers)
+  env$drivers_ <- analytical_drivers
+  script <- resolve_drivers(script, analytical_drivers)
+  if (!is.null(dataset_name)) {
+    script <- gsub(dataset_name,"df_",script)
+    env$df_ <- as.data.frame(fromJSON(dataset))
+  }
+  script <- gsub(output_variable,"env$output_variable_",script)
+  eval(parse(text=script), envir = env)
+  html <- env$output_variable_
+  html
+}
+
+#' @post /dataset
+function(script, df_name, parameters){
+  env <- new.env()
+  token <- decode_jwt_token(script)
+  if (!is_dataset_request_authorized(token))
+    stop("Unauthorized")
+  decoded_script <- get_script_from_token(token)
+  env$parameters_ <- build_parameters(parameters)
+  decoded_script <- resolve_parameters(decoded_script, env$parameters_)
+  decoded_script <- gsub(df_name, "df_", decoded_script)
+  env$df_ <- data.frame()
+  eval(parse(text=decoded_script), envir = env)
+  env$df_
+}
+
+#' @get /libraries
+#' @get /dataset/libraries
+function(){
+  lib <- get_libraries()
+  lib
+}

ROADMAP.md

@@ -39,9 +39,7 @@ The following list of features is planned to be addressed in the short term, by
 
 The following list of features is planned to be addressed in the medium term, by Q2 2020 (release 7.3):
 
-* from data exploration tool (QbE) to cockpit designer in a seamless way 
-
-* AWS S3 compatibility
+* from data exploration tool (QbE) to cockpit designer in a seamless way
 
 * NGSI-LD compatibility
 

cas/src/main/webapp/META-INF/context.xml

@@ -1,4 +1,5 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <Context >
-	<ResourceLink name="hmacKey" global="hmacKey" type="java.lang.String"/>
-</Context>
+	<ResourceLink global="hmacKey" name="hmacKey" type="java.lang.String"/>
+
+ </Context>