Bump version to 2.6.11

README.md

@@ -11,25 +11,65 @@ FormShare 2 is inspired by the excellent [FormHub](<http://github.com/SEL-Columb
 FormShare was created because:
 
 * I want to provide an open-source and **free** platform to private and public organizations to help them manage their data when using ODK.
-* ODK Aggregate, in my personal opinion, is badly designed, buggy, and not interoperable. ODK Central is just not there yet.
+* ODK Aggregate, in my personal opinion, is badly designed, buggy, and not interoperable. ODK Central, though is an enormous improvement to Aggregate, does not parse arbitrary schemata into database tables which makes data cleaning very cumbersome (e. g., cleaning data through Enketo).
 * Forks based on FormHub suffer from the same ills of their father: Django (sorry if I hurt your feelings), no proper repository, rudimentary data cleaning, no auditing, little interoperability, poor or none extensibility... among many others.
 
 FormShare 2 has been written from scratch (not a single line of code comes from Formhub, just ideas, and principles) using Python 3, [Pyramid](https://trypyramid.com/), MySQL, [ElasticSearch](https://www.elastic.co/elasticsearch/), and [PyUtilib](https://github.com/PyUtilib/pyutilib) to deliver a complete and extensible data management solution for ODK Data collection. It took us three years but is finally here :-) and it is Django free!
 
 FormShare **is for organizations** to install it in their server or cloud service to serve ODK XForms and collect and manage the submissions. FormShare is also available as a service at [https://formshare.org](https://formshare.org) for those organizations that lack the capacity or resources to run their installation.
 
+## Features
+
+**Current features**
+
+- User accounts and management
+- Group-based user permissions
+- Projects to organize users, permissions, and forms
+- Separate access for data collectors and data cleaners. This is useful when dealing with hundreds of data collectors that do not need a FormShare user access 
+- User collaborations at project level, e. g., you can allow a colleague to maintain certain aspects of your project.
+- Form and submission management
+  - With easy setup of ODK Collect using QR images
+  - With support for form version updates
+  - With testing and production stages
+  - With form and submission multimedia or data attachments
+  - With a table preview of submission data (even with thousand or millions of records) allowing in-table edits and recording any changes made to the data
+- OData live data feed for analysis with tools like Excel and Power BI. **With all CRUD operation supported**. You can even use [Excel](https://github.com/qlands/MrBot-OData-Add-In) to clean data
+- Extensibility system, e. g., You can write extensions to connect FormShare with Microsoft 365 authentication system
+- Documentation for running on AWS using Docker
+- Data cleaning API integration with R, STATA, or SPSS
+- Real-time map visualization of geo-referenced submissions at project and form level
+- Fill out forms using the web browser through [Enketo](https://github.com/qlands/formshare_enketo_plugin)
+- Download form attachments
+- Filtering submissions by submission metadata (e. g., date and time received on server)
+- Parse submissions into database tables. FormShare will create a MySQL data repository to store your data
+  - The repository is controlled with a primary key e. g., Farmed ID
+  - Duplicated submissions go to a cleaning pipe-line system that makes it easier to compare submissions and decide what to do with the duplicates
+- Private and publishable data downloads in Excel and CSV formats
+  - Mark data fields as sensitive to exclude them from publishable products
+  - Version control on products, e. g., FormShare will let you know if a data export does not have the latest submissions or changes in the database
+  - Make publishable products publicly available knowing that sensitive fields are excluded automatically 
+- Download geo-referenced information in KML format
+
+**Short-term features:**
+
+- Case management (Longitudinal data collection)
+- Connecting data fields with ontological variables. This is useful when comparing variables across studies even if variable names are different
+- Graph visualization with dashboards
+- Real-time data aggregation (pull data from different forms into one common data bucket). This is useful when dealing with slightly different forms for different geographies but where that certain fields could be aggregated into a common pot for analysis
+- Real-time data cleaning scripts using R
+
 ScreenShot
 ----------
 
 ![Image](./Screenshot.png "FormShare home screen")
 
 Releases
 ------------
-The current stable release is 2.6.10 and it is available [here](https://github.com/qlands/FormShare/tree/stable-2.6.10) 
+The current stable release is 2.6.11 and it is available [here](https://github.com/qlands/FormShare/tree/stable-2.6.11) 
 
-The database signature for stable 2.6.10 is 8ad53c7df321
+The database signature for stable 2.6.11 is 8ad53c7df321
 
-The Docker image for stable 2.6.10 is 20201019
+The Docker image for stable 2.6.11 is 20201119
 
 Installation
 ------------
@@ -49,11 +89,11 @@ sudo apt-get install -y docker-compose
 
 # Collect the FormShare source code
 cd /opt
-sudo git clone https://github.com/qlands/FormShare.git -b stable-2.6.10 formshare_source
+sudo git clone https://github.com/qlands/FormShare.git -b stable-2.6.11 formshare_source
 
 # Copy the docker compose file from the source to a new directory
-sudo mkdir formshare_docker_compose_20201019
-sudo cp ./formshare_source/docker_compose/docker-compose.yml ./formshare_docker_compose_20201019/
+sudo mkdir formshare_docker_compose_20201119
+sudo cp ./formshare_source/docker_compose/docker-compose.yml ./formshare_docker_compose_20201119/
 
 # Make the directory structure for FormShare
 sudo mkdir /opt/formshare
@@ -76,11 +116,11 @@ sudo sysctl -w vm.max_map_count=262144
 echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.d/60-vm-max_map_count.conf
 
 # Download all the required Docker Images
-cd /opt/formshare_docker_compose_20201019
+cd /opt/formshare_docker_compose_20201119
 sudo docker-compose pull
 
 # Edit the docker-compose.yml file to set the MySQL root and FormShare admin passwords
-sudo nano /opt/formshare_docker_compose_20201019/docker-compose.yml
+sudo nano /opt/formshare_docker_compose_20201119/docker-compose.yml
 # Press Alt+Shit+3 to show the line numbers in Nano
 
 Edit line 7: Change the root password from "my_secure_password" to your password
@@ -92,6 +132,10 @@ Edit line 65: Change the IP address for the IP address of the machine running th
 
 # Save the file with Ctlr+o Enter . Exit with Ctrl+x
 
+# In AWS if you use MySQL >= 8 in a RDS service you need to add the following permissions to your RDS root user:
+# GRANT SESSION_VARIABLES_ADMIN ON *.* TO 'my_RDS_root_user'@'%';
+# GRANT SYSTEM_VARIABLES_ADMIN ON *.* TO 'my_RDS_root_user'@'%';
+
 # Install Apache Server
 sudo apt-get install -y apache2
 
@@ -124,7 +168,7 @@ sudo service apache2 start
 # Subsequent start will take about 2 minutes. You can check the status with "sudo docker stats". 
 # FormShare will be ready for usage when the container reaches more than 500 kB of MEM USAGE
 # This is the only two commands you need to start FormShare after a server restart
-cd /opt/formshare_docker_compose_20201019
+cd /opt/formshare_docker_compose_20201119
 sudo docker-compose up -d
 
 # Browse to FormShare

docker_compose/docker-compose.yml

@@ -1,8 +1,8 @@
 version: '3'
 services:
-  fsmysql_20201019:
+  fsmysql_20201119:
     image: mysql:5.7.27
-    container_name: fs_mysql_20201019
+    container_name: fs_mysql_20201119
     environment:
       MYSQL_ROOT_PASSWORD: my_secure_password
     volumes:
@@ -11,9 +11,9 @@ services:
       fsnet:
         ipv4_address: 172.28.1.5
 
-  fselasticsearch_20201019:
+  fselasticsearch_20201119:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch_20201019
+    container_name: fs_elasticsearch_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -29,9 +29,9 @@ services:
       fsnet:
         ipv4_address: 172.28.1.1
 
-  fselasticsearch2_20201019:
+  fselasticsearch2_20201119:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch2_20201019
+    container_name: fs_elasticsearch2_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -48,9 +48,9 @@ services:
       fsnet:
         ipv4_address: 172.28.1.2
 
-  formshare_20201019:
-    image: qlands/formshare2:20201019
-    container_name: formshare_20201019
+  formshare_20201119:
+    image: qlands/formshare2:20201119
+    container_name: formshare_20201119
     environment:
       MYSQL_HOST_NAME: 172.28.1.5
       MYSQL_USER_NAME: root

docker_compose_with_apache/docker-compose.yml

@@ -1,8 +1,8 @@
 version: '3'
 services:
-  fsmysql_20201019:
+  fsmysql_20201119:
     image: mysql:5.7.27
-    container_name: fs_mysql_20201019
+    container_name: fs_mysql_20201119
     environment:
       MYSQL_ROOT_PASSWORD: my_secure_password
     volumes:
@@ -11,9 +11,9 @@ services:
       fsnet:
         ipv4_address: 172.28.1.5
 
-  fselasticsearch_20201019:
+  fselasticsearch_20201119:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch_20201019
+    container_name: fs_elasticsearch_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -28,9 +28,9 @@ services:
     networks:
       fsnet:
         ipv4_address: 172.28.1.1
-  fselasticsearch2_20201019:
+  fselasticsearch2_20201119:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch2_20201019
+    container_name: fs_elasticsearch2_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -46,9 +46,9 @@ services:
     networks:
       fsnet:
         ipv4_address: 172.28.1.2
-  fselasticsearch3_20201019:
+  fselasticsearch3_20201119:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch3_20201019
+    container_name: fs_elasticsearch3_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -64,9 +64,9 @@ services:
     networks:
       fsnet:
         ipv4_address: 172.28.1.3
-  formshare_20201019:
-    image: qlands/formshare2:20201019
-    container_name: formshare_20201019
+  formshare_20201119:
+    image: qlands/formshare2:20201119
+    container_name: formshare_20201119
     environment:
       MYSQL_HOST_NAME: 172.28.1.5
       MYSQL_USER_NAME: root
@@ -97,9 +97,9 @@ services:
       fsnet: 
         ipv4_address: 172.28.1.4
 
-  fsapache_20201019:
+  fsapache_20201119:
     image: qlands/httpd:2.4
-    container_name: fsapache_20201019
+    container_name: fsapache_20201119
     ports:
       - 80:80
     networks:

docker_compose_with_fluentd/docker-compose.yml

@@ -1,8 +1,8 @@
 version: '3'
 services:
-  fsmysql_20201019:
+  fsmysql_20201119:
     image: mysql:5.7.27
-    container_name: fs_mysql_20201019
+    container_name: fs_mysql_20201119
     environment:
       MYSQL_ROOT_PASSWORD: my_secure_password
     volumes:
@@ -11,9 +11,9 @@ services:
       fsnet:
         ipv4_address: 172.28.1.5
 
-  fselasticsearch_20201019:
+  fselasticsearch_20201119:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch_20201019
+    container_name: fs_elasticsearch_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -28,9 +28,9 @@ services:
     networks:
       fsnet:
         ipv4_address: 172.28.1.1
-  fselasticsearch2_20201019:
+  fselasticsearch2_20201119:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch2_20201019
+    container_name: fs_elasticsearch2_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -46,9 +46,9 @@ services:
     networks:
       fsnet:
         ipv4_address: 172.28.1.2
-  fselasticsearch_202010193:
+  fselasticsearch_202011193:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
-    container_name: fs_elasticsearch3_20201019
+    container_name: fs_elasticsearch3_20201119
     environment:
       - cluster.name=docker-cluster
       - bootstrap.memory_lock=true
@@ -64,9 +64,9 @@ services:
     networks:
       fsnet:
         ipv4_address: 172.28.1.3
-  formshare_20201019:
-    image: qlands/formshare2:20201019
-    container_name: formshare_20201019
+  formshare_20201119:
+    image: qlands/formshare2:20201119
+    container_name: formshare_20201119
     environment:
       MYSQL_HOST_NAME: 172.28.1.5
       MYSQL_USER_NAME: root
@@ -97,9 +97,9 @@ services:
       fsnet: 
         ipv4_address: 172.28.1.4
 
-  fsfluentd_20201019:
+  fsfluentd_20201119:
     image: qlands/fluentd:v1.7.3-1.0
-    container_name: fsfluentd_20201019
+    container_name: fsfluentd_20201119
     environment:
       WAIT_HOSTS: 172.28.1.1:9200, 172.28.1.4:5900
       WAIT_HOSTS_TIMEOUT: 120
@@ -110,9 +110,9 @@ services:
       fsnet:
         ipv4_address: 172.28.1.6
 
-  fskibana_20201019:
+  fskibana_20201119:
     image: docker.elastic.co/kibana/kibana:6.3.2
-    container_name: fskibana_20201019
+    container_name: fskibana_20201119
     environment:
       ELASTICSEARCH_URL: http://172.28.1.1:9200
     ports:

docker_files/formshare/stable.Dockerfile

@@ -24,7 +24,7 @@ VOLUME /opt/formshare_odata
 RUN mkdir formshare_gunicorn
 RUN python3 -m venv formshare_env
 
-RUN git clone https://github.com/qlands/FormShare.git -b stable-2.6.10 formshare
+RUN git clone https://github.com/qlands/FormShare.git -b stable-2.6.11 formshare
 RUN . ./formshare_env/bin/activate && pip install wheel && pip install -r /opt/formshare/requirements.txt && python /opt/formshare/download_nltk_packages.py
 
 ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.6.0/wait /wait

formshare/plugins/helpers.py

@@ -78,7 +78,7 @@ def get_version():
     This returns the version of FormShare
     :return: The version of FormShare
     """
-    return "2.6.10 (20201019)"
+    return "2.6.11 (20201119)"
 
 
 @core_helper

install_steps.txt

@@ -130,7 +130,7 @@ sudo python3 -m venv formshare_env
 # Grab the FormShare source code
 # ---------------------BEGIN OF IMPORTANT NOTE-------------------------
 # To use the stable version of FormShare do:
-sudo git clone https://github.com/qlands/FormShare.git -b stable-2.6.10 formshare
+sudo git clone https://github.com/qlands/FormShare.git -b stable-2.6.11 formshare
 # To use the development version of FormShare do:
 sudo git clone https://github.com/qlands/FormShare.git formshare
 # ---------------------END OF IMPORTANT NOTE-------------------------

setup.py

@@ -159,7 +159,7 @@
 
 setup(
     name="formshare",
-    version="2.6.10",
+    version="2.6.11",
     description="FormShare",
     long_description=README + "\n\n" + CHANGES,
     classifiers=[