Bump stable to 2.24.0

README.md

@@ -25,59 +25,88 @@ FormShare **is for organizations** to install it in their server or cloud servic
 
 ## Features
 
-**Current features**
-
-- Case management (Longitudinal data collection) (Version >= 2.8.0)
-  - Using the Official ODK Collect App
-  - Intelligent work flow using Official ODK standards: 
-    - Case creator forms will create cases.
-    - Follow-up forms will attach information to each case.
-    - Deactivate forms will deactivate cases. For example, a household that decides to exit a longitudinal study will not appear in follow-up forms after deactivation.
-    - Activate forms will activate cases again.  For example, a household that decides to re-enter a longitudinal study will appear again in follow-up forms after activation.
-    - Move information from case creator forms into follow-up, deactivation and activation forms. For example, the sex of a participant (e. g., female) could be used in follow-up case forms to ask specific questions according to sex (e. g., if female, do they have access to reproductive health services since our last visit?)
-- Support multiple time zones:
-  - FormShare's: This is the time zone of the Linux server running FormShare.
-  - User: This is the time zone of a user logged into FormShare.
-  - Project: Each project can have a different time zone. This should be the time zone where submissions happen.
-  - Assistant: This is the time zone of an assistant logged into FormShare.
-  - Partner: This is the time zone of a partner logged into FormShare.
-
-- User accounts and management
-- Group-based user permissions
-- Projects to organize users, permissions, and forms
-- Separate access for data collectors and data cleaners. This is useful when dealing with hundreds of data collectors that do not need a FormShare user access 
-- Separate access for partners: Partners are trusted individuals outside your organization that require access to products and other resources in FormShare to collaborate with you. For example, a professor at a university (a partner in your project) requiring a KML export to match GPS points with weather data. 
-- User collaborations at project level, e.g., you can allow a colleague to maintain certain aspects of your project.
-- Form and submission management
-  - With easy setup of ODK Collect Projects using QR images
-  - With support for form version updates
-  - With testing and production stages
-  - Showing structural changes between incremental versions.
-  - With form and submission multimedia or data attachments
-  - With a table preview of submission data (even with thousand or millions of records) allowing in-table edits and recording any changes made to the data
-- OData live data feed for analysis with tools like Excel and Power BI. **With all CRUD operation supported (e. g., update)**. You can even use [Excel](https://github.com/qlands/MrBot-OData-Add-In) to clean data.
-- Extensibility system, e.g., You can write extensions to connect FormShare with Microsoft 365 authentication system
-- Documentation for running on AWS using Docker
-- Data cleaning API integration with R, STATA, or SPSS
-- Real-time map visualization of geo-referenced submissions at project and form level
-- Fill out forms using the web browser through [Enketo](https://github.com/qlands/formshare_enketo_plugin)
-- Download form attachments
-- Filtering submissions by submission metadata (e.g., date and time received on server)
-- Parse submissions into database tables. FormShare will create a MySQL data repository to store your data
-  - The repository is controlled with a primary key e.g., Farmed ID
-  - Duplicated submissions go to a cleaning pipe-line system that makes it easier to compare submissions and decide what to do with the duplicates
-- Private and publishable data downloads in Excel and CSV formats
-  - Mark data fields as sensitive to exclude them from publishable products
-  - Version control on products, e.g., FormShare will let you know if a data export does not have the latest submissions or changes in the database
-  - Make publishable products publicly available knowing that sensitive fields are excluded automatically 
-- Download geo-referenced information in KML format
-
-**Short-term features:**
-
-- Graph visualization with dashboards.
-- Connecting data fields with ontological variables. This is useful when comparing variables across studies even if variable names are different
-- Real-time data aggregation (pull data from different forms into one common data bucket). This is useful when dealing with slightly different forms for different geographies but where that certain fields could be aggregated into a common pot for analysis
-- Real-time data cleaning scripts using R
+### Case management (Longitudinal data collection) (Version >= 2.8.0)
+
+- Using the Official ODK Collect App.
+- Intelligent workflow using Official ODK standards: 
+  - Case creator forms will create cases.
+  - Follow-up forms will attach information to each case.
+  - Deactivate forms will deactivate cases. For example, a household that decides to exit a longitudinal study will not appear in follow-up forms after deactivation.
+  - Activate forms will activate cases again.  For example, a household that decides to re-enter a longitudinal study will appear again in follow-up forms after activation.
+  - Move information from case creator forms into follow-up, deactivation, and activation forms. For example, the sex of a participant (e. g., female) could be used in follow-up case forms to ask specific questions according to sex (e. g., if female, do they have access to reproductive health services since our last visit?)
+
+### Support for multiple time zones
+
+FormShare can produce dates and times in the following time zones:
+
+- FormShare: This is the time zone of the Linux server running FormShare.
+- User: This is the time zone of a user logged into FormShare.
+- Project: Each project can have a different time zone. This should be the time zone where submissions happen.
+- Assistant: This is the time zone of an assistant logged into FormShare.
+- Partner: This is the time zone of a partner logged into FormShare.
+
+### User and project management
+
+- User accounts and management.
+- Group-based user permissions.
+- Projects to organize users, permissions, and forms.
+- Separate access for data collectors and data cleaners. This is useful when dealing with hundreds of data collectors that do not need a FormShare user's account.
+- Separate access for partners: Partners are trusted individuals outside your organization that require access to products and other resources in FormShare to collaborate with you. For example, a professor at a university (a partner in your project) requires a KML export to match GPS points with weather data. 
+- User collaborations at a project level, e.g., you can allow a colleague to maintain certain aspects of your project.
+
+### Form and submission management
+
+- Easy setup of ODK Collect Projects using QR images.
+- Support for form version updates.
+- With testing and production stages.
+- Showing structural changes between incremental versions.
+- With form and submission multimedia or data attachments.
+- Table preview of submission data (even with thousands or millions of records) allowing in-table edits and recording any changes made to the data.
+- Fill out forms using the web browser through [Enketo](https://github.com/qlands/formshare_enketo_plugin).
+- Real-time map visualization of geo-referenced submissions at project and form levels.
+- Download form attachments.
+
+### Database management and interoperability 
+
+- Parse submissions into database tables. FormShare will create a MySQL data repository to store your data.
+  - The repository is controlled with a primary key e.g., Farmed ID.
+  - Duplicated submissions go to a cleaning pipeline system that makes it easier to compare submissions and decide what to do with the duplicates.
+- Filtering submissions by submission metadata (e.g., date and time received on server).
+- Data cleaning API integration with R, STATA, or SPSS.
+- OData real-time data feed for analysis with tools like Excel and Power BI. **With all CRUD operations supported (e. g., update)**. You can even use [Excel](https://github.com/qlands/MrBot-OData-Add-In) to clean data.
+- Data dictionary with personal information protection.
+
+### Product management
+
+- Private and publishable data downloads in Excel, CSV, and JSON formats.
+  - Publishable products exclude protected fields (personal information protection) automatically.
+  - Version control on products, e.g., FormShare will let you know if a data export does not have the latest submissions or changes in the database.
+- Download geo-referenced information in KML format.
+
+### Performance, concurrency, and parallelism
+
+Though FormShare with the default settings can handle a load that would fit most organizations, it can be configured to handle any load. FormShare has been tested, using [JMeter](https://jmeter.apache.org/), under extreme traffic (1000 parallel submissions ) in an eight years old laptop with 16 GB of RAM and an [i7-4712HQ](https://www.intel.com/content/www/us/en/products/sku/78932/intel-core-i74712hq-processor-6m-cache-up-to-3-30-ghz/specifications.html).
+
+- Each submission is transactional. This means that a submission is either processed completely or thoroughly discarded for ODK Collect to re-send it.
+- Under extreme traffic,  FormShare can store a complex survey like [RHoMIS](https://www.rhomis.org/) at 18 submissions per second with zero failures.
+- Data exports support concurrent processing. A survey like RHoMIS with 100,000 submissions and millions of rows would take less than a minute to export to JSON or CSV.
+- The performance of the user interface or the data cleaning interface is not affected by the number of submissions.
+
+### Analytics
+
+- Real-time analytics from R using the [FormShare R Package](https://cran.r-project.org/web/packages/FormShare/index.html).
+- Real-time analytics from platforms like [Tableau](https://public.tableau.com/en-us/s/) and [Power BI](https://powerbi.microsoft.com/en-us/) using the [FormShare OData extension](https://github.com/qlands/formshare_odata_plugin).
+
+### Extensibility and others
+
+- Extensibility system, e.g., You can write extensions to connect FormShare with Microsoft 365 authentication system. See a list of extensions [below](#Customization-and-Extension).
+- Documentation for running on AWS using Docker.
+
+### Future features
+
+- Simple graph visualization with dashboards.
+- Connecting data fields with ontological variables. This is useful when comparing variables across studies even if variable names are different.
+- Real-time data aggregation (pull data from different forms into one common data bucket). This is useful when dealing with slightly different forms for different geographies but where that certain fields could be aggregated into a common pot for analysis.
 
 ScreenShot
 ----------
@@ -86,11 +115,11 @@ ScreenShot
 
 Releases
 ------------
-The current stable release is 2.23.0 and it is available [here](https://github.com/qlands/FormShare/tree/stable-2.23.0) 
+The current stable release is 2.24.0 and it is available [here](https://github.com/qlands/FormShare/tree/stable-2.24.0) 
 
-The database signature for stable 2.23.0 is e2d6d5aa3700
+The database signature for stable 2.24.0 is e2d6d5aa3700
 
-The Docker image for stable 2.23.0 is 20221219
+The Docker image for stable 2.24.0 is 20221229
 
 Installation
 ------------
@@ -110,9 +139,9 @@ sudo apt-get install -y docker-compose
 
 # Get the Docker Compose file
 cd /opt
-sudo mkdir formshare_docker_compose_20221219
-cd formshare_docker_compose_20221219
-sudo wget https://raw.githubusercontent.com/qlands/FormShare/stable-2.23.0/docker_compose/docker-compose.yml
+sudo mkdir formshare_docker_compose_20221229
+cd formshare_docker_compose_20221229
+sudo wget https://raw.githubusercontent.com/qlands/FormShare/stable-2.24.0/docker_compose/docker-compose.yml
 
 # Make the directory structure for FormShare
 sudo mkdir /opt/formshare
@@ -137,11 +166,11 @@ sudo sysctl -w vm.max_map_count=262144
 echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.d/60-vm-max_map_count.conf
 
 # Download all the required Docker Images
-cd /opt/formshare_docker_compose_20221219
+cd /opt/formshare_docker_compose_20221229
 sudo docker-compose pull
 
 # Edit the docker-compose.yml file to set the MySQL root and FormShare admin passwords
-sudo nano /opt/formshare_docker_compose_20221219/docker-compose.yml
+sudo nano /opt/formshare_docker_compose_20221229/docker-compose.yml
 # Press Alt+Shit+3 to show the line numbers in Nano
 
 Edit line 10: Change the root password from "my_secure_password" to your password
@@ -189,7 +218,7 @@ sudo service apache2 start
 # Subsequent start will take about 2 minutes. You can check the status with "sudo docker stats". 
 # FormShare will be ready for usage when the container reaches more than 500 kB of MEM USAGE
 # This is the only two commands you need to start FormShare after a server restart
-cd /opt/formshare_docker_compose_20221219
+cd /opt/formshare_docker_compose_20221229
 sudo docker-compose up -d
 
 # Browse to FormShare

docker_compose/docker-compose.yml

@@ -1,9 +1,9 @@
 version: '3'
 services:
-  fsmysql_20221219:
+  fsmysql_20221229:
     image: mysql:8.0.23
     command: --default-authentication-plugin=mysql_native_password
-    container_name: fs_mysql_20221219
+    container_name: fs_mysql_20221229
     cap_add:
       - SYS_NICE
     environment:
@@ -14,15 +14,15 @@ services:
       fsnet:
         ipv4_address: 172.28.1.5
 
-  fses20221219n01:
+  fses20221229n01:
     image: docker.elastic.co/elasticsearch/elasticsearch:7.14.2
-    container_name: fses20221219n01
+    container_name: fses20221229n01
     environment:
       - network.host=172.28.1.1
-      - node.name=fses20221219n01
+      - node.name=fses20221229n01
       - cluster.name=fs-es-cluster
-      - discovery.seed_hosts=fses20221219n02
-      - cluster.initial_master_nodes=fses20221219n01,fses20221219n02
+      - discovery.seed_hosts=fses20221229n02
+      - cluster.initial_master_nodes=fses20221229n01,fses20221229n02
       - bootstrap.memory_lock=true
       - xpack.security.enabled=false
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
@@ -36,15 +36,15 @@ services:
       fsnet:
         ipv4_address: 172.28.1.1
 
-  fses20221219n02:
+  fses20221229n02:
     image: docker.elastic.co/elasticsearch/elasticsearch:7.14.2
-    container_name: fses20221219n02
+    container_name: fses20221229n02
     environment:
       - network.host=172.28.1.2
-      - node.name=fses20221219n02
+      - node.name=fses20221229n02
       - cluster.name=fs-es-cluster
-      - discovery.seed_hosts=fses20221219n01
-      - cluster.initial_master_nodes=fses20221219n01,fses20221219n02
+      - discovery.seed_hosts=fses20221229n01
+      - cluster.initial_master_nodes=fses20221229n01,fses20221229n02
       - bootstrap.memory_lock=true
       - xpack.security.enabled=false
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
@@ -58,9 +58,9 @@ services:
       fsnet:
         ipv4_address: 172.28.1.2
 
-  formshare_20221219:
-    image: qlands/formshare2:20221219
-    container_name: formshare_20221219
+  formshare_20221229:
+    image: qlands/formshare2:20221229
+    container_name: formshare_20221229
     environment:
       MYSQL_HOST_NAME: 172.28.1.5
       MYSQL_USER_NAME: root

docker_files/formshare/build_stable.sh

@@ -1 +1 @@
-sudo docker build --no-cache --rm -t qlands/formshare2:20221219 -f ./stable.Dockerfile .
+sudo docker build --no-cache --rm -t qlands/formshare2:20221229 -f ./stable.Dockerfile .

docker_files/formshare/stable.Dockerfile

@@ -1,4 +1,4 @@
-FROM qlands/odktools:20221218
+FROM qlands/odktools:20221228
 
 MAINTAINER QLands Technology Consultants
 
@@ -30,7 +30,7 @@ COPY ./docker_files/mosquitto/access.acl /root
 RUN mkdir formshare_gunicorn
 RUN python3 -m venv formshare_env
 
-RUN git clone https://github.com/qlands/FormShare.git -b stable-2.23.0 formshare
+RUN git clone https://github.com/qlands/FormShare.git -b stable-2.24.0 formshare
 RUN . ./formshare_env/bin/activate && pip install wheel && pip install -r /opt/formshare/requirements.txt && python /opt/formshare/download_nltk_packages.py
 
 ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.6.0/wait /wait

formshare/plugins/helpers.py

@@ -113,7 +113,7 @@ def get_version():
     This returns the version of FormShare
     :return: The version of FormShare
     """
-    return "2.23.0 (20221219)"
+    return "2.24.0 (20221229)"
 
 
 @core_helper

install_steps.md

@@ -257,7 +257,7 @@ sudo chown -R $whoami formshare_env
 
 ```sh
 cd /opt
-sudo git clone https://github.com/qlands/FormShare.git -b stable-2.23.0 formshare
+sudo git clone https://github.com/qlands/FormShare.git -b stable-2.24.0 formshare
 sudo chown -R $whoami formshare
 ```
 

setup.py

@@ -173,7 +173,7 @@
 
 setup(
     name="formshare",
-    version="2.23.0",
+    version="2.24.0",
     description="FormShare",
     long_description=README + "\n\n" + CHANGES,
     classifiers=[