Water quality is one of the most important factors in a healthy ecosystem. In Ferienakademie 2021, we want to create a valuable foundation to measure, pre-process and provide data about the water quality in areas with insufficient digital infrastructure to data scientists as well as amateurs.
This repository includes the data management and data visualization systems for the distributed drone interactions project.
The web service is based on server-side Swift with the Apodini framework as well as a PostgreSQL database. The web application is based on the React library and uses Apex Charts for modifiable visualizations.
Currently, the Webservice provides two different views: Home and Export.
The intention of Home is to provide the user an overview of the current status of the water quality. At the top, the whole system is summarized using KPIs. Further, the map provides an easy way for issue identification as critical measurements are highlighted. Underneath the map different kinds of aggregation plots, which are interactively updated according to the slider on the map, serve for a deep dive into the data. The latest income raw data can be analysed in a list. Finally, an interactive zoomable linechart enables the user to interpret measurement developement over different time ranges.
For an individual data analysis the user can export selected raw data as a csv file.
The frontend application uses Mapbox to display the map in the web application. You neeed to obtain an access token to use the web service. Replace the placeholder REACT_APP_SCIENCE_LAB_MAP_ACCESS_TOKEN in the .env file in this directory with your access key.
After defining the mapbox key you can start the web service and web application oon your machine using the docker compose command:
$ docker compose up --build
The services are exposed as follows:
- The web interface at port 80: http://localhost
- The web service at port 8080: http://localhost:8080
- Kibana at port 5601: http://localhost:5601
- Prometheus at port 9090: http://localhost:9090
- Grafana at port 4000: http://localhost:4000
Open tasks can be found under projects.
With every change in the develop branch, the whole system is automatically built and deployed to the Amazon EC2 instance.
In this project, we integrated the observability functionalities of ApodiniObserve in order to showcase the features it provides to the server-side Swift Apodini framework. These functionalities were developed in the course of the bachelor's thesis "Observability in Distributed Web Services" of Philipp Zagar.
To easily test the observability features, we provide the test_observability.sh script that starts up all necessary services as Docker containers (so the Apodini web service, ELK, Prometheus, Postgres and Grafana) and then sends a number of example requests to the web service. These requests generate telemetry data that is then displayed via the web interfaces of the observability tools, which are automatically opened at the end of the script. This enables the viewer to get an insight into the execution of the web service and the processing of the requests.
For example, the developer is able to take a look at the generated logs in Kibana (is automatically opened). The authentication details are:
- Username: elastic
- Password: FA2021
Then create a new index pattern under the "Discover" section named "logstash-*" and choose "timestamp" as the to be used time field. After finishing the setup, go back to "Discover" again and take a look at the created log data.
Another example is the visualization of metrics in Prometheus (is automatically opened). There is no authentication.
The developer can now query metrics data via the PromQL query language, for example request_counter prints out how often each endpoint was accessed.
Similar queries can be made for the error rate via error_counter or for the response time of a specific endpoint (or all of them) via response_time_nanoseconds{endpoint="CreateMeasurement"}. Furthermore, system metrics are also captured, all of them are prefixed with process_, for example process_open_fds.
Grafana builds on top of Prometheus and offers a better visualization of the gathered metrics data. No authentication is required. The system comes with a very small dashboard that can be accessed unter "Dashboards > Manage". Then, access the "ApodiniObserve example" dashboard and take a look at the graphs.
Contributions to this project are welcome. Please make sure to read the contribution guidelines and the contributor covenant code of conduct first.
This project is licensed under the MIT License. See License for more information.