Architectural Design
This page details the architecture of the system and gives an overview of how the various parts of the solution fit together.
The vAirify application's main requirement is to highlight variations between the forecast and in-situ data for various pollutants. To this end it retrieves data from two external data providers, namely the Copernicus Atmosphere Monitoring Service (CAMS) for the pollutant forecast and OpenAQ for the in-situ data. There are plans to add additional data sources for the In-Situ data, but these are still being finalised.
The vAirify application also includes a web front end to allow ECMWF users various views of the data.
Zooming in on the vAirify application we can see the solution is made up of 4 applications, and 2 data stores.
Below is an explanation of each of these. They are currently all hosted on a single Linux machine running Docker and Nginx with Journald used for logging.
- vAirify UI : A dockerised React user interface to highlight differences between the actual and forecasted pollutants. This includes a summary page to show global patterns and give a breadth of information across multiple cities with AG Grid, as well as a more detailed city view to display finer grained information using eCharts. There is also a map showing the various measurement stations using MapLibre and OpenStreetMap. There is also a global map on the summary page of the application which was created using react-three-fiber to visualise data from the bitmap image store.
- vAirify API : A dockerised Python Fast API instance serving the UI with information from the Mongo DB store
- Bitmap Image Store : This is a store of forecast data that has been preprocessed from raw GRIB files into bitmap data textures; in effect pictoral snapshots of global pollutant data at various times. This allows us to overlay these images onto a world map to show the forecasted movement of pollutants on a global scale. For simplicity these are currently stored in a docker volume by the forecast ETL on the disk of the host machine. These are then referenced directly from, and served as static files by, the UI.
- Air_Quality_Dashboard_Db : A dockerised Mongo instance serving as the main data store for the application. All forecasted and in-situ data is stored here, alongside metadata for the mapped bitmap images, and the city locations the application supports.
- Forecast ETL : A dockerised Python script that calls CAMS to retrieve forecast data. This forecast data is download as a GRIB file which is then read and transformed for storage.
- In-Situ ETL : A dockerised Python script that calls OpenAQ to retrieve in-situ data. This data is read directly from an API call and the information transformed for storage. In order to transform some of the In-Situ data from parts per million (ppm) to micro-grams per metre cubed (µg/m³) a temperature and surface pressure reading is required. To obtain this a forecasted value for these is used, therefore requiring calling the CAMS API here too.
NB : All diagrams are created using Draw.io, and the raw drawio files can be found here. They are based on the C4 visualisation model.