Artifact Stats
The artifact stats metrics are a collection of sanity check metrics. They include things like binary size, memory usage on start, and compile time. These will give insight into whether something is off with the system. Eventually, these metrics will also be useful to compare NoisePage with other DBMSs.
-
Artifact Stats Script
- Python
- Performance Storage Service
- Django
- Time-Series Database
- TimescaleDB
- Data Visualization
- Grafana
The Jenkins pipeline executes the artifact stats python script which runs a "collectors". The collectors measure the different artifact metrics and all these metrics are bundled together. The bundle of metrics is sent to the performance storage service via a RESTful API call. The performance storage service validates the payload, converts it into a format suitable for storage, and stores it in the time-series database. Grafana queries the database in order to display visualizations of the results.
The following is a chart show the code dependencies in the performance storage service, relating to the artifact stats.
See the Timescaledb Schema wiki for details about the schema.
One of the most important quality attributes for this feature was modifiability. We wanted to make it as simple as possible for a CMU-DB developer to collect new metrics about the system. We designed the artifact stats script in a way to minimize code changes when collecting a new metric. In order to collect a new metric the developer just needs to add a new class that extends the BaseCollector class. Any metrics that the new class collects will automatically be sent to the performance storage service without needing to make any changes to the service or the database schema. For more information about adding new collectors check out the artifact stats README.md
In addition to not needing to change the database or API service, the dashboard is also dynamic. That means it will display all metrics that are collected without any changes to the dashboard. The downside to this design decision is that all the metrics must use numeric values and the units for the axis in the dashboard charts must be generic. Due to this contains we recommend naming the metrics with a unit in the metric key (i.e. compile_time_sec).