The current project is the sum of all work of Group 7's project for the first assignment of the Software Engineering Processes (XB_0089) course.
The presentation of the project in Google Slides is available here.
The project of choice is Babel. Latest release of the project as of Jun 11th, 2024, is v2.15.0.
_"Babel is a Python library that provides an integrated collection of utilities that assist with internationalizing and localizing Python applications (in particular web-based applications.)"
from Babel's README.md
| Requirement | Babel |
|---|---|
| Hosted on GitHub | Repository Link |
| Open Source License | BSD-3-Clause |
| Automated unit tests | Test Directory |
| Existing branch coverage < 100% | 89% |
| Contributors | 150 |
| Lines of code | 17 KLOC 11 KLOC (excluding tests) |
We calculate the project's total coverage using the coverage.py tool.
We first set up the project's environment, following its documentation and install the necessary dependencies.
make setup_sepOr without
make:virtualenv venv > /dev/null cp -r sep_coverage venv/lib/python3.12/site-packages/ (. ./venv/bin/activate && pip install -r requirements.txt && export PYTHONPATH=$(realpath venv/lib/python3.12/site-packages/) && python setup.py import_cldr && pip install --editable .) > /dev/null 2>&1
A virtual environment venv is required to run coverage. We first installed virtualenv and pip using each operating system's package manager (either brew or pacman).
After that, we copy the local package sep_coverage (created by ourselves) to the virtual environment's site-packages directory (further expanded in Own coverage tool section).
⚠️ WARNING: The setup assumes that the latest version of python (3.12) is installed on the system, as well as thevirtualenvandpippackages. If this is not the case, the setup will fail.
The project's documentation was followed to install the project's dependencies and import the CLDR (Common Locale Data Repository) data.
However, it does not mention the need to set the PYTHONPATH environment variable to the virtual environment's site-packages directory.
This is to ensure that some tests that create a separate virtual environment will maintain the correct path to the sep_coverage package.
Also, there are some external packages that need to be installed.
The entire process is summed up in the setup_sep target of the Makefile.
make coverage_externOr without
make:. ./venv/bin/activate && coverage run --omit='sep_coverage/*','tests/*' -m pytest && coverage report > coverage_report.txt cat coverage_report.txt
We first enter the python virtual environment by sourcing the activate script in the venv directory.
The coverage.py tool has a run command that runs the coverage test and a report command that generates a report of the coverage in plain text.
These two commands will represent our two-step process in running the coverage.
The run command takes one argument, --omit, which is used to specify which files to omit from the coverage report and another argument -m pytest to run the tests using the pytest module.
We are using pytest to run the tests that generate the coverage for the project. Pytest is also the testing framework used by the project itself.
IMPORTANT: We omit the files in the sep_coverage/ package and the tests/ directory from the coverage report. This is because the sep_coverage package is our own coverage tool, and the tests directory contains are not of interest for the purposes of coverage, as the tests themselves generate the coverage. We are only interested in the coverage of the babel package.
Finally, we generate the coverage report using the report command.
The entire process is summed up in the coverage_extern target of the Makefile.
The entire result file, containing coverage data for every single file in the project, can be found in report/extern_coverage_before.txt.
| Total Statements | Total Missed | Coverage |
|---|---|---|
| 4526 | 490 | 89.17% |
We create our own coverage tool, sep_coverage, using manual instrumentation to measure the coverage of some parts of the project.
The tool is documented under sep_coverage/README.md.
⚙️ Prerequisite: Setup (only has to be done once)
make coverage_sepor without
make:. ./venv/bin/activate && python3 sep_coverage.py
The sep_coverage tool is built as a python package under the sep_coverage/ directory.
Because of this, it must be copied under the virtual environment's site-packages directory to be imported by the project.
At each instrument() call, the tool stores the identifier of the statement in a set mapped to each entity (file, class, function, method) that the statement belongs to.
Each entity is indexed in sep_coverage/program_entity.py, containing a range of statements that it covers.
After execution, we can use the set of each entity to calculate how many statements were reached, as the length of the set. The total number of statements is equal to the upper bound of the range, minus the lower bound plus one.
We can use this information to calculate the coverage of each entity, given the following formula, where C = coverage, M = total statements, N = total missed statements:
When the tool is run, the coverage test is executed, and then the coverage data is printed to stdout.
Each team member look at the coverage report of generated by coverage.py (specifically the html report created by running coverage html) and choose at least two functions or methods with low coverage to instrument.
The html coverage report page provides useful information, such as which exact lines are not covered by the tests.
Each function, method, class (Entity) is instrumented by the following syntax:
import sep_coverage
sep_coverage.instrument([ENTITY_FLAG_1, ENTITY_FLAG_2, ...], instrument_id)This method allows us to store the following coverage information for each instrumented line of code:
- Which entity the line belongs to (file, class, function or method)
- The instrument identifier of the line
- Whether the line was executed during the tests
The coverage report generated by our tool, before the coverage improvement can be found in report/sep_coverage_before.txt.
| Total Statements | Total Missed | Coverage |
|---|---|---|
| 57 | 25 | 56.14% |
Since we are only instrumenting a small part of the project, the coverage is expected to be low. We specifically targeted parts of the code that were not covered by the tests, in order to work on improving coverage in the next step.
| Member | Entities Instrumented | Total lines of code |
|---|---|---|
| Daniel | FixedOffsetTimezone _locales_to_names() |
14 |
| Gleb | get_timezone() _get_time() |
26 |
| Mateusz | _get_tzinfo() _get_tzinfo_or_raise() _get_tzinfo_from_file() |
49 |
| Radu | format_date() __str__() |
38 |
Daniel Halasz
Mateusz Kwiatkowski
Radu Nicolae
Gleb Mishchenko
Below is the evidence (code) that each student created/enhanced 2 tests, repsectively to their instrumented functions.
Daniel Halasz
Mateusz Kwiatkowski
Radu Nicolae
Gleb Mishchenko
Each team member after instrumenting their respective functions, and after looking at the coverage report of generated by coverage.py implemented specific tests to increase the coverage of these functions.
Each function got tested in a distinct, specific way, unique to its purpose. For example in tests_mateusz.py the coverage of the _get_tzinfo_or_raise() function is achieved by making sure certain errors and assertions are raised (or not).
with pytest.raises(LookupError):
_helpers._get_tzinfo_or_raise("Continent/City")
tzinfo = _helpers._get_tzinfo_or_raise("America/New_York")
assert tzinfo is not None
assert tzinfo.zone == "America/New_York"This method allowed each team member to successfully increase the overall coverage of their chosen functions, as, for example, more statements and if-else branches are reached after implementing the tests.
The correctness of our tests is presented with the following screenshot, after running pytest with the new tests.
The coverage report generated by our tool before the coverage improvement is summarized in the following table
| Total Statements | Total Missed | Coverage |
|---|---|---|
| 57 | 25 | 56.14% |
Since we specifically targeted parts of the code that were not covered by the general tests, and each team member managed to increase their instrumented function coverage almost to its fullest, we report the following coverage after implementing the enhanced tests
| Total Statements | Total Missed | Coverage |
|---|---|---|
| 57 | 4 | 92.98% |
This improvement is also showcased in the following screenshot. After running the suggested tool, coverage.py, on the initial directory and the directory with the ehnanced tests, we compared the two reports with diff.
With this we conclude that the total overall improvement that we reached amounted to 1.12%. The following screenshots of running coverage.py without and with the enhanced tests show the total coverage measured before and after.
