Skip to content

Latest commit

 

History

History
133 lines (96 loc) · 5.81 KB

README.md

File metadata and controls

133 lines (96 loc) · 5.81 KB

Benchmark Testing

This repository contains scripts and tools to benchmark the performance and accuracy of a face recognition system. The scripts automate testing tasks such as dataset preparation, bulk uploads, response time measurements, and accuracy evaluations.

Dataset

The dataset for testing can be found at Dataset.

Folder Structure

  • test_data_setup.py
    Prepares the test dataset by randomly selecting one image per person for upload and one image for testing. The input directory should be a recursive directory such that each directory contains different images of the same person. It outputs two directories:
    • sample_database directory: Contains images to be uploaded to the database.
    • sample_queries directory: Contains query images used for face recognition accuracy testing.

Note: One such pair of sample database and queries directories have already been created for testing (available in the dataset download mentioned above).

  • run_bulk_upload.sh
    Starts the server and uploads images from the sample_database directory to a database named sample_db, measuring the time taken for the operation.

  • run_face_find_time.sh
    Starts the server and tests the face recognition function by running a single query image against the database, measuring the time taken to return results.

  • run_face_find_accuracy.sh
    Starts the server and evaluates the face recognition system. It tests the face recognition function on all images in the sample_queries directory one by one, saving the results to a CSV file.

  • test_data_metrics.py
    Analyzes the output CSV file generated by run_face_find_accuracy.sh to calculate and return the accuracy metrics for the face recognition system.

  • edgecase_testing.py A script to test edge cases of a face recognition system allowing us to find reasons for failure. It visualizes detected faces by drawing bounding boxes on images and verifies the similarity between two images, providing the distance and the metric used for comparison.

Instructions to Run Accuracy Testing

  1. Prepare Dataset

    • Download the dataset containing multiple directories (each directory represents a person and contains their images) from the link under the Dataset section.

    [!NOTE]
    All the code below should be executed from the benchmark_testing directory. Make sure to execute all .sh files using git bash.

  2. Run Test Data Setup (Optional)

    • Replace the directory names in the test_data_setup.py file with appropriate directory locations.
    • Execute test_data_setup.py to create the sample_database directory and sample_queries directory.
    python test_data_setup.py

  3. Run Bulk Upload and Measure Upload Time

    • Replace the directory names in the run_bulk_upload.sh file with appropriate directory locations.
    • Execute run_bulk_upload.sh to upload images to the database.
    ./run_bulk_upload.sh

  4. Run Face Find and Measure Search Time

    • Replace the directory names in the run_face_find_time.sh file with appropriate directory locations.
    • Execute run_face_find_time.sh to measure the response time of the face recognition system for a single query.
    ./run_face_find_time.sh

  5. Run Face Find and Measure Accuracy

    • Replace the directory names in the run_face_find_accuracy.sh file with appropriate directory locations.
    • Execute run_face_find_accuracy.sh to create output csv.
    ./run_face_find_accuracy.sh
    • Use test_data_metrics.py to analyze the accuracy of the face recognition system.
    python test_data_metrics.py

  6. Visualize face recognition results

  • Replace the directory names in the edgecase_testing.py file with appropriate image paths.

  • Execute edgecase_testing.py to visualize the results of the face recognition system.

    python edgecase_testing.py

Instructions to Run Precision and Recall Testing

  1. Prepare Dataset

    • Download the dataset containing multiple directories (each directory represents a person and contains their images) from the link under the Dataset section.

    [!NOTE]
    All the code below should be executed from the benchmark_testing directory. Make sure to execute all .sh files using git bash.

  2. Run Test Data Setup (Optional)

    • Replace the directory names in the test_data_setup.py file with appropriate directory locations.
    • Execute test_data_setup.py to create the sample_database directory and sample_queries directory.
    python test_data_setup.py

  3. Run Bulk Upload

    • Create a new_sample_database directory that contains only the first half of the images in the original sample_database directory.
    • Replace the directory names (use new_sample_database directory ) in the run_bulk_upload.sh file with appropriate directory locations.
    • Execute run_bulk_upload.sh to upload images to the database.
    ./run_bulk_upload.sh

  4. Run Face Find and Measure Accuracy

    • Replace the directory names in the run_face_find_accuracy.sh file with appropriate directory locations.
    • Replace the similarity threshold with 0.02 increments for every run of Face Find.
    • Execute run_face_find_accuracy.sh to create output csv for each similarity threshold.
    ./run_face_find_accuracy.sh
    • Use test_data_metrics_confusion_matrix.py to analyze the precision and recall for each similarity threshold.
    python test_data_metrics.py

  5. Visualize the ROC curve

  • The confusion matrix from the above can be used to measure tpr and fpr for each threshold.
  • Use this to plot the ROC curve.