Summary:
- From a total of 408 masked face images provided from the codename “Round” acquisition system, Trueface correctly identified 386, giving a TIR (True Identification Rate) of 94.6%.
- The Trueface model did not experience much of a reduction in accuracy for masked face images, especially in comparison to some of the competitors’ models. Trueface significantly outperformed the median masked face performance of 77% on several of the acquisition systems, demonstrating that the Trueface algorithm is quite effective, even with obstructions to the face.
- From a total of 544 non-masked face images provided from the codename “Stone” acquisition system, Trueface correctly identified 534, giving a TIR (True Identification Rate) of 98.2%.
- The Maryland Test Facility used a Trueface production model that had not been trained on masked-faces so we expect the performances we have discussed here to improve in the future.
Each year, the U.S Department of Homeland Security (DHS) Science and Technology Directorate (S&T) hosts Biometric Technology Rallies. These rallies engage the biometrics industry and are meant to accelerate understanding of biometric and identity technologies, thus driving efficiencies and adoption by the DHS. Unlike other facial recognition benchmarks like NIST FRVT, LFW, and Megaface Challenge whose tests are run only on datasets of existing imagery, the MdTF Rallies collect data in the real world. Results are therefore much more akin to real-world environments vs results generated in a lab/sandbox.
In 2018, the rally tested biometric image (acquisition) systems, like podiums, booths and e-gates on their ability to capture images of participants. The 2019 rally tested acquisition and biometric matching algorithms (matching systems) on their combined ability to capture images and identify individuals.
In 2020, both acquisition and matching systems were tested but due to COVID-19 and the resulting changes to how we travel, matching systems were also judged on accuracy when subjects were wearing masks. Trueface was selected to participate in the 2020 Biometric Technology Rally as a face recognition matching system.
Overview of 2020 Biometrics Rally. Numbers shown in the graphic summarize median performance across all participants.
The 2020 rally was designed to emulate a high-throughput scenario such as a customs checkpoint where travelers debarking from an international flight or cruise could have their identity verified quickly while filling their customs application at an automated kiosk.
Trueface Results: Non-Masked Faces
Trueface TIR using different acquisition systems on unmasked faces.
From a total of 544 non-masked face images provided from the codename “Stone” acquisition system, Trueface correctly identified 534, giving a TIR (True Identification Rate) of 98.2%.
Trueface Results: Masked Faces
Trueface TIR using different acquisition systems on masked faces.
From a total of 408 masked face images provided from the codename “Round” acquisition system, Trueface correctly identified 386, giving a TIR (True Identification Rate) of 94.6%.
The Trueface model did not experience much of a reduction in accuracy for masked face images, especially in comparison to some of the competitors’ models. Trueface significantly outperformed the median masked face performance of 77% on several of the acquisition systems, demonstrating that the Trueface algorithm is quite effective, even with obstructions to the face.
Identification even with high obfuscation.
The Maryland Test Facility used a Trueface production model that had not been trained on masked-faces so we expect the performances we have discussed here to improve in the future. Trueface is continuously working towards improving its face recognition models in our ever-evolving social environment. The team is placing great emphasis on training with underrepresented demographics to further reduce bias.
Acknowledgment: “This publication is based upon work conducted under the U.S. Department of Homeland Security Cooperative Research and Development Agreement №20-TCBI-013.”
Disclaimer: The views and/or conclusions contained in this document are those of the author(s) and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security (DHS), and do not constitute a DHS endorsement of the equipment tested or evaluated.