Transferable Adversarial LFW (TALFW) Database


Description

Welcome to Transferable Adversarial LFW (TALFW) Database, a renovation of Labeled Faces in the Wild (LFW), to evaluate the robustness of deep face recognition models, when the accuracy of LFW has been saturated.

Face recognition have achieved great success in the deep learning era. In unconstrained environment, face recognition performance is reaching saturation levels on the LFW database. However, the existence of transferable adversarial examples may severely hinder the robustness of deep face models. Based on transfer-based methods for generating black-box adversarial examples, we generate a transferable adversarial face database named Transferable Adversarial Labeled Faces in the Wild (TALFW).

In the recent literature, the LFW database has been widely used to evaluate the performance of deep face models by testing on the 3000 positive and 3000 negative face pairs, which involve 7,701 face images. Based on the greedy algorithm, we choose the minimum number of candidate face images to cover the maximum face pairs. Then we modify the candidate images in an imperceptible way. Compared with the original LFW database, 4,069 face images are modified. The perturbation is only 1.34, measured by root mean squared deviation. Compared with the original LFW database, the only difference is the imperceptible noise, therefore the evaluation protocol of the TALFW database is exactly the same as the LFW database, which makes it an easy-to-use and outstanding test database for the community.

Ten positive (left) and ten negative (right) face pairs in the TALFW database are shown.


Experimental Results

In black-box setting without queries, we evaluate four commercial APIs and four state-of-the-art (SOTA) algorithms and discover the significant performance degradation on the TALFW database. The severe degradation clearly shows the vulnerability of deep face-recognition models even with massive training data.


Comparison of Accuracy (%) on the LFW and TALFW databases of SOTA Deep Face Models.

Method LFW TALFW
Centerface1 98.78% 70.65%
SphereFace2 99.27% 62.47%
VGGFace23 99.43% 71.47%
ArcFace4 99.82% 63.45%

Comparison of Accuracy (%) on the LFW and TALFW databases of Commercial APIs.

Method LFW TALFW
Amazon5 98.47% 69.28%
Microsoft6 98.12% 70.93%
Baidu7 97.72% 72.07%
Face++8 96.95% 73.90%
Fusion of four APIs 99.65% 72.33%

Comparison of Accuracy (%) on the LFW and TALFW databases of Defensive Methods.

    (There has been a consensus that the improvement in robustness would bring performance degradation on clean test images. Since we aim to evaluate the robustness of deep face models while at the same time keeping the recognition performance of original images at a relatively high level, we check the performance of defensive models on the TALFW database while keep the accuracy on the LFW database no less than 99%.)

Method LFW TALFW
No Defense9 99.78% 54.15%
JPEG Encoding10 99.55% 73.93%
Gaussian Blur10 99.57% 77.95%
Adversarial Training11 99.62% 82.17%
  1. Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In ECCV, 2016.
  2. Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017.
  3. Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In FG, 2018.
  4. Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
  5. Amazon’s Rekognition Tool. https://aws.amazon. com/rekognition/.
  6. Microsoft Azure. https://www.azure.cn.
  7. Baidu Cloud Vision Api. http://ai.baidu.com.
  8. Face++ Research Toolkit. https://www.faceplusplus.com.cn/.
  9. ResNet-50 model trained on MS-Celeb-1M using ArcFace.
  10. Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In ICLR Workshop, 2017.
  11. Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR, 2017.

Download the database

This database is publicly available. Now, we provide: (1) the original images(250x250) and (2) the aligned images(112x112) using Baidu Netdisk, Google Drive and Dropbox.

Please send an email to Email1 and Email2 for files of this database.

For more details of the TALFW database, please refer to the paper "Towards Transferable Adversarial Attack against Deep Face Recognition".


Reference

Please cite as:


Yaoyao Zhong and Weihong Deng, “Towards Transferable Adversarial Attack against Deep Face Recognition”, IEEE Transactions on Information Forensics and Security, 2020.

BibTeX entry: 
@article{zhong2020towards,
  author    = {Zhong, Yaoyao and Deng, Weihong},
  title     = {Towards transferable adversarial attack against deep face recognition},
  journal   = {IEEE Transactions on Information Forensics and Security},
  year      = {2020},
  doi       = {10.1109/TIFS.2020.3036801}
}

Contact

Please contact Yaoyao Zhong and Weihong Deng for questions about the database.