Driven by our mission to increase trust in AI, Adversa’s AI Red Team is constantly exploring new methods of assessing and protecting mission-critical AI applications.

Recently, we’ve discovered a new way of attacking Facial Recognition systems and decided to show it in practice. Our demonstration reveals that current AI-driven facial recognition tools are vulnerable to attacks that may lead to severe consequences.

There are well-known problems in facial recognition systems such as bias that could lead to fraud or even wrong prosecutions. Yet, we believe that the topic of attacks against AI systems requires much more attention. We aim to raise awareness and help enterprises and governments deal with the emerging problem of Adversarial Machine Learning.

We’ve developed a new attack on AI-driven facial recognition systems, which can change your photo in such a way that an AI system will recognize you as a different person, in fact as anyone you want.  

It happens due to the imperfection of currently available facial recognition algorithms and AI applications in general. This type of attack may lead to dire consequences and may be used in both poisoning scenarios by subverting computer vision algorithms and evasion scenarios like making stealth deepfakes.

The new attack is able to bypass facial recognition services, applications, and APIs including the most advanced online facial recognition search engine on the planet, called PimEyes, according to the Washington Post. The main feature is that it combines various approaches together for maximum efficiency.

This attack on PimEyes was built with the following methods from our attack framework:

• For better Transferability, it was trained on an ensemble of various facial recognition models together with random noise and blur.
• For better Accuracy, it was designed to calculate adversarial changes for each layer of a neural network and use a random face detection frame.
• For better Imperceptibility, it was optimized for small changes to each pixel and used special functions to smooth adversarial noise.

We follow a principle of responsible disclosure and currently making coordinated steps with organizations to protect their critical AI applications from this attack so we can’t release the exploit code publicly yet.

We present an example of how PimEyes.com, the most popular search engine for public images and similar to Clearview, a commercial facial recognition database sold to law enforcement and governments, has mistaken a man for Elon Musk in the photo.

The new black-box one-shot, stealth, transferable attack is able to bypass facial recognition AI models and APIs including the most advanced online facial recognition search engine PimEye.com.

You can see a demo of the “Adversarial Octopus” targeted attack below.

Uniquely, the attack is a black-box attack that was developed without any detailed knowledge of the algorithms used by the search engine, and the exploit is transferable to any AI application dealing with faces for internet services, biometric security, surveillance, law enforcement, and any other scenarios. 

The existence of such vulnerabilities in AI applications and facial recognition engines, in particular, may lead to dire consequences.

• Hacktivists may wreak havoc in the AI-driven internet platforms that use face properties as input for any decisions or further training. Attackers can poison or evade the algorithms of big Internet companies by manipulating their profile pictures.

• Cybercriminals can steal personal identities and bypass AI-driven biometric authentication or identity verification systems in banks, trading platforms, or other services that offer authenticated remote assistance. This attack can be even more stealthy in every scenario where traditional deepfakes can be applied.

• Terrorists or dissidents may secretly use it to hide their internet activities in social media from law enforcement. It resembles a mask or fake identity for the virtual world we currently live in.

The main feature of this attack is that it’s applicable to multiple AI implementations including online APIs and physical devices. It’s constructed in a way that it can adapt to the target environment. That’s why we call it Adversarial Octopus. Besides that, it shares three important features of this smart creature.

Unfortunately, there is no one-size-fits-all protection from such attacks due to the fundamental issues in deep learning algorithms. It’s a complex problem that involves multiple actions where each of them can reduce the risks of such attacks. AI isn’t born protected, you have to grow and train it with security in mind to make it Trusted as we grow our kids to be healthy. 

Here are the main steps to perform: 

1. Start with adversarial testing or AI Red Teaming. It’s an analog of traditional penetration testing for AI systems. It’s an absolutely necessary thing for any AI and is similar to hand washing to prevent viruses.

2. Develop and train AI with respect to adversarial scenarios by adversarial training, model hardening, data cleaning, and other defense methods, like training kids to do sport to stay healthy.

3. Learn about and detect new threats for AI in critical decisions making applications by constantly following the latest inventions in adversarial machine learning, like training kids for advanced skills if they plan to be involved in extreme sports activities.

In the wake of interest in practical solutions for ensuring AI system’s security against advanced attacks, we have developed our own technology for testing facial recognition systems for such attacks.

We are looking for early adopters and forward-thinking technology companies to partner with us on implementing adversarial testing capabilities in your SDLC and MLLC lifecycles to increase trust in your AI applications and provide your customers reliable AI service.