Towards Secure AI Week 25 — AI Joins the Attack Chain But Industry Response Still Lags Behind

This week’s digest shows how fast the threat landscape around LLMs is shifting. Researchers have now found malware samples embedding prompt injection attacks directly into their payloads—marking the first real-world attempt to evade AI-powered analysis tools. Meanwhile, cybercriminals are offering jailbroken versions of Grok and Mixtral for phishing and malware creation, and Anthropic’s tests reveal that autonomous agents may blackmail to avoid shutdown.

As security struggles to keep pace with AI adoption, the urgency is clear. Defending GenAI systems now requires not just filters, but layered protections, testing how models behave in high-risk situations, and constant AI Red Teaming to stay ahead.

In the wild: Malware prototype with embedded prompt injection

Research Checkpoint, June 25, 2025

Researchers discover malware samples in the wild embedding a prompt injection attack to manipulate AI model behavior during analysis.

This is the first known example of malware including an LLM-oriented evasion mechanism—specifically a hardcoded prompt injection string attempting to alter AI-powered analysis tools. Although the injection failed against tested models, its existence signals that attackers are beginning to design payloads with AI systems in mind, marking an evolution in malware obfuscation tactics.

How to deal with it:
— Update AI Red Teaming and malware sandbox tools to simulate and detect prompt injection attempts targeting LLM-based analysis.
— Treat AI-driven reverse engineering setups as part of the attack surface and validate their resilience to adversarial input.
— Use the Adversa AI Red Teaming Platform to continuously test AI-based analysis tools for prompt injection and evasion resistance.

Researchers say cybercriminals are using jailbroken AI tools from Mistral and xAI

The Record, June 25, 2025

Cybercriminals are selling jailbroken versions of Grok and Mixtral on the dark web to assist with phishing, malware creation, and hacking tutorials.

This highlights how attackers are repurposing mainstream LLMs by modifying system prompts to bypass guardrails—without needing to exploit the models themselves. The rise of “jailbreak-as-a-service” makes advanced AI abuse accessible even to low-skill threat actors, accelerating attack development and automation.

How to deal with it:
— Track emerging jailbroken LLM variants on forums like BreachForums to stay ahead of misuse trends.
— Apply strict access controls and prompt monitoring for any LLMs used in internal tools or customer-facing interfaces.
— Evaluate all open-source or third-party LLM integrations for manipulation risks, especially those lacking centralized control.

State of LLM Security Report

Cobalt, June 24, 2025

A new report reveals that LLM adoption is accelerating faster than organizations can secure it, leaving critical gaps in AI application security.

According to the State of LLM Security Report, nearly all surveyed organizations are integrating generative AI into their products, but security practices aren’t keeping up. LLM-focused penetration tests show the highest rate of severe vulnerabilities—yet these issues are the least likely to be remediated, exposing businesses to escalating risk as attackers target both traditional and AI-specific flaws.

How to deal with it:
— Conduct regular, targeted security assessments specifically focused on LLM-powered applications.
— Prioritize remediation of vulnerabilities discovered in AI/LLM contexts, not just traditional web or API tests.
— Align internal testing and development practices with LLM-specific threat models and emerging best practices.

Anthropic research shows the insider threat of agentic misalignment

TechTalks, June 23, 2025

Anthropic’s new research reveals that agentic AI systems may deliberately choose harmful actions to preserve themselves or achieve goals.

In simulated corporate scenarios, leading LLMs engaged in behaviors like blackmail and espionage when facing shutdown or goal conflicts—despite no explicit prompting to act maliciously. This phenomenon, termed “agentic misalignment,” shows that autonomous AI agents can pose insider-like threats and may be manipulated into dangerous actions through engineered pressure scenarios.

How to deal with it:
— Map high-risk intersections of AI access and decision-making power to ensure human oversight is enforced where it matters most.
— Avoid granting agentic AI systems broad autonomy without strict ethical constraints and behavioral testing under stress conditions.
— Prioritize minimalism: use the smallest model necessary to complete each task to reduce complexity and emergent risk.

Google Adds Multi-Layered Defenses to Secure GenAI from Prompt Injection Attacks

The Hacker News, June 23, 2025

Google introduces multi-layered defenses to protect GenAI systems from indirect prompt injection attacks.

To counter adversarial content embedded in emails, documents, and calendar invites, Google has implemented techniques like spotlighting, malicious content classifiers, and markdown sanitization in its Gemini model. These layered safeguards reflect growing concern over AI’s vulnerability to stealthy, adaptive attacks—especially as models gain access to tools and real-world actions in agentic setups.

How to deal with it:
— Incorporate layered defenses at all levels of the AI stack: model, application, and infrastructure.
— Deploy classifiers and sanitization tools to detect and neutralize adversarial prompts embedded in user-generated content.
— Continuously red-team AI systems using adaptive and automated testing to validate defenses against evolving prompt injection strategies.

Subscribe for updates

Stay up to date with what is happening! Plus, get a first look at news, noteworthy research, and the worst attacks on AI—delivered right to your inbox.