AI systems are no longer just responding to prompts — they’re acting, adapting, and making decisions. This week’s stories reveal how traditional security tools like SIEM, firewalls, and EDR fail to protect GenAI and Agentic AI systems, and why new approaches like continuous AI Red Teaming, identity enforcement, and jailbreak simulation are becoming essential.
Thts why the new A new guide from respected CSA (Cloud Security Allance) introduces a specialized AI Red Teaming framework for Agentic AI systems, and Adversa AI were the active contributors and reviewers of this framework.
The future of secure AI depends on proper security validation, real-time governance, observability, and readiness — not just post-incident response
SC Media, May 27, 2025
A detailed expert column outlines why traditional security tools like firewalls, EDR, and SIEM are inadequate for securing AI systems such as GenAI workflows, LLMs, and Agentic AI — and offers practical alternatives to address emerging AI-specific threats.
Traditional tools weren’t designed to detect or mitigate AI-specific risks like prompt injection, data poisoning, model theft, or autonomous decision drift. As AI systems become more powerful and widely integrated, security blind spots grow — requiring a fundamental shift from legacy tooling to AI-aware defenses, monitoring, and governance frameworks.
How to deal with it:
— Integrate security throughout the MLSecOps lifecycle using frameworks like OWASP LLM Top 10, MITRE ATLAS, and NIST AI RMF.
— Deploy AI-specific tools for AI Red Teaming, model scanning, prompt monitoring, drift detection, and policy enforcement.
— Build cross-functional teams to ensure shared visibility between data science, engineering, and security units.
Cloud Security Alliance, May 28, 2025
A new guide introduces a specialized Red Teaming framework for Agentic AI systems, offering concrete methods to test vulnerabilities such as permission escalation, hallucination, memory manipulation, orchestration flaws, and supply chain risks across complex AI agents.
The Adversa AI team was proud to contribute to this extremely imporyant and timely initiative.
Agentic AI systems operate autonomously — planning, reasoning, and acting without direct user input — which introduces new security challenges traditional red teaming cannot address. As these agents enter enterprise and critical infrastructure environments, continuous red teaming becomes essential to validate context boundaries, detect hidden behaviors, and reduce systemic risk.
How to deal with it:
— Adapt AI Red Teaming procedures to test full AI agent workflows, inter-agent dependencies, and real-world failure scenarios.
— Validate enforcement of role boundaries, input/output integrity, and agent decision logic under stress using AI Red Teaming solutions specific for Agentic AI
— Use the guide’s test prompts and vulnerability mapping to simulate attacks and improve detection and containment strategies.
Computer Weekly, May 30, 2025
A deep-dive analysis of the organizational, technical, and security risks that arise when deploying Agentic AI — and why enterprises must implement strict guardrails, identity controls, and governance frameworks from the start to avoid failure.
As Agentic AI moves from pilots to enterprise integration, most projects fail due to a lack of access control, identity clarity, and observability. Without strong governance, agents risk acting unpredictably, accessing sensitive data without authorization, and creating feedback loops or decision errors that compound over time — especially in regulated or high-impact environments.
How to deal with it:
— Assign AI agents unique identities aligned with human and machine role-based access structures to prevent identity fragmentation.
— Build guardrails across model, tooling, and orchestration layers, ensuring agents access only relevant, contextual, and authorized data.
— Implement observability mechanisms to monitor agent reasoning, data flows, and compliance with governance policies from day one.
arXiv, May 26, 2025
A new academic framework, GUARD, introduces an automated system that role-plays malicious users to generate realistic jailbreak prompts in natural language, enabling systematic testing of LLM guideline adherence and model robustness against misuse.
Despite growing safety efforts, LLMs remain vulnerable to jailbreaks that bypass filters through well-crafted prompts. GUARD simulates realistic adversarial behavior using role-based LLM collaboration (e.g., Translator, Generator, Evaluator, Optimizer) and tests models like Vicuna, LLaMA, and even ChatGPT against ethical and regulatory guidelines. This helps evaluate whether safety mechanisms actually prevent harmful outputs — especially in real-world, human-style attacks.
How to deal with it:
— Integrate adversarial prompt generation frameworks like GUARD into your AI Red Teaming and evaluation pipeline to test ethical compliance.
— Assess model responses against regulatory guidelines using role-based simulation, not just static filter lists.
— Extend LLM testing beyond text to include multi-modal agents (e.g., vision-language models) and validate response behavior under naturalistic pressure.
arXiv, May 26, 2025
A large-scale study reveals that Red Teaming success against LLMs follows a scaling law based on capability gaps—showing that as models grow more powerful, human-like attackers become increasingly ineffective.
As LLMs grow in general and Agentic capability, AI Red Teaming becomes a weak-to-strong problem: attackers (even humans) fail when the model being tested is significantly more capable. The study shows that stronger models are more successful at jailbreaking, and that social science skills (e.g. persuasion) are better predictors of attack success than technical (STEM) performance. This highlights urgent needs: benchmarking manipulative ability, understanding attack dynamics, and preparing for a future where fixed-capability Red Teamers (like humans) may no longer suffice.
How to deal with it:
— Model and monitor capability gaps between LLMs and AI Red Teamers to assess the real strength of your testing process.
— Benchmark persuasive and manipulative abilities of LLMs—not just factual accuracy—to track emerging attack potential.
— Invest in scalable, automated AI Red Teaming frameworks that evolve alongside LLM capabilities and reflect human-like black-box threats.
Stay up to date with what is happening! Get a first look at news, noteworthy research and worst attacks on AI delivered right in your inbox.
The Institute for Critical Infrastructure Technology (ICIT) has published a new report that connects the OWASP-LLM Top 10 risks with real-world AI security practices. This is more than just a ...
