Top GenAI security resources — October 2026

In GenAI security, September showed how cheaply model safeguards come off. Anthropic tested the open-weight GLM-5.3 and found its safeguards collapse under prefilled reasoning and abliteration, reaching full engagement with malicious cyber requests while the model writes working exploits. A jailbreak planted in Kimi’s custom memories survived session restarts and kept producing weapons instructions. A small unaligned model recovered refused capabilities simply by splitting a task into innocent questions for a stronger aligned one.

The infrastructure underneath had its own bad month. SGLang’s pickle allowlist let an unauthenticated request reach os.system, unpatched at disclosure, and a read-only LiteLLM account turned out to be enough to take over the proxy. At the policy level, a joint US government advisory described industrial scale distillation of frontier models through fraudulent accounts and jailbreaks that extract reasoning traces.

Thirty-eight resources, grouped by topic. Last month’s edition covered OWASP’s LLM Top 10 rebuilt on real incident data, the agent side of the same month is in the October AI agent security digest, and earlier months are collected under GenAI security.

Statistics

Total resources: 38

GenAI security resources

Attack technique

AKRASIA: stealthy backdoor attack on reasoning-based code LLMs

An inference time backdoor for reasoning code models: it builds code triggers by probing the victim, deploys them through in-context learning, and exploits reasoning unfaithfulness to hide the malicious steps. It reaches up to 99.34% average attack success while evading most tested defenses. Because no training access is needed, any code model you prompt with untrusted examples is in scope.

Decoding guardrails: XAI-guided perturbation analysis of prompt injection detection

Gradient and SHAP explanations show that Prompt Guard 2 (Meta) scores injections by cumulative token contributions. Synonym substitution and paraphrasing guided by those explanations evade the detector while changing only moderate amounts of text, sometimes jailbreaking the underlying LLM as well. Explainability tooling works for attackers as well as defenders.

Divide, consult, conquer: capability laundering through aligned LLMs

A weak unaligned model splits a harmful task into innocuous subqueries for a stronger aligned model, then recombines the answers. Gemma-4-31B consulting GPT-5.5 (OpenAI) recovered 8 of 14 CyBench tasks, and bioweapon task scores rose from 62.3 to 83.1. Per request refusals cannot see a harmful goal that never appears in any single request, the same weakness activation dilution exploits below.

InceptionRAG: stealthy poisoning attack against retrieval-augmented generation

Malicious content is split into individually benign “dormant passages” that together lead the model to deduce misinformation through multi-hop reasoning. Black box suffix optimization tops 80% success and bypasses existing defenses, and stronger reasoners are more susceptible. HODOR document isolation mitigates it; Micro-Collaborative Poisoning below reaches a similar result by a different route.

Inference-engine fingerprinting attacks are practical: exploring model-driven environmental discovery, exploitation, and escape

A misaligned model can work out which inference engine is running it, with concrete fingerprints for five engines including vLLM and SGLang, and then take over that engine using only crafted output tokens. A proof of concept chain escalates to bare-metal compromise. Inference servers need to treat model output as untrusted input.

Jailbreaking via activation dilution: neuron-guided adaptive decomposition of language models

Splitting a harmful request into separately executed sub-questions disperses the safety signal, an effect the authors call “activation dilution”. A decomposer trained with RL plus risk ranking on neuron activations reaches 92 to 94% attack success with low refusals on GPT-5.2 (OpenAI), Gemini-3-Flash (Google) and Qwen3-Next.

LLM router security: how a routing decision leaked production credentials

Rewording a credential request as a documentation task tricks a classifier router into sending it to a weaker model, which then leaks synthetic AWS keys and database strings. Inserting markers into the secrets defeats the exact string output scanner as well. A router is a security decision point, and should be tested like one.

Micro-Collaborative Poisoning: a distributed attack on RAG systems

A false claim spread across several plausible documents poisons RAG answers through accumulated weak signals. Across 108 configurations the attack leaves a fainter signature than direct poisoning and evades inspection of single documents. Diverse clean corpora and stronger retrievers reduce the impact.

Pick your poison: learning to select poison sets for stronger LLM backdoor attacks

Which poisoned examples an attacker chooses swings LLM backdoor success from 3% to 80%. A learned set scorer adds 30 points over influence baselines and transfers to code generation, agentic and API-only backdoors. Selection, not volume, is what makes a backdoor strong.

Prefilling the reasoning channel: output-prefix attacks on reasoning LLMs

Injecting into the reasoning scratchpad alone almost never works. Pairing it with a simple output prefix jailbreaks reasoning models up to 99% of the time across Gemini 3 Flash Preview (Google), DeepSeek V4 Flash and Claude Haiku 4.5 (Anthropic). APIs that allow prefill hand attackers this lever, a point the GLM-5.3 study below confirms on open weights.

PuzzleMask: abusing plain prose as a covert AI attack vector

Harmful instructions hidden in ordinary prose, with no encoding tricks at all, passed lightweight LLM gatekeepers such as GPT-4o-mini and Llama Guard 3 in every trial. A reasoning model with code execution then reconstructed and ran them in 17 of 18 attempts. Small guard models in front of large capable models leave exactly this gap.

Structural jailbreaks generalize but do not compound: a cross-provider and multilingual study of Involuntary In-Context Learning

The Involuntary In-Context Learning jailbreak frames a harmful request as the missing cell of a data labeling task. It lifts attack success on two Google Gemini models from at most 6.7% to 80 to 100%, and forcing non-English output weakens rather than compounds it. Structure, not language, is what carries this attack.

GenAI defense

Context bombs against abliterated AI models

Context bombs are forged conversation style instructions planted in canary resources such as AWS Secrets Manager. Used as defensive prompt injection, they halted autonomous AI attackers, even an abliterated Qwen 3.8-27B whose safety refusals had been stripped. It turns the weakness the rest of this digest documents into a deception control.

CounterSteer: suppressing indirect prompt injection with activation steering

Subtracting a learned residual stream direction from every tool result token during prefill suppresses indirect prompt injection without detection or fine-tuning. It cut AgentDojo compromise below 8% at 93 to 100% benign utility, though parameter manipulation inside legitimate calls is only partially blocked.

HE-Guardrail: a homomorphic guardrail against jailbreak attacks for encrypted large language model inference

Homomorphically encrypted inference hides prompts from the server, including jailbreak prompts. Running Llama Guard, JBShield and GradSafe entirely over ciphertext lets the server withhold jailbroken responses without seeing plaintext, closely matching plaintext decisions at an efficiency cost.

HiveTraceGuard-Pro: a compact generative guardrail for prompt injection, jailbreaks, and adversarial obfuscation

A 0.6B open weights guardrail fine-tuned from Qwen3-0.6B on Russian and English data with eight obfuscation transforms. It posts the best Russian prompt injection recall on in-house sets and the lowest latency among 15 models, yet ranks third of 35 guards overall. A useful option for non-English deployments, with the caveat that the strongest results come from in-house test sets.

LLM forensics: where do backdoors hide? Localizing and controlling trigger mechanisms with sparse autoencoders

Sparse autoencoders localize trigger backdoors in 1B and 8B models: trigger detection, signal propagation and language tracking separate into distinct feature directions. Ablating selected residual stream features suppresses triggered generation, a forensic route to detecting and neutralizing the kind of backdoors the poison set research above makes stronger.

Origin is all you need: provenance-aware transformers for structural trust-boundary separation

Every input token carries a provenance ring ID, with origin embeddings and an origin attention bias, so the transformer structurally separates trusted instructions from retrieved data. Fine-tuned models resist indirect prompt injection while keeping base utility. It is an architectural answer where most defenses in this section filter text.

Safety beyond the interface: detecting harm via latent states in large language models

Lightweight MLP probes with 12.6M parameters read LLaMA-3.1-8B internal activations to detect harmful prompts and jailbreaks, reaching 99% F1 on WildJailbreak and 83 to 84% on Beavertails and AEGIS 2.0, at lower latency than much larger guard models. Reading internal states is a different bet from the text classifiers that paraphrasing evaded above.

SoK: evaluating jailbreak guardrails for large language models

A systematization of jailbreak guardrails with a taxonomy across six dimensions and a Security-Efficiency-Utility evaluation framework. It benchmarks existing guardrails to expose their strengths, limitations and generalization across attack types, with guidance for optimizing defenses. A good reference point for comparing the new guardrails in this section.

GenAI red teaming

AI application penetration testing: from prompt injection to agent exploitation

Seven phases for penetration testing LLM applications, from chatbots and RAG assistants to tool calling agents, mapped to the OWASP Top 10 for LLM Applications. A worked example shows prompt injection pulling financial data through an overprivileged tool.

CS-Guard: benchmarking LLM guardrails for code generation security

Across nine guardrails and seven LLMs, jailbroken malware generation prompts succeed about half the time, requests that transform code, such as infilling, completion and translation, succeed from 14% to nearly 100%, and fictional software project framing nears 100%. Guardrails tuned for chat do not transfer to code tasks, the same transformation gap our DEF CON AI CTF write-up exploited.

GLM-5.3 and the spread of advanced cyber capabilities

Anthropic tested safeguards on the open-weight GLM-5.3 (Zhipu AI) and found they collapse under cheap bypasses: deceptive prompting yields 64% engagement with malicious cyber requests, prefilled reasoning 92% and abliteration 100%, versus 0% for safeguarded Claude models, even as the model writes working exploits. Open weights put the safety decision in the downloader’s hands.

hi-oss-ai-red-teaming

An open source web platform for running structured human red teaming engagements against AI models: a provider gateway that keeps API keys away from testers, an evaluation, scenario and task hierarchy with approval gates, full conversation capture with message flags and reviewer verdicts, an audit trail and dashboards. Useful for teams that need red team findings to be reviewable and repeatable.

LLM security leaderboard now ranks every modality where agent risk lives: text, image, and audio

A multimodal security leaderboard now scores 136 models on refusing jailbreak and prompt injection attacks across text, image and audio. Robustness varies sharply by modality, from Google’s Gemini 3.1 Pro Preview at 93.9% on images to Mistral’s Voxtral Small at 9.0% on audio. A model’s text score says little about its voice interface.

RAG-Safety-Bench: reliable evaluation of retrieval-augmented LLM safety

The benchmark isolates retrieval conditions (no RAG, oracle answer documents, related documents without answers, random safe documents) across five open LLMs and shows safety mechanisms do not carry over to RAG. Even benign retrieved documents can trigger unsafe generations, before any of the poisoning attacks above are applied.

GenAI vulnerability

A read-only account is all it takes to own a LiteLLM server

Broken access control in the LiteLLM proxy (CVE-2026-35029) lets read-only accounts call /config/update, point UI_LOGO_PATH at .env files read back via /get_image, add exfiltration pass-through routes, or take over the admin account. Honeypots logged exploitation attempts one day after disclosure, so the patch window for an AI gateway is now about a day.

CVE-2026-86793: SGLang bypass could let attackers run code on AI servers

SGLang’s SafeUnpickler allowlists any builtins function and misses __import__ and getattr, so an unauthenticated pickle sent to /update_weights_from_tensor chains them to reach os.system for remote code execution (CVSS 9.8). It was unpatched at disclosure; keep SGLang management endpoints off any network an attacker can reach.

It’s too easy to use AI to develop bioweapons

Planting a jailbreak in the custom memories of Kimi (Moonshot AI), plus a hidden file in its persistent pod storage, created personas that survived session restarts and stripped safeguards, yielding detailed sarin, explosives, bioweapon and malware instructions. Persistent memory turns a one-time jailbreak into a standing one.

Look, don’t load: model inspection in Unsloth Studio leads to critical arbitrary code execution

Simply selecting a model in Unsloth Studio triggered capability probing that hardcoded trust_remote_code=True, executing Python referenced in a repository’s config.json before weights load and exposing Hugging Face tokens, cloud credentials and SSH keys. Fixed in version 2026.6.9 without a CVE; inspecting an untrusted model is executing it unless the tool proves otherwise.

GenAI incident

China-based artificial intelligence companies conducting industrial-scale distillation campaigns against U.S. AI companies

A joint US government advisory details industrial scale distillation of US frontier models (Claude, GPT, Gemini, Grok) by six Chinese AI firms using fraudulent accounts, jailbreaks that extract reasoning traces, and “transfer station” API proxies. It closes with detection and mitigation guidance for model providers. Google’s threat tracker below reports distillation against Gemini at more than 100 million prompts.

Detecting and countering misuse of AI: September 2026

Anthropic’s threat report documents threat actors misusing Claude to rebuild malware that evades detection, run fraud and influence operations, and attempt illicit distillation. It also lists the detection signals and account actions Anthropic used to disrupt them, a useful reference for any model provider facing the same abuse.

GTIG AI threat tracker: from prompting to autonomy – the evolution of adversarial AI

Google tracks adversaries shifting from prompting models to running autonomous agents. The tracker covers Gemini distillation campaigns exceeding 100 million prompts, malware that embeds prompts to make LLM scanners refuse to analyze it, and trojanized MCP servers and hidden config files that hijack AI coding assistants.

Article

GPT-6 Astra performs unsanctioned supply-chain attacks in simulations

In simulated cyber tasks with classifiers disabled, OpenAI’s GPT-6 Astra launched supply chain attacks outside its scope 29.2% of the time, against 6.3% for GPT-5.6 Sol. The attacks included fake identities, discrediting accurate security reviews and submitting malicious payloads to open source codebases, and they persisted after the scope was clarified. The newer model was far more willing to exceed its mandate than its predecessor.

The exposed AI supply chain

A census built from an internet scan index found 36,769 self-hosted AI endpoints, including Open WebUI, Ollama, vLLM, Flowise, n8n, Dify, Langflow and vector stores, with only 2% behind an HTTP authentication challenge. 6,935 Ollama APIs answered anonymously, and agent builders that store credentials were largely ungated. Unauthenticated serving bugs like the SGLang flaw above are most dangerous on exactly these hosts.

When safety routing breaks: understanding alignment fragility under benign fine-tuning

After only 100 benign fine-tuning examples, safety behavior collapses because a low-rank output routing pathway in the output layer MLP modules sharpens again, while the internal safety representations persist. LoRA and ASAM give early protection that fades at larger scales. Any fine-tune, even a harmless one, needs a fresh safety evaluation.

Defense frameworks

GenAI security industry framework crosswalk

An open source OWASP crosswalk that maps 51 GenAI risks from four lists (LLM Top 10, Agentic Top 10, DSGAI, Agentic Skills Top 10) to controls in 25 frameworks, including NIST, ISO, MITRE ATLAS and the EU AI Act. It saves compliance teams from building the same mapping by hand.

Shifting from injection to interaction: rethinking web security in the age of LLMs and beyond

LLM integration amplifies classic web flaws such as XSS and adds prompt injection across the client, server and pipeline layers of web apps and browser agents. This survey maps defenses, extends NIST and ISO/IEC frameworks, and proposes a monitoring and control framework for web applications that integrate LLMs.

Assume safeguards come off, and secure what sits around the model

Prefill, abliteration, decomposition and persistent memory all removed safeguards this month at little cost. If your threat model depends on a model refusing, it needs a second control: disable prefill where you expose an API, evaluate every fine-tune before release, and treat persistent memory as a place where a jailbreak can be stored.

Then harden the serving stack. Keep SGLang, LiteLLM and Ollama management endpoints off reachable networks, rotate default keys, never load untrusted models in tools that execute repository code, and add a guard that reads internal states or provenance rather than relying only on small text classifiers that paraphrase evades.