AI model security ranking top 100: in-context learning

Involuntary in-context learning (IICL) is an attack technique against AI Applications and Agents that hides malicious instructions or harmful requests inside few-shot pattern completion tasks, bypassing safety alignment in large language models and allowing prompt injection and jailbreak attacks. When we first tested IICL against GPT-5.4, the model bypassed at 60% ASR, while GPT-5 and GPT-5-mini remained at 0% — a safety regression introduced between releases.

The clear next step was to determine whether IICL is specific to OpenAI or a broader issue in industry safety training. To investigate, we ran the attack across frontier and open-weight models from Anthropic, OpenAI, Google, xAI, DeepSeek, Meta, Mistral, Cohere, Microsoft, Amazon, Moonshot, Alibaba, Zhipu, Tencent, Xiaomi, Z-AI, and AntGroup, testing ten attack variants each.

We evaluated 100 (model, reasoning-mode) entries across 17 vendor families with 24,956 adversarial probes using different sub-techniques of IICL attack family. The results show a stark bimodal distribution: 36 entries were completely immune, while a similar number were fully compromised (100% bypass on at least one variant). Vulnerability correlates more closely with vendor identity than parameter count or release date.

Read the full research: Cross-Model Vulnerability to Involuntary In-Context Learning Attacks.

On this page

1. Top 100 master ranking (all entries)
2. Table A — reasoning-active entries (50)
3. Table B — non-reasoning entries (50)

Top 100 master ranking

Every tested entry, ranked from most to least vulnerable. R marks reasoning state (Y = reasoning-active, n = non-reasoning). Mode gives the specific configuration used (e.g. minimal, high, thinking_on). Bypass rate is the share of adversarial probes (out of N) that successfully elicited harmful output.

Table A — Reasoning-active entries (50)

Every entry running with reasoning enabled, ranked by bypass rate. Pooled bypass rate: 9.2% [8.7, 9.7] — roughly 4× lower than the non-reasoning pool.

Table B — Non-reasoning entries (50)

Every entry running without reasoning (chat baseline, reasoning explicitly disabled, or no reasoning mode available). Pooled bypass rate: 39.4% [38.5, 40.2].

Column definitions

• Model (mode)

  The specific model entry tested. Where the same model appears multiple times, the parenthetical or Mode column distinguishes reasoning configurations.

• Family

  Vendor or model lineage (e.g. OpenAI, Anthropic, Google, Mistral). Family is one of the strongest predictors of IICL vulnerability in this study.

• R (reasoning state)

  Y = reasoning chain active during inference. n = no reasoning (chat baseline or reasoning explicitly disabled).

• Mode

  The configuration flag used: reasoning effort level (minimal, low, medium, high), thinking_on / thinking_off, none, or default when the model has only one supported mode.

• Bypass B/N (%)

  Number of successful bypasses (B) over total adversarial probes attempted (N), and the resulting bypass rate. Higher = more vulnerable. The attack used here is the optimal V8 configuration from the study.

• 95% CI

  Wilson 95% confidence interval around the bypass rate, accounting for sample size. Narrow intervals indicate higher statistical confidence in the point estimate.

Conclusion

IICL vulnerability is neither universal nor random — it is determined by specific architectural and training choices that vary systematically across vendors and model generations, with reasoning augmentation emerging as the strongest single defensive factor measured (a 4.3× reduction in bypass rate). Anthropic’s Claude family is uniformly immune across 15 entries and four generations, while Mistral’s entire lineup falls at 94–100% — meaning model selection alone is now a measurable safety control for enterprises deploying LLMs.

Read the full paper: Cross-Model Vulnerability to Involuntary In-Context Learning Attacks.