ICIT Securing AI: Addressing the OWASP Top 10 for Large Language Model Applications — TOP 10 insights

The Institute for Critical Infrastructure Technology (ICIT) has published a new report that connects the OWASP-LLM Top 10 risks with real-world AI security practices. This is more than just a list of threats. It is a practical guide designed to help teams secure large language models (LLMs) in real-world systems.

Today, many organizations are rushing to deploy generative AI. However, most are not prepared for the unique risks LLMs introduce. That’s why this report matters. It shows how each risk works in practice and explains what steps you can take right now to reduce exposure.

In this guide, you’ll find ten key lessons. Each one focuses on a major risk, explains it in plain language, and offers simple but effective ways to respond. If you apply these insights during development, testing, and deployment, your AI systems will be safer, more reliable, and easier to manage over time.

1. How to Prevent Prompt Injection Attacks

“Your brand can be hacked at the speed of a sentence—prompt-proof every interface before your customers do.” Share on X

Prompt injection is one of the most dangerous and misunderstood threats in generative AI security. Attackers craft malicious inputs that manipulate the model into ignoring or rewriting its original instructions.

This can lead to unauthorized access, output of sensitive data, or generation of harmful actions. Because LLMs rely on natural language rather than rigid code, even subtle prompt manipulations can bypass guardrails entirely. Without proper input validation, your AI systems remain vulnerable to silent compromises.

What You Can Do:

1. Call LLMs through a broker service that strips risky verbs like file-system or shell commands unless specifically allowed.
2. Incorporate adversarial prompt-fuzzing into CI/CD pipelines to catch leaks early.
3. Use signed context boundaries like <<BEGIN-SECURE>> blocks, and reject outputs with altered signature zones.

2. Avoiding LLM Leaks of Sensitive Information

“LLMs don’t leak because they’re evil—they leak because you gave them the data with no leash.” Share on X

Large language models trained on sensitive or proprietary data can unintentionally echo private information such as personally identifiable information (PII), internal documents, or confidential business logic. Even if the original data is no longer accessible directly, models may still surface fragments in response to cleverly phrased prompts.

This creates a serious compliance risk under data privacy laws like GDPR or HIPAA, and can lead to reputational and legal consequences. The risk is heightened in customer-facing applications where prompt diversity is high.

What You Can Do:

1. Treat training corpora and embeddings with the same DLP policies as raw data.
2. Add guardrails that redact things like credit cards or SSNs in responses.
3. Use a human-in-the-loop review for high-risk or sensitive prompts.

3. Mitigating AI Supply Chain Vulnerabilities

“Trusting an unknown model is installing a rogue admin you can’t interview.” Share on X

The AI supply chain is complex and often opaque. Pretrained models, open-source libraries, third-party APIs, and vector databases all introduce dependencies that may be compromised. A single tainted weight file or unverified retraining dataset can introduce backdoors or poisoned logic into your system.

Worse, many of these components are updated automatically, meaning malicious code could propagate silently. Without full visibility and control over your model’s software bill of materials (SBOM), you may unknowingly inherit risks from upstream.

What You Can Do:

1. Collect signed SBOMs (Software Bill of Materials) for every model and plugin.
2. Run CVE scans during every build.
3. Demand continuous vendor proof that training data hasn’t been compromised.

4. Defending Against Data and Model Poisoning

“Bad data is the zero-day you already shipped to prod.” Share on X

Data poisoning and model tampering are stealthy threats where attackers inject corrupted examples into training data or manipulate model weights directly. These attacks are designed to lie dormant until triggered by a specific phrase or pattern, at which point the model behaves in a way the attacker intends.

The impact can range from biased recommendations to direct misuse of system functionality. Because these backdoors are statistically rare, they often evade detection in standard evaluation pipelines—especially if you don’t proactively test for them.

What You Can Do:

1. Hash and sign all training data.
2. Regularly scan embeddings to find statistical outliers.
3. Use canary prompts to verify that the model doesn’t react to known poison triggers.

5. Securing LLM Output to Avoid Liability

“An LLM can be 99% right and 100% liable.” Share on X

Even when a model functions correctly 99% of the time, its outputs can still generate serious legal or operational consequences. LLMs may hallucinate plausible-sounding but false information, suggest unsafe actions, or format their output in a way that breaks downstream integrations.

In regulated environments such as healthcare, finance, or legal services, these errors can lead to lawsuits, compliance violations, or customer harm. If no safeguards exist to validate or constrain generated content, the organization—not the model—is held accountable.

What You Can Do:

1. Enforce strict output formats like JSON or Markdown, rejecting anything malformed.
2. Cross-check critical outputs against trusted knowledge sources.
3. Collect user feedback and escalate risky responses to human reviewers.

6. Limiting Excessive Autonomy in AI Agents

“Give an agent too many keys and it will eventually open the wrong door.” Share on X

Autonomous agents powered by LLMs are increasingly wired into business systems—from email automation and ticketing tools to payment APIs and scheduling workflows. When these agents are given too much authority or access without strict controls, they can misfire catastrophically. This includes sending unintended messages, deleting records, or initiating financial transactions based on misunderstood instructions.

The combination of emergent behavior, incomplete instructions, and excessive access makes unregulated agents one of the fastest-growing risks in enterprise AI adoption.

What You Can Do:

1. Use the principle of least privilege—default to read-only, escalate to write after multi-party approval.
2. Log every action with tamper-evident hashing.
3. Simulate worst-case runaway scenarios during development cycles.

7. Preventing System Prompt Leakage

“If attackers can read your system prompt, they own your policy.” Share on X

System prompts define how your LLM behaves, including safety rules, tone, and hidden instructions. If attackers manage to extract these prompts, they can reverse-engineer internal policies, target specific weaknesses, or craft jailbreaks tailored to your exact configuration.

This type of leakage often happens unintentionally—via reflection, summarization, or poorly scoped user queries. Once exposed, your defenses become predictable, giving attackers a permanent advantage until the system is redesigned or retrained.

What You Can Do:

1. Inject prompts at runtime from a secured secrets manager.
2. Never hard-code them into the application.
3. Use AI red teaming to constantly test for prompt extraction and alert on any leak.

8. Securing Vectors and Embeddings

“Your search index is a goldmine—lock it down before someone mines it.” Share on X

Vector databases and embeddings store representations of your data that can be reverse-engineered if left unsecured. Attackers may exploit APIs to reconstruct sensitive content, inject biased vectors, or alter search results.

In RAG (Retrieval-Augmented Generation) systems, poisoned embeddings can skew the context that models rely on, leading to misinformation or bias amplification. If your vector store lacks encryption, access control, or anomaly detection, it becomes a soft target for data exfiltration or manipulation.

What You Can Do:

1. Encrypt embeddings both at rest and in transit.
2. Use mutual TLS for all vector DB access.
3. Detect anomalies in similarity scores and re-test embeddings after model updates.

9. Reducing AI Hallucinations and Misleading Output

“An eloquent hallucination is still a lie—guard against confident nonsense.” Share on X

Hallucinations are one of the most visible and trust-eroding problems in LLMs. A model may fabricate facts, invent sources, or confidently deliver answers that sound right but are entirely false. These hallucinations can spread misinformation, damage brand reputation, or lead users to make harmful decisions.

In enterprise or mission-critical settings, even one hallucinated detail—such as a fake diagnosis or fabricated contract term—can create irreversible damage. Without structured validation and source grounding, these errors go unnoticed until it’s too late.

What You Can Do:

1. Force citation mode for all factual outputs.
2. Train users to prompt for grounded answers.
3. Use uncertainty detectors to route unclear outputs for human review.

10. Controlling AI Resource Consumption

“The cheapest denial-of-service is a politely worded, trillion-token prompt.” Share on X

LLMs are computationally expensive, and adversaries can exploit this by sending massive, looping, or obfuscated prompts that consume GPU time, spike memory, or trigger runaway costs.

This form of denial-of-service doesn’t require hacking—just clever prompting. Without per-user token limits, rate throttling, or anomaly detection, attackers or even unintentional users can bring down your service or inflate your cloud bill dramatically. Left unchecked, this creates a fragile system prone to outages and financial waste.

What You Can Do:

1. Set per-user token and CPU limits with exponential backoff.
2. Track GPU/CPU usage patterns.
3. Red-team resource attacks using mirrored traffic and synthetic spikes.

From Awareness to Action: Securing AI Systems at Scale

AI security is no longer a theoretical concept—it’s a core part of your operational reliability. As language models become deeply integrated into products and workflows, their behavior directly impacts trust, uptime, and compliance. Simply knowing the risks is not enough. You need to turn that awareness into repeatable, testable security practices.

Start by embedding secure development habits early in your lifecycle. Then, continue with automated testing, regular monitoring, and cross-functional review. The goal is not perfection, but resilience—your systems should fail safely, recover quickly, and protect users even under attack.

To make this shift sustainable, many teams are now adopting AI Red Teaming as a continuous process. Instead of waiting for issues to surface in production, they simulate real-world attacks in development to reveal vulnerabilities early. This approach not only reduces risk but also builds team-wide confidence in AI deployments.

If you’re ready to strengthen your AI defenses, consider using the Continuous Adversa AI Red Teaming Platform. It helps security and engineering teams proactively test LLM behavior, discover vulnerabilities, and harden models—before attackers do.

Subscribe for updates

Stay up to date with what is happening! Get a first look at news, noteworthy research and worst attacks on AI delivered right in your inbox.