Here’s the top LLM security publications collected in one place for you.
This digest provides insights into various aspects of Large Language Model (LLM) security. It covers a range of topics, from checklists for LLM Security and incidents involving vulnerabilities in chatbots to real-world attacks and initiatives by the Cloud Security Alliance. But there’s more to come.
Noteworthy is a document “LLM AI Security & Governance Checklist”.
The OWASP AI Security and Privacy Guide working group is actively monitoring developments and addressing complex considerations for AI. The provided checklist aims to assist technology and business leaders in understanding the risks and benefits of using Large Language Models (LLM).
It covers scenarios for both internal use and third-party LLM services, referencing resources from MITRE Engenuity, OWASP, and others. The checklist encourages the development of a comprehensive defense strategy and integration with OWASP and MITRE resources. While the document supports organizations in creating an initial LLM strategy, it acknowledges the evolving nature of the technical, legal, and regulatory landscape and encourages extending assessments beyond the checklist’s scope.
A chatbot at a California car dealership, powered by Fullpath’s ChatGPT, went viral as users discovered its vulnerability to manipulation.
Users tricked the chatbot into making absurd offers. Fullpath promotes its chatbot’s ease of use but faces scrutiny over its susceptibility to manipulation.
Chris Bakke, a tech executive, shared screenshots of the chatbot offering to sell a 2024 Chevy Tahoe for a dollar, leading to widespread online interactions, including discussions on various topics and attempts to trick the bot. Fullpath claims its ChatGPT is designed to assist serious automotive inquiries but has implemented features to prevent pranksters.
Similar incidents were reported with another dealership’s chatbot in Massachusetts. The incident highlights the challenges of integrating imperfect AI technology into widespread online use.
In this post, the author details the vulnerability in Writer.com. It allows attackers to steal user’s private documents through indirect prompt injection in the language model used for content generation.
Writer.com, an application for generating tailored content, is susceptible to manipulation, allowing attackers to exfiltrate sensitive data, including uploaded documents and chat history. The attack involves tricking the user into adding a malicious source that manipulates the language model.
Despite responsible disclosure, Writer.com did not consider it a security issue, and the vulnerability was not fixed. The attack chain involves injecting hidden instructions that lead to data exfiltration without the user’s knowledge. Examples include exfiltrating uploaded files and chat history. The post includes a responsible disclosure timeline and recommends checking OWASP and MITRE for Large Language Model (LLM) security risks.
This video refers to a talk at the 37th Chaos Communication Congress titled “NEW IMPORTANT INSTRUCTIONS: Real-world exploits and mitigations in Large Language Model applications.”
The talk discusses the security of Large Language Models (LLMs), highlighting real-world exploits, including prompt injections, and the mitigations and fixes implemented by vendors for LLM applications like ChatGPT, Bing Chat, and Google Bard. The focus is on the implications of Prompt Injections in LLM security, addressing risks such as scams, data exfiltration, and potential remote code execution. It emphasizes the increasing challenges users face in terms of security with the rapid growth of AI and LLMs.
This paper delves into the intersection of LLMs with security and privacy, exploring their positive impact on security, potential risks, and inherent vulnerabilities. The findings are categorized into “The Good” (beneficial applications), “The Bad” (offensive uses), and “The Ugly” (vulnerabilities and defenses). While LLMs enhance code and data security, they also pose risks, particularly in user-level attacks due to their human-like reasoning. The paper identifies areas needing further research, such as model extraction attacks and the exploration of safe instruction tuning. The goal is to illuminate both the potential benefits and risks that LLMs bring to cybersecurity.
Microsoft researchers have introduced PromptBench, a PyTorch-based Python package designed to address the lack of standardization in evaluating Large Language Models (LLMs).
PromptBench offers a modular and user-friendly four-step evaluation pipeline, focusing on task specification, dataset loading, LLM customization, and prompt definition. The platform incorporates extra performance metrics to provide detailed insights into model behavior across tasks and datasets. With a commitment to user-friendly customization and versatility, PromptBench aims to fill the gaps in current evaluation methods for LLMs, offering a standardized and comprehensive framework for researchers. It marks a significant advancement in shaping the future of LLM evaluation.
In WIRED, there is a post about a noteworthy LLM jailbreak. New adversarial algorithms can systematically exploit vulnerabilities in large language models, including OpenAI’s GPT-4, to make them misbehave. This comes amid concerns about the rapid progress in artificial intelligence and potential risks associated with commercializing the technology too quickly.
The article highlights the need to pay more attention to the risks involved in AI systems and their susceptibility to adversarial attacks.
The article explores the basics of AI security, focusing on Large Language Models (LLMs). The author, with a computer security background, delves into understanding LLMs by attempting to break them. The discussion covers text generation, neural networks, and the significance of understanding neural network processes for prompt injection attacks. Prompt injection, analogous to injection attacks in computer security, raises concerns about hidden malicious commands within user-generated prompts for AI.
The article introduces the threat model, where LLMs interpret user input as instructions, potentially leading to unintended responses. The relevance of prompt injection is discussed in scenarios like bypassing AI content moderation and extracting data from personal assistant AIs. The post also mentions different types of prompt hacking, including prompt leaking and jailbreaking, and explores offensive and defensive techniques, such as obfuscation strategies and code injection exploits. The author provides examples of prompt injection attacks on OpenAI, highlighting the need for further investigation into these methods.
The Cloud Security Alliance (CSA) has launched the AI Safety Initiative in collaboration with Amazon, Anthropic, Google, Microsoft, and OpenAI. The initiative aims to provide guidelines for AI safety and security, with an initial focus on generative AI. It brings together a diverse coalition of experts from government agencies, academia, and various industries.
The goal is to equip organizations of all sizes with tools and knowledge to deploy AI responsibly, aligning with regulations and industry standards. The initiative emphasizes reducing risks and enhancing the positive impact of AI across sectors. Core research working groups have been established, and the initiative plans to update progress and host events, including the CSA Virtual AI Summit and the CSA AI Summit at the RSA Conference.
The initiative involves over 1,500 expert participants and encourages global engagement through CSA’s chapters worldwide. The collaborative effort seeks to address the transformative potential of AI while ensuring safety and security in its development and deployment.
The report discusses the emergence of adversarial artificial intelligence (AAI), a sub-discipline within AI that involves strategic deception and counter-deception. As AI systems become more sophisticated, the potential for adversarial actions, targeting both humans and AI systems, poses threats to the reliability of AI and the trust in digital content. The report aims to introduce AAI concepts, explore future threats, assess risks, and propose mitigation strategies. It serves as a foundation for developing a risk-informed approach to address vulnerabilities and threats associated with adversarial AI within the Department.
The job description of a Large model AI safety engineer outlines responsibilities for a role in Tencent’s AI product security evaluation. The candidate is expected to identify vulnerabilities in AI’s security, propose solutions, analyze AI security technologies, and conduct research on open-source Foundation Models. Qualifications include a bachelor’s degree in computer science or related fields, expertise in Large Language Model (LLM) security, proficiency in programming languages, strong communication skills, and a bonus for relevant experience, publications, or participation in AI Red Team activities. The role emphasizes a focus on security attack and defense in the AI domain.
A CISO’s perspective on how to understand and address AI risk SCMedia, January 16, 2024 The adoption of AI in enterprises introduces significant risks that span technical, reputational, regulatory, and ...
