CSO Online, January 22, 2024
The intersection of natural language prompts and training sources poses unique threats, including prompt injection, prompt extraction, phishing schemes, and the poisoning of models. Traditional security tools find it challenging to keep pace with these dynamic risks, necessitating a proactive and adaptive approach to secure the deployment of LLMs.
Addressing the concerns surrounding LLM security, prominent entities such as the Open Web Application Security Project (OWASP), the National Institute of Standards (NIST), and the European Union (EU) have swiftly responded by identifying and prioritizing vulnerabilities associated with LLMs. The EU AI Act, for instance, introduces a compliance checker to assist organizations in assessing the risk levels of their AI applications. This regulatory landscape emphasizes the need for organizations to stay abreast of evolving threats and align their security measures with industry standards.
As organizations grapple with the security implications of LLMs, a new frontier is emerging in the form of AI versus AI. The development of natural language web firewalls, AI discovery tools, and AI-enhanced security testing tools signifies an ongoing battle to fortify AI systems against evolving threats. In this dynamic landscape, fostering a comprehensive AI security policy becomes imperative. By embracing innovative tools and techniques, organizations can proactively safeguard their systems, ensuring the responsible and secure integration of LLMs into their operations.
Steve Wilson, the founder of OWASP fo AI, delves into the heart of these threats, providing an exclusive focus on LLMs and eschewing generalized AI security discussions. The book emphasizes the unique characteristics and vulnerabilities inherent in LLMs, shedding light on the critical aspects that demand attention from developers and security teams alike.
Wilson’s expertise extends beyond theoretical discussions, drawing from the collective wisdom of over 400 industry experts who contributed to the creation of the OWASP Top 10 for LLMs list. This collaborative effort ensures that the guide offers real-world insights and practical strategies to help navigate the complex security landscape presented by LLM applications. Whether you are in the process of architecting a new application or integrating AI features into an existing one, this book serves as an invaluable resource, offering guidance on mastering the security challenges associated with the next frontier in AI.
In a rapidly evolving technological landscape, where the deployment of LLMs introduces unprecedented capabilities, understanding and addressing security concerns becomes paramount. Wilson’s book serves as a beacon for developers and security teams, providing them with the tools and knowledge needed to grapple with the practical realities of securing applications leveraging LLMs. As the industry continues to advance, this resource stands as a go-to guide for ensuring the safety and security of AI systems, contributing to a more robust and resilient digital future.
DarkReading, January 24, 2024
A recent report from the Berryville Institute of Machine Learning (BIML) highlights that more than a quarter of the risks associated with LLMs stem from this lack of visibility.
The BIML report, titled “An Architectural Risk Analysis of Large Language Models,” aims to equip Chief Information Security Officers (CISOs) and security practitioners with a comprehensive understanding of the risks posed by machine learning (ML) and AI models, specifically focusing on LLMs and the next-generation large multimodal models (LMMs). Gary McGraw, co-founder of BIML, emphasizes the need to unravel the black box and answer critical questions regarding the risk decisions made by major players like Google, Open AI, Microsoft, and Meta on behalf of users.
In response to the growing adoption of AI models in businesses, organizations like the US National Institute of Standards and Technology (NIST) are working towards establishing a common language for discussing threats to AI. The NIST’s “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations” paper categorizes the AI landscape into predictive AI models (PredAI) and generative AI systems (GenAI), such as LLMs. The need for a unified terminology becomes increasingly crucial as generative AI technology integrates into mainstream product lines, prompting companies to prepare for the potential risks.
The challenges of securing LLMs extend to the very process of their creation. Analytical hacks drive the development of generative AI (GenAI) models, where large data sets train machines to correlate inputs and outputs without explicit programming knowledge. A significant portion of the risks identified in the BIML report, including 23 directly related to black-box issues, underscores the complexity of understanding the inner workings of LLMs. The call for government regulations focuses on holding LLM foundation model creators accountable for the risks introduced into the AI landscape. Despite the challenges, the ongoing pursuit of better defenses and policies remains crucial in navigating the evolving landscape of AI security.
VentureBeat, January 25, 2024
IBM has unveiled a robust security framework, the IBM Framework for Securing Generative AI, dedicated to addressing the unique challenges posed by generative AI throughout its entire lifecycle. This initiative comes at a time when organizations are increasingly relying on IBM’s watsonX portfolio, which integrates models and governance capabilities, to bolster their generative AI capabilities.
At the heart of IBM’s approach are three foundational tenets: securing the data, the model, and the usage. Emphasizing the importance of maintaining secure infrastructure and AI governance, the framework provides comprehensive guidance on mitigating the diverse security threats associated with generative AI. Sridhar Muppidi, IBM Fellow and CTO at IBM Security, underscores the continued relevance of core data security practices while acknowledging the need for specialized measures to counter unique risks such as data poisoning, bias, and data diversity.
One emerging risk area highlighted by Muppidi is prompt injection, where users attempt to maliciously modify the output of a model through a prompt. To counter these evolving risks, the framework encourages the implementation of new controls, reflecting the dynamic nature of AI security. Unlike a singular tool, the IBM Framework for Securing Generative AI serves as a comprehensive set of guidelines, offering insights into the most likely security threats and recommending top defensive approaches. Additionally, the framework introduces new categories in security, including Machine Learning Detection and Response (MLDR), AI Security Posture Management (AISPM), and Machine Learning Security Operation (MLSecOps), demonstrating a commitment to a holistic approach that spans the entire lifecycle of generative AI.
The Register, January 25, 2024
Kaspersky’s recent findings reveal a troubling surge in the creation and sale of malicious AI prompts, reaching 249 instances in 2023 alone. While LLMs have not yet evolved to generate polymorphic malware or execute complete attack chains, there is a discernible interest among cybercriminals in exploiting AI for illegal activities. The ease of achieving tasks through a single prompt significantly reduces the entry barriers into various domains, raising concerns about the potential misuse of AI technologies.
In this evolving threat landscape, Kaspersky’s research sheds light on a thriving underground market where criminals not only craft malicious prompts but also peddle them to individuals lacking the technical acumen to develop their own. Additionally, a burgeoning market for stolen ChatGPT credentials and compromised premium accounts adds a layer of complexity to the security challenges posed by AI. While speculation has surrounded the potential use of AI in crafting polymorphic malware, Kaspersky’s current findings indicate no tangible evidence of such malware in operation. However, the authors caution that this may change in the future.
Beyond the creation of malicious prompts, Kaspersky’s research underscores instances where the inherent vulnerabilities of AI systems are exploited for illicit gains. By repeating a sample prompt, researchers were able to extract sensitive information from ChatGPT, highlighting the need for enhanced security measures. The study also delves into the darker side of AI applications, revealing the existence of AI-powered software advertised by malware operators. This software not only analyzes and processes information but also aids criminals in safeguarding their activities by automatically switching cover domains. As AI continues to reshape various sectors, including cybersecurity, these findings emphasize the imperative of sustained vigilance and proactive measures to ensure the security and safety of AI technologies against malicious intent.
Stay up to date with what is happening! Get a first look at news, noteworthy research and worst attacks on AI delivered right in your inbox.
Here’s the top LLM security publications collected in one place for you. This digest provides insights into various aspects of Large Language Model (LLM) security. It covers a range of ...
