A comprehensive security risk checklist for AI chatbots

This is why we’ve prepared a checklist that sets out the most important potential security threats to be on the alert for when working on LLM-based AI chatbots. It is intended to serve as a developers’ guide to identifying security gaps and implementing effective mitigation strategies. With the increasing integration of these technologies into everyday operations, it has become crucial to address possible threats that can lead to data breaches, unauthorised access and compromised systems.

If you’re interested in AI chatbot security and want to discover what our experts and special guests have to say in an interactive panel discussion, then sign up for our latest live webinar, “Mastering AI chatbot security”.

We recommended using this checklist as a tool for regularly assessing security and implementing appropriate practices. Each of the points describes a threat and suggests strategies for its mitigation, making for a more informed approach to application security.

This is a vulnerability where user inputs are crafted to alter the behaviour of an LLM in unintended ways. Attackers can manipulate the model’s responses, bypass safety mechanisms and potentially execute harmful commands.

Mitigation strategies

• Implement strict input validation and sanitisation.
• Use context-specific prompts to constrain model behaviour.
• Employ filtering mechanisms to detect and reject malicious input.

Insufficient checks on LLM outputs can lead to unintended consequences, such as executing dangerous commands, exposing sensitive information or enabling Cross-Site Scripting (XSS) attacks.

Mitigation strategies

• Use output encoding based on the context (e.g. HTML, JSON).
• Implement strict output validation.
• Monitor and log LLM outputs to detect anomalies.

Malicious actors can introduce harmful data during the training phase, causing biased or malign behaviour in LLMs. This manipulation undermines the integrity of model outputs.

Mitigation strategies

• Ensure robust vetting and sanitisation of training data.
• Regularly audit data sources for integrity and security.
• Maintain a version control system for datasets to track changes.

Attackers can overwhelm an LLM by flooding it with excessive or complex requests, triggering service outages and resource depletion.

Mitigation strategies

• Implement rate limiting and request quotas for API access.
• Use resource allocation management to monitor and control usage.
• Deploy timeouts for processing long or intensive requests.

LLMs may inadvertently disclose sensitive information, including personally identifiable information (PII), proprietary data or internal system instructions.

Mitigation Strategies

• Scrub sensitive data from output and limit what is processed by the model.
• To avoid unintentional disclosures, educate users on safe interactions with AI chatbots.
• Rigorously test the model to uncover potential leakage points.

Addressing the security risks associated with AI chatbots is critical to maintaining user trust, protecting sensitive data and ensuring the overall integrity of chatbot applications. Adhering to this checklist will enable organisations to be proactive in mitigating these risks and enhancing the security posture of their AI systems. Ongoing education and adaptation to evolving threats will be vital to safeguarding the future of AI-driven interactions.

The text was created based on:

OWASP Foundation, OWASP Top 10 for Large Language Model Applications

Translated from the Polish by Caryl Swift