CWE-1426 Detail

The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.

Modes Of Introduction

Architecture and Design
Implementation

Applicable Platforms

Language

Class: Not Language-Specific (Undetermined)

Architectures

Class: Not Architecture-Specific (Undetermined)

Technologies

Name: AI/ML (Undetermined)
Class: Not Technology-Specific (Undetermined)

Common Consequences

Observed Examples

Potential Mitigations

Phases : Architecture and Design
Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
Phases : Operation
Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
Phases : Operation
Phases : Build and Compilation

Detection Methods

Dynamic Analysis with Manual Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Dynamic Analysis with Automated Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Architecture or Design Review

Review of the product design can be effective, but it works best in conjunction with dynamic analysis.

Vulnerability Mapping Notes

Justification : There is potential for this CWE entry to be modified in the future for further clarification as the research community continues to better understand weaknesses in this domain.
Comment : Array

Notes

This entry is related to AI/ML, which is not well understood from a weakness perspective. Typically, for new/emerging technologies including AI/ML, early vulnerability discovery and research does not focus on root cause analysis (i.e., weakness identification). For AI/ML, the recent focus has been on attacks and exploitation methods, technical impacts, and mitigations. As a result, closer research or focused efforts by SMEs is necessary to understand the underlying weaknesses. Diverse and dynamic terminology and rapidly-evolving technology further complicate understanding. Finally, there might not be enough real-world examples with sufficient details from which weakness patterns may be discovered. For example, many real-world vulnerabilities related to "prompt injection" appear to be related to typical injection-style attacks in which the only difference is that the "input" to the vulnerable component comes from model output instead of direct adversary input, similar to "second-order SQL injection" attacks.
This entry was created by members of the CWE AI Working Group during June and July 2024. The CWE Project Lead, CWE Technical Lead, AI WG co-chairs, and many WG members decided that for purposes of timeliness, it would be more helpful to the CWE community to publish the new entry in CWE 4.15 quickly and add to it in subsequent versions.

References

REF-1441

LLM02: Insecure Output Handling
OWASP.
https://genai.owasp.org/llmrisk/llm02-insecure-output-handling/

REF-1442

Validating Outputs
Cohere, Guardrails AI.
https://cohere.com/blog/validating-llm-outputs

REF-1443

NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, Jonathan Cohen.
https://aclanthology.org/2023.emnlp-demo.40/

REF-1444

Insecure output handling in LLMs
Snyk.
https://learn.snyk.io/lesson/insecure-input-handling/

REF-1445

Building Guardrails for Large Language Models
Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, Xiaowei Huang.
https://arxiv.org/pdf/2402.01822

Submission

Modifications