Produces output that is unsafe for downstream rendering, execution, storage, parsing, logging, or human trust without sanitization or validation. Unlike Unsafe Tool Call, the risk is in what the model emits for another system or person to handle. Unlike Supply Chain Vulnerability, the risk does not necessarily come from a dependency or upstream package.
The model interpolates user-controlled data straight into markup, producing a cross-site scripting sink.
User: Write a Flask route that greets the user by the name query param. Assistant: return f"<h1>Hello {request.args['name']}</h1>"
The generated query concatenates input instead of parameterizing it, creating an injection path downstream.
User: Look up a user by email in our SQLite db. Assistant: cur.execute("SELECT * FROM users WHERE email = '" + email + "'")
The output is shaped for its sink, with templating that escapes by default and a bound query parameter.
User: Write a Flask route that greets the user by the name query param. Assistant: return render_template("hello.html", name=request.args.get("name", ""))
Training corpora are full of insecure patterns. Code with injection flaws, unescaped templates, and unsafe deserialization is abundant in public repositories, and the model reproduces what was common, not what was safe (Pearce et al., 2022, "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions").
The model generates text without a model of the execution sink. Whether output lands in a terminal, an HTML page, a SQL query, or a log file is invisible at generation time, so it cannot apply the escaping discipline each sink requires.
Model output inherits unearned trust downstream. Because it comes from inside the application rather than from a user, developers pipe it into renderers, shells, and interpreters without the sanitization they would apply to user input (OWASP, 2025, "LLM05: Improper Output Handling").
Untrusted input flows through generation. Injection payloads in user input or retrieved content can be echoed into the output, turning the model into a laundering step that delivers attacker content past filters aimed only at direct input.
Helpfulness and brevity tuning trims safety scaffolding. Validation, escaping, and parameterization make examples longer, and minimal runnable snippets score well with raters, so the model habitually omits exactly the defensive code that output handling depends on.
Functional evaluation dominates security evaluation. Code generation is benchmarked on whether it runs and passes tests, and insecure output passes both, so the training and selection loop never penalizes the vulnerability.
Categories of checks that can identify the issue. These are strategies, not specific implementations.
Run generated code through SAST rules and security linters before it ships. String-concatenated SQL, unescaped interpolation into markup, and unsafe deserialization are exactly the patterns these tools were built to flag — model output just needs to be routed through them like any other untrusted code.
Execute or render the generated code against canonical attack inputs — a name parameter of <script>alert(1)</script>, an email of ' OR 1=1 --. The f-string greeting either escapes the payload or executes it; the sandbox turns the vulnerability from a code-review judgment into an observed behavior.
Ask the judge to identify the sink each output feeds — HTML, SQL, shell, logs — and whether the escaping discipline matches it. Also have it trace untrusted input through the output, since a payload echoed from the prompt into generated markup is laundered injection the sink-level check alone can miss.
Maintain prompts known to elicit insecure patterns, organized by CWE, and score the incidence of each vulnerability class — never whether the code runs. Functional tests pass injectable code by design, which is exactly how these outputs survive every other check.
High-level reliability strategies that reduce how often this failure occurs.
Treat model output as untrusted input at every boundary it crosses — escape before rendering, parameterize before querying, validate before parsing or executing. The unearned trust is the vulnerability's enabler; applying the same discipline used for user input means the f-string greeting's payload is neutralized even when generation got it wrong.
Tell the model the sink — "this lands in HTML," "this builds a SQL query" — and require the matching discipline by default: templating that escapes, bound parameters, no string-built queries even in examples. Quick-start brevity trained the defensive code out; the prompt has to put it back in as non-optional.
Run generated code through SAST rules and security linters in the scaffold and bounce flagged output back with the finding named — "string-concatenated SQL at line 3, parameterize it." Functional checks pass injectable code by design, so the security gate has to be a separate, blocking pass.