Your LLM Prompt Won’t Save You From This
Ganta Hemanth
You are building a system that uses an LLM as an engine — not a chatbot, but a pipeline. The model translates documents, extracts data, generates structured content, and fills templates. The output goes directly into code. A stray character breaks things.
You write a careful prompt: Return ONLY the translated text, nothing else. Do NOT add explanations, notes, or commentary. The model mostly listens. Then, on 5% of calls, it returns something like this:
That response was supposed to be a JSON field value. Instead, it carried a code fence, a JSON wrapper, and a greeting. Inserted programmatically into a document, it corrupted the output and surfaced in production as a malformed record — no exception, no alert, just wrong data delivered confidently.
That response was supposed to slot into a JSON document as the value for a field called purpose. Instead, it carried a conversational opener, a code fence, and a JSON wrapper. Inserted programmatically, it corrupted the output and surfaced in production as a malformed document.
The prompt was good. The model didn’t follow it. Not always — just often enough to matter.
Why This Happens
LLMs are stochastic. Even at low temperatures, they don’t produce identical output for identical input. A model that follows formatting instructions 95% of the time still fails on 5% of calls.
In a system processing 1,000 segments per day, that’s 50 corrupted outputs daily.
The root cause isn’t a poorly-written prompt. It’s the nature of the model. LLMs were trained on enormous amounts of text that includes markdown code blocks, smart quotes, and conversational openers. Those patterns are baked in. No instruction permanently suppresses them.
Prompt engineering reduces the frequency. A sanitization layer eliminates the consequence.
Three Artifact Types
After analyzing production logs, every formatting problem fell into one of three categories. Here are real examples from our system — input text, what the model actually returned, and what the sanitizer produced:
Real examples from production logs: code fence, smart quote wrapping, and preamble — before and after sanitization.
Code fences appear when the input resembles structured data. The model wraps the translation in markdown fences, sometimes with a language hint. Quote wrapping is more subtle — the model surrounds the entire response in quotes, either ASCII or Unicode smart quotes that look identical at a glance but silently corrupt JSON parsing downstream. Preambles appear when the model decides to be conversational despite explicit instructions to the contrary.
Each pattern is trivial in isolation. Inserted into a JSON document being reconstructed programmatically, any one of them breaks the output completely.
The Fix
Three compiled regex patterns and a fallback:
Three decisions worth noting.
The code fence pattern uses re.DOTALL so . matches newlines. Without it, multi-line content inside a fence breaks the capture group and the fence passes through uncleaned.
The quote pattern handles three quote styles explicitly — ASCII ", Unicode left \u201c, and Unicode right \u201d. Mixed combinations appear in real logs: a response that opens with a smart quote and closes with a straight one. The character class ["\u201c\u201d] catches all variants.
Every preamble pattern anchors to ^. Without the anchor, "Translation:" buried mid-sentence would get stripped — corrupting legitimate content that uses the word "translation" as part of its text.
Why Not Just Fix the Prompt?
The prompt already says exactly that. Return ONLY the translated text, nothing else.
Adding more rules — “Do not use code fences,” “Do not wrap in quotes” — reduces frequency. It doesn’t reach zero. Temperature set to 0.1 still leaves residual randomness. More rules in the prompt also mean more input tokens on every request, repeated thousands of times per day. The cost compounds.
The right mental model: the prompt sets the expected case. The sanitizer handles the tail. They are not substitutes — both earn their place.
Where It Sits
The sanitizer lives at the client boundary, immediately after the raw response arrives. Here’s where it fits in the broader pipeline:
The sanitizer sits at the client boundary. If the result is empty after sanitization, the function falls back to the original source text. The Sarvam client skips this step entirely.
The sanitizer runs inside the OpenAI and Anthropic translation clients, immediately after the raw response arrives:
The Sarvam client — which uses a purpose-built translation model rather than a general-purpose LLM — doesn’t run the sanitizer at all. Sarvam’s Mayura model returns clean translated text without formatting artifacts. The sanitizer is a response to general-purpose LLM behavior specifically.
This placement matters. The service layer above these clients never sees raw LLM output. By the time a TranslationResult reaches the service, it already contains clean, usable text.
The Fallback Principle
The last two lines of the function are the most important:
If sanitization strips everything — because the model returned noise, or the patterns were too aggressive — the function returns the original source text rather than an empty string.
The worst case is untranslated text, not missing text. A field reading “Home renovation and repairs” in English is recoverable. A field reading "" corrupts the document silently and may propagate errors through downstream systems.
The segments_fallback counter in the API response tracks how often this fires. In a healthy system it should be near zero. If it starts climbing, something changed — possibly the model, possibly the input distribution — and the prompts or patterns need attention.
The Principle Generalizes
Translation is a specific instance of a broader problem. Any system that takes LLM output and uses it programmatically — inserting into templates, parsing as structured data, embedding in user-facing content — will eventually encounter formatting artifacts.
The patterns differ by use case. The shape of the problem doesn’t. The model was told to return clean text. Sometimes it won’t.
The solution is always some version of the same thing: a pure function that takes raw output and a safe fallback, strips known artifacts, and returns usable text. It’s cheap to build, trivial to test, and prevents a class of production bugs that are otherwise invisible until they compound.
Write the prompt carefully. Then write the sanitizer anyway.
© 2026 krtrimaIQ Cognitive Solutions. All Rights Reserved
Email: info@krtrimaiq.ai