"Unexpected End of Hex Escape": Cutting Text Safely for LLM APIs

Here is a line of code that looks harmless in any codebase that builds prompts:

const subject = email.subject.slice(0, 40);

Put that subject into a prompt, send the prompt to an LLM API, and most of the time nothing happens. But if character 40 lands in the middle of an emoji, the API can reject the whole request with something like:

400 invalid_request_error: unexpected end of hex escape

And it will keep rejecting it, on every request that includes that subject, until the subject leaves the prompt. One short email title can quietly break every reply to one person. Here is why, and how we made our prompt-building code immune to it.

Why a cut can break JSON

JavaScript strings are sequences of UTF-16 code units. Characters outside the Basic Multilingual Plane, which includes nearly every emoji, take two code units: a high surrogate followed by a low surrogate.

"😀".length          // 2
"😀".charCodeAt(0)   // 0xD83D (high)
"😀".charCodeAt(1)   // 0xDE00 (low)

.slice(), .substring() and .length all count code units, not characters. So "Lunch 😀".slice(0, 7) keeps the high half and drops the low half. The result is a string with a lone surrogate: not valid Unicode, but a perfectly legal JavaScript string.

Since ES2019, JSON.stringify writes a lone surrogate as an escape instead of emitting invalid UTF-8:

JSON.stringify("Lunch 😀".slice(0, 7))  // "\"Lunch \\ud83d\""

That escape is valid JSON syntax, but it encodes text that is not valid Unicode. A strict parser on the other end, which is what you want guarding an API, refuses it. The error mentions a hex escape because that is where the parser gave up.

The wording depends on the provider, but the cause is the same, and it is not rare. In the last few months, open-source AI clients have shipped fixes for this exact failure against several different model APIs, usually after one bad string got saved into a conversation’s history and broke every request after it.

If you write Ruby or Python 3, you mostly get to skip this post: their string indexes count code points, so s[0, 40] cannot split a pair. JavaScript, TypeScript, Java and C# count UTF-16 units, and they can.

Why it hides

Three things make this bug easy to ship and hard to see:

It needs an exact alignment. The cut has to land precisely between the two halves. A 40-character cap only fails on strings where an emoji straddles position 40.
It fails the whole request. The lone surrogate can sit in a minor field (a subject line in a list of recent mail) and still take down the entire call.
It persists. A string that sits in context for a day fails every request for a day. Retrying does not help, because the retry sends the same bytes.

Fix 1: cut on code-point boundaries

We replaced raw cuts of user, vendor and model text with one helper that has the same signature as String.prototype.slice and only ever moves a bound that falls inside a surrogate pair:

const isHigh = (c: number) => c >= 0xd800 && c <= 0xdbff;
const isLow = (c: number) => c >= 0xdc00 && c <= 0xdfff;

const bound = (s: string, i: number) =>
  i < 0 ? Math.max(s.length + i, 0) : Math.min(i, s.length);

const splitsPair = (s: string, i: number) =>
  i > 0 && i < s.length && isHigh(s.charCodeAt(i - 1)) && isLow(s.charCodeAt(i));

export function safeSlice(s: string, start = 0, end = s.length): string {
  let a = bound(s, start);
  let b = bound(s, end);
  if (splitsPair(s, a)) a += 1; // a start inside a pair moves forward
  if (splitsPair(s, b)) b -= 1; // an end inside a pair moves back
  return b > a ? s.slice(a, b) : "";
}

Two properties matter. It handles negative indexes like slice, so it is a drop-in replacement. And the result is never longer than slice would have returned, so no length cap anywhere quietly grows.

A note on scope: this guarantees well-formed strings, which is what an API needs. It does not keep a grapheme intact. A family emoji built from several code points joined by zero-width joiners can still be cut between its parts. For text a person will read, Intl.Segmenter with granularity: "grapheme" gives you cuts that look right too.

Fix 2: repair the request body as a backstop

Code changes. Someone will write a new .slice next month. So at the one place we serialize a request to the model API, we scan the JSON body for a lone surrogate escape and replace it with \ufffd, the Unicode replacement character. The request goes through with a visible “�” instead of failing.

Modern runtimes have this built in: String.prototype.isWellFormed() and toWellFormed() (ES2024) check and repair a string directly. Either way, the backstop turns a hard failure into a cosmetic one, and the helper keeps the cosmetic one from happening.

Fix 3: find every cut, by path, not by date

The tempting audit is “check the code that changed recently.” The bug that bites you is often older than that. A cut written months ago sits on a path that only now carries emoji-heavy text.

We searched the codebase for every cut of a string and sorted them by what they cut. About 740 cut sites came up. About 380 were safe as they were: ASCII identifiers, dates, hashes, cuts at an index we know is ASCII. The other 360 or so, everything that touches text a person, a vendor or a model wrote, moved onto the helper.

Fix 4: tests that put an emoji at every edge

The tests are what keep this fixed.

Exhaustive helper test. For a set of tricky strings (plain emoji, zero-width-joiner families, flags), assert every possible start and end. That is 9,624 assertions, and each checks that the result is well-formed and no longer than slice would give.

Poisoned prompt test. Build the whole prompt, the real builder with every context section, where every free-text input is filled with emoji. Do it in four alignments (even and odd offsets on both sides), so that a cut at any cap length splits a pair in at least one of them. Then walk every string in the resulting request and assert it is well-formed.

const POISON = { even: "😀".repeat(2500), odd: `a${"😀".repeat(2500)}` };

function wellFormedDeep(v: unknown): boolean {
  if (typeof v === "string") return v.isWellFormed();
  if (Array.isArray(v)) return v.every(wellFormedDeep);
  if (v && typeof v === "object") return Object.values(v).every(wellFormedDeep);
  return true;
}

The second test is the one that matters most. It does not care where the cut lives or who wrote it. If any code on the prompt path ever splits a pair again, the test fails in CI before the code ships.

The checklist

If your code builds prompts in JavaScript or TypeScript:

Search for .slice(, .substring( and .substr( on anything a user, vendor or model wrote.
Replace those with a code-point-safe cut, or Intl.Segmenter for text people read.
Repair the serialized request body as a backstop with toWellFormed() or an escape scan.
Add one test that builds the full prompt with emoji in every input, at more than one alignment.

It is an afternoon of work, and it removes a failure that is otherwise invisible until it happens to the one person whose inbox has the wrong subject line.

"Unexpected End of Hex Escape": Cutting Text Safely for LLM APIs
One emoji, one 400