Unescape HTML entities last, or a tag-stripping regex eats your code

Every code block in the blog drafts I was reviewing appeared cut off at the
first <: a line like assertTrue(optimum and then, abruptly, prose. I spent
a while suspecting the publishing pipeline had corrupted the stored posts. The
posts were fine. The five-line script I had written to view them was doing
the damage.

TL;DR — When flattening HTML to plain text with regexes, decode entities
(html.unescape) exactly once, as the very last step. Decode any earlier and
every < stored inside a code block becomes a bare < that a later
tag-stripping pass will match as markup and delete.

The setup

I keep an engineering blog on WordPress. Articles are written in Markdown,
rendered to HTML, and pushed through the REST API as drafts. Code samples end
up inside <pre><code> blocks with their special characters entity-escaped:
if x <= 0: is stored as if x <= 0:. That escaping is the one fact this
story turns on. To proofread the drafts from a terminal, I fetched each post's
raw HTML (GET /wp-json/wp/v2/posts/<id>?context=edit) and flattened it to
text with a small throwaway Python script.

What I expected

The script pulled the <pre> blocks out, converted <h2> tags to Markdown
headings, and stripped whatever tags remained:

text = re.sub(
    r"<pre[^>]*>(.*?)</pre>",
    lambda m: "\n[CODE]\n" + html.unescape(re.sub(r"<[^>]+>", "", m.group(1))) + "\n[/CODE]\n",
    raw, flags=re.S)
text = re.sub(r"<h2[^>]*>", "\n## ", text)
text = re.sub(r"<[^>]+>", "", text)   # strip everything else
print(html.unescape(text))

Each piece looks defensible on its own. Code blocks need their entities
decoded to read naturally, so the lambda decodes them; the last line decodes
whatever remains. The middle pass strips any tag the earlier ones missed.

What actually happens

    if self.rotations_left

— and the rest of the block was gone, along with the [/CODE] marker after
it. The lambda had decoded <= back into <= and spliced that decoded
code into the document. The final tag-stripping pass then treated the
reconstituted < as the start of a tag and consumed everything up to the next
> in the document — which was the closing bracket of the next paragraph's
<p> tag, several lines away.

What stopped me from filing a bug against the publisher was checking the
stored bytes instead of my rendering of them:

i = raw.find("rotations_left")
print(repr(raw[i - 20 : i + 60]))
# '    if self.rotations_left &lt;= 0:\n        return False\n'

The <= was intact on the server. The corruption existed only inside my
viewer.

The fix

Never re-inject decoded text into a stream that a markup-aware pass will visit
again. The final html.unescape was already correct; the fix is deleting the
early one from the lambda:

text = re.sub(
    r"<pre[^>]*>(.*?)</pre>",
    lambda m: "\n[CODE]\n" + re.sub(r"<[^>]+>", "", m.group(1)) + "\n[/CODE]\n",
    raw, flags=re.S)
text = re.sub(r"<h2[^>]*>", "\n## ", text)
text = re.sub(r"<[^>]+>", "", text)
print(html.unescape(text))            # decode entities exactly once, last

The code blocks come out whole, <= and all.

Why it works

Entity escaping is the boundary between content and markup: in stored HTML, a
markup < is bare and a content < is <. Every markup-aware operation —
stripping tags, splitting on elements — depends on that distinction, and
html.unescape dissolves it. So decoding is not a per-block cleanup you apply
wherever convenient; it is the final exit from HTML into plain text, safe only
when no markup-aware pass runs after it. A real parser (html.parser,
BeautifulSoup's get_text()) separates structure from text before you ever
see an entity, which makes this mistake unrepresentable.

The debugging lesson is the same one as in
Pruning no-op moves by event type skips actions that change state
silently: when a rendering disagrees with
reality, check the stored state before blaming the writer. One repr() around
the suspect offset of the raw payload settled in a minute what staring at
pretty output could not.

What I did not test

Only HTML produced by python-markdown from my own posts. Entities other than
<, > and &, nested <pre> blocks, and CDATA sections were
never exercised.
I claim a real HTML parser avoids the ordering trap by construction; I did
not run BeautifulSoup on these posts to confirm it this session.
Python 3 only, html.unescape from the standard library.

Facts

context: proofreading WordPress drafts by fetching raw HTML over the REST API and flattening it to text with a few Python regexes
problem: code blocks appeared truncated at every "<" because an early html.unescape() turned stored "&lt;" back into a bare "<" that a later tag-stripping regex consumed as markup
solution: run html.unescape() exactly once, after every markup-aware pass; verify suspected corruption against the raw payload with repr() before blaming the writer
verified_on: 2026-08-16
applies_to: [regex-based HTML-to-text conversion, Python html.unescape, WordPress REST API content.raw]
does_not_apply_to: [conversions built on a real HTML parser that separates structure from text]