One Hyphen Almost Took Down Our System: The Hidden World of Unicode Dashes

The short version

A customer copied an order number from Microsoft Word, and our system returned “order not found”. After hours of debugging, the strings looked identical to the naked eye. A single hexdump revealed the truth: one was ASCII 0x2D, the other was Unicode 0xE2 0x80 0x93. The bug was not in our logic — it was in the Unicode dash variants silently introduced by editors, input methods, and copy-paste.

This post explains why Unicode contains 20+ different “horizontal lines”, how they sneak into production data, and how to defend your system with input sanitization, normalization, and cross-platform automation scripts.

You have probably lived through this scenario: a customer reports “order not found”, you paste the order number into the admin panel, hit Enter, and the record is right there. You copy the customer’s original message, paste it again, and the search returns nothing. You place the two strings side by side on your monitor, zoom in, and they look exactly the same.

This is the classic Unicode dash trap: your eyes are correct, but the bytes are wrong.

Figure 1: Original cover illustration. All code, characters, and outputs in this post are synthetic teaching data. No real customer information is included.

1. Background: An Order Number That Haunted Us

We had a simple order lookup endpoint: accept an order ID string, run SELECT * FROM orders WHERE id = ?, and return the result.

One Friday afternoon, support escalated a ticket: the customer entered order-2024-001 and got “order not found”. Support used the exact same string in the backend and the record appeared immediately.

The first suspect was caching. Cache cleared — no change. The second suspect was database replication lag. Binlog inspected — clean. The third suspect was request encoding. Request and response bodies captured — byte-for-byte identical.

Finally, someone saved the customer’s raw text to a file and ran hexdump -C. The truth surfaced:

• Support input: 6f 72 64 65 72 2d 32 30 32 34 2d 30 30 31
• Customer input: 6f 72 64 65 72 e2 80 93 32 30 32 34 e2 80 93 30 30 31

0x2D is the ASCII Hyphen-Minus, the character produced by pressing the minus key on a standard keyboard. 0xE2 0x80 0x93 is the Unicode EN DASH (U+2013), a longer dash commonly produced by Word’s AutoCorrect, rich-text web pages, or input methods with “smart punctuation” enabled.

The customer did nothing wrong. They simply copied the order number from a Word document, and Word helpfully replaced -- with –, or the input method had smart punctuation turned on. The Unicode character traveled unchanged through the clipboard and into our system.

2. Symptoms: Identical to the Eye, Different to the Machine

What makes this class of bug so painful is that every standard debugging technique tells you “the data is fine”.

2.1 String comparison fails

s1 = "order-2024-001"   # Support input
s2 = "order–2024–001"   # Customer input
print(s1 == s2)         # False
print(len(s1))          # 14
print(len(s2))          # 14

Same length, same printed output, but == returns False. In most terminals and editors, the two strings are visually indistinguishable.

Figure 2: Real Python terminal output. s1 and s2 look identical to the human eye, but == returns False and their UTF-8 encodings are completely different.

2.2 Database queries return empty

SELECT * FROM orders WHERE id = 'order-2024-001';
-- 1 row returned

SELECT * FROM orders WHERE id = 'order–2024–001';
-- 0 rows returned

Databases do not perform Unicode normalization for you. = is a byte-level comparison; if the bytes differ, the rows do not match.

Figure 3: Real SQLite in-memory database query. Same table, same query pattern — only the ASCII hyphen version matches.

2.3 grep and logs cannot save you

grep 'order-2024-001' /var/log/app.log
# Matches only the ASCII version; EN DASH lines are silently skipped

Figure 4: Real grep output. The log clearly contains “similar” order numbers, but grep only performs byte-level exact matching.

3. Analysis: Why Unicode Has So Many “Horizontal Lines”

To understand this problem, we need to go back to the ASCII era.

3.1 The ASCII compromise: one character, three jobs

When ASCII was designed in the 1960s, character space was precious. The minus key was assigned to 0x2D with the official name Hyphen-Minus. In practice, it served three completely different purposes:

1. Hyphen: joining words, e.g., well-known
2. Minus sign: arithmetic, e.g., 5 - 3
3. Dash: indicating ranges or breaks, e.g., pages 10-20

This “one size fits all” approach made sense for typewriters, but it became a disaster in the age of professional typesetting and internationalization. Different languages and contexts demand different lengths, line-breaking rules, and semantics for “horizontal lines”.

3.2 The Unicode solution: one code point per line

The Unicode Consortium decided to stop compromising. They assigned a unique code point to each typographically distinct dash:

Figure 5: Original comparison chart. Left to right: code point, character, official name, typical usage. Note that U+2212 MINUS SIGN lives in the Mathematical Operators block, not General Punctuation.

Beyond these 9 core members, there are also Soft Hyphen (U+00AD, invisible), Armenian Hyphen (U+058A), Hebrew Maqaf (U+05BE), Mongolian Todo Soft Hyphen (U+1806), Canadian Syllabics Hyphen (U+1400), Hyphen Bullet (U+2043), Two-Em Dash (U+2E3A), Three-Em Dash (U+2E3B), Wave Dash (U+301C), Wavy Dash (U+3030), Katakana-Hiragana Prolonged Sound Mark (U+30FC), Vertical Em Dash (U+FE31), Vertical En Dash (U+FE32), Small Em Dash (U+FE58), Small Hyphen-Minus (U+FE63), and more.

Over 20 variants in total. Most of them render as a simple horizontal line in common fonts, but their Unicode code points, UTF-8 encodings, semantics, and line-breaking rules are all different.

3.3 How these characters infiltrate your system

Figure 6: Original diagram. Copy-paste is the #1 carrier of Unicode dash variants into production systems.

1. Microsoft Word AutoCorrect: Word automatically replaces -- with – (En Dash) and --- with — (Em Dash). When users copy text, these characters are carried along unchanged.
2. macOS Smart Punctuation: The system-wide “smart quotes and dashes” setting replaces straight quotes and hyphens as you type.
3. Web rich-text copy: Many professionally typeset websites use proper Unicode punctuation. Copying from these pages preserves the original characters.
4. Excel / CSV exports: If the original cell contains fullwidth or special dashes, the exported file retains them exactly.

4. Root Cause: Byte-Level “Identical Twins”

Let us pin this down at the byte level with hexdump:

Figure 6: Real hexdump -C output. ASCII hyphen is 1 byte 0x2D; EN DASH is 3 bytes 0xE2 0x80 0x93; Fullwidth is 3 bytes 0xEF 0xBC 0x8D.

Key observations:

• order-2024-001 (ASCII): 15 bytes
• order–2024–001 (EN DASH): 19 bytes
• order－2024－001 (Fullwidth): 19 bytes

Different lengths, different bytes, but potentially identical appearance. This is the root cause of every “ghost bug” in this category.

A useful analogy: the ASCII hyphen and Unicode dashes are like identical twins. They wear the same clothes and stand in front of you, and you cannot tell them apart. But a DNA test (byte-level inspection) will prove they are completely different people.

5. The Fix: A Three-Layer Defense

5.1 Layer 1: Input Sanitization

At the moment data enters your system, map all Unicode dash variants to the ASCII Hyphen-Minus.

import unicodedata

DASH_MAP = {
    "\u2010": "-",  # U+2010 HYPHEN
    "\u2011": "-",  # U+2011 NON-BREAKING HYPHEN
    "\u2012": "-",  # U+2012 FIGURE DASH
    "\u2013": "-",  # U+2013 EN DASH
    "\u2014": "-",  # U+2014 EM DASH
    "\u2015": "-",  # U+2015 HORIZONTAL BAR
    "\u2212": "-",  # U+2212 MINUS SIGN
    "\uFF0D": "-",  # U+FF0D FULLWIDTH HYPHEN-MINUS
}

def sanitize(text: str) -> str:
    # NFKC normalization: fold compatibility characters to basic forms
    text = unicodedata.normalize("NFKC", text)
    # Explicit mapping for remaining special dashes
    for bad, good in DASH_MAP.items():
        text = text.replace(bad, good)
    return text

Figure 7: Real Python output. Five different “dash” inputs are all normalized to the ASCII - after sanitization.

5.2 Layer 2: Normalization

unicodedata.normalize("NFKC", text) performs Unicode compatibility decomposition followed by canonical composition. It will:

• Convert fullwidth characters (e.g., Ａ) to halfwidth (A)
• Fold certain compatibility dashes (like U+FF0D) to ASCII -
• But it will not fold all dashes (EN DASH U+2013 remains unchanged under NFKC)

Therefore, NFKC and the explicit DASH_MAP must be used together.

5.3 Layer 3: Validation

After sanitization, enforce a whitelist or regex on critical fields:

import re

ORDER_ID_PATTERN = re.compile(r"^[A-Za-z0-9\-]+$")

def validate_order_id(order_id: str) -> bool:
    return bool(ORDER_ID_PATTERN.match(order_id))

Figure 8: Original flowchart. Raw Input → NFKC Normalize → Dash Mapping → Validation → Storage. All five steps are essential.

5.4 Database-layer supplement

If historical data already contains Unicode dashes, you can perform runtime mapping during queries:

-- PostgreSQL example: runtime mapping with translate
SELECT * FROM orders
WHERE translate(id, '‐‑‒–—―−－', '--------') = 'order-2024-001';

-- Or create a functional index
CREATE INDEX idx_orders_id_normalized
ON orders (translate(id, '‐‑‒–—―−－', '--------'));

6. One-Click Automation Scripts

The following scripts cover Windows 11, Ubuntu 26.04, and macOS 26. They require no third-party services and can be executed manually or handed to an AI Agent for automatic configuration.

6.1 Windows 11 (PowerShell)

# sanitize-dashes.ps1
# Usage: .\sanitize-dashes.ps1 -InputFile "input.txt" -OutputFile "output.txt"

param(
    [Parameter(Mandatory=$true)]
    [string]$InputFile,
    [Parameter(Mandatory=$true)]
    [string]$OutputFile
)

$dashMap = @{
    [char]0x2010 = '-'  # HYPHEN
    [char]0x2011 = '-'  # NON-BREAKING HYPHEN
    [char]0x2012 = '-'  # FIGURE DASH
    [char]0x2013 = '-'  # EN DASH
    [char]0x2014 = '-'  # EM DASH
    [char]0x2015 = '-'  # HORIZONTAL BAR
    [char]0x2212 = '-'  # MINUS SIGN
    [char]0xFF0D = '-'  # FULLWIDTH HYPHEN-MINUS
}

$content = Get-Content -Path $InputFile -Raw -Encoding UTF8

foreach ($bad in $dashMap.Keys) {
    $content = $content.Replace($bad, $dashMap[$bad])
}

# NFKC normalization via .NET
$content = $content.Normalize([System.Text.NormalizationForm]::FormKC)

Set-Content -Path $OutputFile -Value $content -Encoding UTF8 -NoNewline
Write-Host "Sanitized file written to $OutputFile"

6.2 Ubuntu 26.04 (Bash + Python)

#!/bin/bash
# sanitize-dashes.sh
# Usage: ./sanitize-dashes.sh input.txt output.txt

set -euo pipefail

INPUT="${1:?Usage: $0 input.txt output.txt}"
OUTPUT="${2:?Usage: $0 input.txt output.txt}"

python3 - <<PYEOF
import unicodedata

DASH_MAP = {
    "\u2010": "-", "\u2011": "-", "\u2012": "-", "\u2013": "-",
    "\u2014": "-", "\u2015": "-", "\u2212": "-", "\uFF0D": "-",
}

with open("$INPUT", "r", encoding="utf-8") as f:
    content = f.read()

content = unicodedata.normalize("NFKC", content)
for bad, good in DASH_MAP.items():
    content = content.replace(bad, good)

with open("$OUTPUT", "w", encoding="utf-8") as f:
    f.write(content)

print(f"Sanitized: $INPUT -> $OUTPUT")
PYEOF

6.3 macOS 26 (Bash + Python)

#!/bin/bash
# sanitize-dashes-macos.sh
# Usage: ./sanitize-dashes-macos.sh input.txt output.txt

set -euo pipefail

INPUT="${1:?Usage: $0 input.txt output.txt}"
OUTPUT="${2:?Usage: $0 input.txt output.txt}"

python3 - <<PYEOF
import unicodedata

DASH_MAP = {
    "\u2010": "-", "\u2011": "-", "\u2012": "-", "\u2013": "-",
    "\u2014": "-", "\u2015": "-", "\u2212": "-", "\uFF0D": "-",
}

with open("$INPUT", "r", encoding="utf-8") as f:
    content = f.read()

content = unicodedata.normalize("NFKC", content)
for bad, good in DASH_MAP.items():
    content = content.replace(bad, good)

with open("$OUTPUT", "w", encoding="utf-8") as f:
    f.write(content)

print(f"Sanitized: $INPUT -> $OUTPUT")
PYEOF

6.4 Agent Auto-Configuration Method

If you use an AI Agent (such as Kimi Code, Claude Code, Cursor, etc.), you can hand it the following prompt directly:

Please create a Unicode dash sanitization tool for my project. Requirements:
1. Provide implementations in Python, Node.js, and Shell
2. Map U+2010-U+2015, U+2212, and U+FF0D to ASCII -
3. Perform NFKC normalization first, then explicit mapping
4. Include unit tests covering at least 5 different dash inputs
5. Use only standard libraries, no third-party dependencies

The Agent will generate the code, tests, and documentation automatically. You only need to review and merge.

7. Q&A

Q1: Why not just run a one-time UPDATE to fix all historical data?

You can, but be careful. If certain fields (like user nicknames or article titles) legitimately contain EN DASH, a blanket replacement will corrupt data. It is recommended to sanitize only machine-internal identifiers (order IDs, SKUs, UUIDs, etc.).

Q2: Will NFKC normalization change other characters too?

Yes. NFKC folds fullwidth letters, fullwidth digits, and various compatibility symbols into their basic forms. This is usually exactly what we want — but only if you know what you are doing. If your business needs to preserve fullwidth characters, skip NFKC and use only the explicit DASH_MAP.

Q3: Can the frontend intercept these characters during user input?

Yes, and it is recommended. Apply the same mapping in JavaScript before form submission:

const DASH_MAP = {
  '\u2010': '-', '\u2011': '-', '\u2012': '-', '\u2013': '-',
  '\u2014': '-', '\u2015': '-', '\u2212': '-', '\uFF0D': '-',
};
function sanitize(text) {
  text = text.normalize('NFKC');
  for (const [bad, good] of Object.entries(DASH_MAP)) {
    text = text.split(bad).join(good);
  }
  return text;
}

However, frontend sanitization must never replace backend sanitization — malicious users can bypass the frontend and call the API directly.

Q4: Is there a more general solution beyond just dashes?

Yes. Unicode provides UTS #39 Unicode Security Mechanisms, which specifically handles “confusables” — characters that look alike but are different. Python’s confusable_homoglyphs library and Node.js’s unicode-confusables implement this standard. For most business scenarios, however, explicitly mapping 8–10 common dashes is sufficient.

Q5: Why do EN DASH and Hyphen look different in my editor but identical to the customer?

Font differences. Monospace fonts (Menlo, Consolas, JetBrains Mono) usually design distinct widths for different dashes, making them easy to distinguish. But proportional fonts (Arial, Helvetica, PingFang) may render them almost identically. Customers typically use Word, web pages, and mobile apps — all of which use proportional fonts.

8. Conclusion

The Unicode dash problem is a classic “invisible bug”: the data looks perfectly normal, but the bytes have already deviated from expectations. Its root cause is not programmer error, but the complexity of the entire digital ecosystem — Word’s AutoCorrect, OS-level smart punctuation, and rich-text copy-paste all silently alter user input.

The core defense against this class of problem can be summarized in one sentence: do not trust the appearance of user input; trust the bytes.

Perform NFKC normalization + explicit dash mapping at the moment data enters the system. Enforce whitelist validation on critical fields. Consider compatibility for historical data at the database layer. With these three layers of defense in place, your system will remain rock-solid even if a customer copies an order number from a document written in Martian.

All code in this post was tested on macOS 26 / Python 3.14. All screenshots are real terminal outputs with host and path information redacted.