[GH-ISSUE #849] Web-sourced text with non-standard Unicode characters renders as one character per line #701

New issue

Closed

opened 2026-10-26 21:25:41 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:25:41 +00:00

Owner

Originally created by @zafarhussainzh-sketch on GitHub (Jun 1, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/849

Labels: bug, output formatting, unicode

Environment:

Platform: DeepSeek (Web interface / API) — specify if using chat or API

Date observed: June 1, 2026

Input: A request for company financial data (Canaccord Genuity)

Steps to Reproduce:

Prompt the AI to search the web for recent financial data (e.g., "Canaccord Genuity assets under administration 2026").

The AI retrieves web content containing numeric data like "105.8 billion".

The AI integrates that text into a longer response paragraph without explicit sanitization.

Observe the output — the numeric value appears broken into single characters per line, with no spaces between words.

Expected behavior:
The text should render as a normal continuous string: "Assets Under Administration of
105.8
b
i
l
l
i
o
n
a
n
d
r
e
c
o
r
d
c
l
i
e
n
t
a
s
s
e
t
s
i
n
C
a
n
a
d
a
o
f
105.8billionandrecordclientassetsinCanadaof73 billion."

Actual behavior:
Each character of the problematic segment is printed on its own line, with no spaces, like:

text
105.8
b
i
l
l
i
o
n
...
Frequency:
Consistent when source text contains certain non-standard Unicode whitespace characters (e.g., soft hyphens, non-breaking spaces, zero-width spaces) that are misinterpreted as line breaks.

Root Cause Analysis:
The AI's text rendering pipeline appears to treat any character outside the standard ASCII or UTF-8 space set as a newline control character. When web-scraped content includes invisible formatting characters (U+00A0 NO-BREAK SPACE, U+00AD SOFT HYPHEN, U+200B ZERO WIDTH SPACE), they are not normalized to standard spaces before output. Instead, they trigger a line break, causing the text to fragment character by character.

Suggested Fix:

Input Sanitization: Before output, scan all generated text for Unicode control characters and replace them with their ASCII equivalents:

Replace U+00A0 (non-breaking space) with U+0020 (standard space)

Remove U+00AD (soft hyphen) entirely

Remove U+200B (zero-width space) entirely

Normalization Step: Apply Unicode normalization form NFKC to all output strings. This will collapse many ambiguous whitespace characters into standard spaces.

Post-processing Check: After final text assembly, run a regex check that flags any line containing only a single character followed by a newline — if found, rejoin those fragments before display.

Impact:
This bug corrupts numerical data, legal references, and any text containing formatted numbers, making the output unreadable and potentially causing confusion or errors in user-facing communications.

Additional Notes:
This behavior was observed on June 1, 2026, while processing a financial report. The same issue has been observed by me many times since last couple of months and is consist. May have affected other users when AI retrieves web data with non-standard characters or compiles and snatch data internally from other placeholders or documents. A robust sanitization layer would improve output reliability.

Originally created by @zafarhussainzh-sketch on GitHub (Jun 1, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/849 Labels: bug, output formatting, unicode Environment: Platform: DeepSeek (Web interface / API) — specify if using chat or API Date observed: June 1, 2026 Input: A request for company financial data (Canaccord Genuity) Steps to Reproduce: Prompt the AI to search the web for recent financial data (e.g., "Canaccord Genuity assets under administration 2026"). The AI retrieves web content containing numeric data like "105.8 billion". The AI integrates that text into a longer response paragraph without explicit sanitization. Observe the output — the numeric value appears broken into single characters per line, with no spaces between words. Expected behavior: The text should render as a normal continuous string: "Assets Under Administration of 105.8 b i l l i o n a n d r e c o r d c l i e n t a s s e t s i n C a n a d a o f 105.8billionandrecordclientassetsinCanadaof73 billion." Actual behavior: Each character of the problematic segment is printed on its own line, with no spaces, like: text 105.8 b i l l i o n ... Frequency: Consistent when source text contains certain non-standard Unicode whitespace characters (e.g., soft hyphens, non-breaking spaces, zero-width spaces) that are misinterpreted as line breaks. Root Cause Analysis: The AI's text rendering pipeline appears to treat any character outside the standard ASCII or UTF-8 space set as a newline control character. When web-scraped content includes invisible formatting characters (U+00A0 NO-BREAK SPACE, U+00AD SOFT HYPHEN, U+200B ZERO WIDTH SPACE), they are not normalized to standard spaces before output. Instead, they trigger a line break, causing the text to fragment character by character. Suggested Fix: Input Sanitization: Before output, scan all generated text for Unicode control characters and replace them with their ASCII equivalents: Replace U+00A0 (non-breaking space) with U+0020 (standard space) Remove U+00AD (soft hyphen) entirely Remove U+200B (zero-width space) entirely Normalization Step: Apply Unicode normalization form NFKC to all output strings. This will collapse many ambiguous whitespace characters into standard spaces. Post-processing Check: After final text assembly, run a regex check that flags any line containing only a single character followed by a newline — if found, rejoin those fragments before display. Impact: This bug corrupts numerical data, legal references, and any text containing formatted numbers, making the output unreadable and potentially causing confusion or errors in user-facing communications. Additional Notes: This behavior was observed on June 1, 2026, while processing a financial report. The same issue has been observed by me many times since last couple of months and is consist. May have affected other users when AI retrieves web data with non-standard characters or compiles and snatch data internally from other placeholders or documents. A robust sanitization layer would improve output reliability.

bsakaguchi

2026-10-26 21:25:41 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:25:43 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 2, 2026):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Jul 2, 2026): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:25:43 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 16, 2026):

false

@github-actions[bot] commented on GitHub (Jul 16, 2026): false