[GH-ISSUE #863] [Research Report] Empirical Identification of Chinese CoT Trigger in DeepSeek-R1/V3 — 50 Controlled Runs, Root Cause Analysis #715

New issue

Closed

opened 2026-10-26 21:25:56 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:25:56 +00:00

Owner

Originally created by @AbuAnsari-06 on GitHub (Jun 29, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/863

Overview

I conducted an independent empirical investigation into the Chinese Chain-of-Thought (CoT) language-switching behavior observed in DeepSeek-R1/V3. This is not a complaint or a feature request — it is a structured research report with controlled test data, a proposed root cause mechanism, and a concrete recommendation for the development team.

All testing was performed manually on the DeepSeek web interface (chat.deepseek.com) across 50 runs in June 2026. Each run was conducted in a completely fresh, separate browser window to eliminate session carryover.

The Anomaly

When submitting English-language prompts, DeepSeek-R1/V3 sometimes produces Chinese-language Chain-of-Thought reasoning despite the prompt being entirely in English. This behavior:

Occurs silently — the final response may still appear in English while the internal reasoning is entirely in Chinese
Cannot be suppressed by a "Respond in English" system override at the prompt level
Does not occur on all prompts — it is trigger-dependent

Methodology

Platform: DeepSeek web chat (chat.deepseek.com) — not the API
Isolation: Each individual run was conducted in a fresh, separate browser window with no prior conversation history
Detection: Chinese CoT was identified by direct visual inspection of the model's thinking block (the collapsible reasoning pane in DeepSeek's web UI) and the final response text
Structure: 5 test prompts × 10 runs each = 50 total runs. Each prompt was run 5 times with a System Override instruction and 5 times without
System Override used: SYSTEM OVERRIDE: Respond exclusively in US English. prepended to the prompt

Test Results

Test	Prompt Condition	Override: Chinese CoT	Override: Chinese Output	No Override: Chinese CoT	No Override: Chinese Output
T1	Full GRC prompt — dense SOC2/ISO 27001 compliance jargon + scoring out of 10 + maturity tiers + 10-day sprint	5/5 (100%)	0/5 (0%)	5/5 (100%)	3/5 (60%)
T2	Same structural elements as T1 (tiers, sprint, scoring, tools) but stripped of dense explanatory prose	0/5 (0%)	0/5 (0%)	5/5 (100%)	4/5 (80%)
T3	Mobile dev language ranking — scoring format: out of 10	2/5 (40%)	0/5 (0%)	5/5 (100%)	1/5 (20%)
T4	Mobile dev language ranking — scoring format: percentile 0–100	4/5 (80%)	0/5 (0%)	4/5 (80%)	1/5 (20%)
T5	Mobile dev language ranking — scoring format: qualitative High/Medium/Low	0/5 (0%)	0/5 (0%)	5/5 (100%)	2/5 (40%)

Key Findings

Finding 1 — System Override controls output but not CoT
The override instruction was 100% effective at suppressing Chinese final output across all conditions. However, it had zero measurable effect on CoT language when technical detail density was high (T1: 5/5 Chinese CoT even with override). This dissociation confirms that CoT language is determined at an earlier processing stage than the output decoder.

Finding 2 — Detail density is the primary variable
T1 and T2 used structurally identical elements (maturity tiers, time-boxing, numeric scoring, specific tooling) but differed only in explanatory prose density. T1 (full narrative prompt) triggered 100% Chinese CoT with override. T2 (skeletal prompt, same structure) triggered 0% Chinese CoT with override. This isolates high detail density — not structural elements alone — as the primary activator.

Finding 3 — Numeric scoring format is the secondary variable
Holding topic and structure constant across T3, T4, and T5 (all mobile dev language rankings), varying only the scoring format produced dramatically different CoT rates under override:

out of 10 → 40% Chinese CoT
percentile 0–100 → 80% Chinese CoT (worse than /10)
High / Medium / Low → 0% Chinese CoT

The numeric precision of the scoring format — not merely the presence of an evaluation request — activates the trigger.

Proposed Root Cause

DeepSeek-R1's MoE router classifies task type during the early transformer layers. Prompts combining high technical detail density with numeric scoring formats are statistically overrepresented in Chinese-language technical training corpora (CSDN, Zhihu). The router assigns these tasks to Chinese-language reasoning experts before any system-prompt override can influence the generation path.

This is consistent with:

DeepSeek's published RLVR findings documenting a 5.6% accuracy drop when single-language CoT is enforced
The community observation that CoT language is uncontrollable from the user side

This behavior appears to be a design trade-off between reasoning accuracy and language consistency, not a conventional bug.

Recommendation

No user-side prompt engineering can fully override the router's early-stage classification. The only viable solution is an API-level parameter such as:

"reasoning_language": "en"

This would allow users to explicitly control CoT language without affecting the model's expert routing or accuracy.

Full Report

Complete methodology, all 5 prompt texts, the full 50-run data table, root cause analysis, limitations, and recommendations are documented in a formal report available here:

👉 GitHub Repository — Full Report & Data

Independent research conducted by Abu Horairah Ansari, B.Tech Computer Science student. No institutional affiliation. Raw test data available on request.

Originally created by @AbuAnsari-06 on GitHub (Jun 29, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/863 ## Overview I conducted an independent empirical investigation into the Chinese Chain-of-Thought (CoT) language-switching behavior observed in DeepSeek-R1/V3. This is not a complaint or a feature request — it is a structured research report with controlled test data, a proposed root cause mechanism, and a concrete recommendation for the development team. All testing was performed manually on the DeepSeek web interface (chat.deepseek.com) across 50 runs in June 2026. Each run was conducted in a completely fresh, separate browser window to eliminate session carryover. --- ## The Anomaly When submitting English-language prompts, DeepSeek-R1/V3 sometimes produces Chinese-language Chain-of-Thought reasoning despite the prompt being entirely in English. This behavior: - Occurs silently — the final response may still appear in English while the internal reasoning is entirely in Chinese - Cannot be suppressed by a "Respond in English" system override at the prompt level - Does not occur on all prompts — it is trigger-dependent --- ## Methodology - **Platform:** DeepSeek web chat (chat.deepseek.com) — not the API - **Isolation:** Each individual run was conducted in a fresh, separate browser window with no prior conversation history - **Detection:** Chinese CoT was identified by direct visual inspection of the model's thinking block (the collapsible reasoning pane in DeepSeek's web UI) and the final response text - **Structure:** 5 test prompts × 10 runs each = 50 total runs. Each prompt was run 5 times with a System Override instruction and 5 times without - **System Override used:** `SYSTEM OVERRIDE: Respond exclusively in US English.` prepended to the prompt --- ## Test Results | Test | Prompt Condition | Override: Chinese CoT | Override: Chinese Output | No Override: Chinese CoT | No Override: Chinese Output | |------|-----------------|----------------------|--------------------------|--------------------------|------------------------------| | T1 | Full GRC prompt — dense SOC2/ISO 27001 compliance jargon + scoring out of 10 + maturity tiers + 10-day sprint | 5/5 (100%) | 0/5 (0%) | 5/5 (100%) | 3/5 (60%) | | T2 | Same structural elements as T1 (tiers, sprint, scoring, tools) but stripped of dense explanatory prose | 0/5 (0%) | 0/5 (0%) | 5/5 (100%) | 4/5 (80%) | | T3 | Mobile dev language ranking — scoring format: out of 10 | 2/5 (40%) | 0/5 (0%) | 5/5 (100%) | 1/5 (20%) | | T4 | Mobile dev language ranking — scoring format: percentile 0–100 | 4/5 (80%) | 0/5 (0%) | 4/5 (80%) | 1/5 (20%) | | T5 | Mobile dev language ranking — scoring format: qualitative High/Medium/Low | 0/5 (0%) | 0/5 (0%) | 5/5 (100%) | 2/5 (40%) | --- ## Key Findings **Finding 1 — System Override controls output but not CoT** The override instruction was 100% effective at suppressing Chinese final output across all conditions. However, it had zero measurable effect on CoT language when technical detail density was high (T1: 5/5 Chinese CoT even with override). This dissociation confirms that CoT language is determined at an earlier processing stage than the output decoder. **Finding 2 — Detail density is the primary variable** T1 and T2 used structurally identical elements (maturity tiers, time-boxing, numeric scoring, specific tooling) but differed only in explanatory prose density. T1 (full narrative prompt) triggered 100% Chinese CoT with override. T2 (skeletal prompt, same structure) triggered 0% Chinese CoT with override. This isolates high detail density — not structural elements alone — as the primary activator. **Finding 3 — Numeric scoring format is the secondary variable** Holding topic and structure constant across T3, T4, and T5 (all mobile dev language rankings), varying only the scoring format produced dramatically different CoT rates under override: - `out of 10` → 40% Chinese CoT - `percentile 0–100` → 80% Chinese CoT (worse than /10) - `High / Medium / Low` → 0% Chinese CoT The numeric precision of the scoring format — not merely the presence of an evaluation request — activates the trigger. --- ## Proposed Root Cause DeepSeek-R1's MoE router classifies task type during the early transformer layers. Prompts combining high technical detail density with numeric scoring formats are statistically overrepresented in Chinese-language technical training corpora (CSDN, Zhihu). The router assigns these tasks to Chinese-language reasoning experts before any system-prompt override can influence the generation path. This is consistent with: - DeepSeek's published RLVR findings documenting a **5.6% accuracy drop** when single-language CoT is enforced - The community observation that CoT language is uncontrollable from the user side This behavior appears to be a **design trade-off between reasoning accuracy and language consistency**, not a conventional bug. --- ## Recommendation No user-side prompt engineering can fully override the router's early-stage classification. The only viable solution is an **API-level parameter** such as: ```json "reasoning_language": "en" ``` This would allow users to explicitly control CoT language without affecting the model's expert routing or accuracy. --- ## Full Report Complete methodology, all 5 prompt texts, the full 50-run data table, root cause analysis, limitations, and recommendations are documented in a formal report available here: 👉 [GitHub Repository — Full Report & Data](https://github.com/AbuAnsari-06/deepseek-cot-language-investigation) --- *Independent research conducted by Abu Horairah Ansari, B.Tech Computer Science student. No institutional affiliation. Raw test data available on request.*

bsakaguchi

2026-10-26 21:25:56 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:25:57 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 30, 2026):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Jul 30, 2026): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:25:57 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 14, 2026):

false

@github-actions[bot] commented on GitHub (Aug 14, 2026): false