[GH-ISSUE #53] Router-swappable experts: has profile-based routing been considered for DeepSeekMoE? #43

New issue

Open

opened 2026-10-26 21:02:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:02:02 +00:00

Owner

Originally created by @washingtoneimae-dot on GitHub (Jul 31, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/53

The DeepSeekMoE architecture uses fine-grained experts with a learned router (top-K selection via affinity scores). This works well but means expert pools are locked — swapping or adding an expert requires retraining the router.

Has the team considered replacing the learned router with profile-based routing? Specifically:

Each expert carries a calibrated profile vector (benchmark scores: MMLU, HumanEval, GSM8K, etc.)
A prompt profiler φ(x) maps input → profile space
Router: cos_sim(φ(x), expert_profiles) → softmax → top-K

This makes experts swappable (update one array, no retraining) and routing decisions fully interpretable ("Expert 3 was chosen because its math profile score of 0.97 matched the input's math profile of 0.94").

We built a working implementation proving this at small scale (99.9% routing accuracy, 38× swap isolation, identical speed to learned routing). The router has zero learned parameters — cosine similarity is pure math — reducing routing parameters from 106M (V3 scale) to ~500K (shared profiler).

Code + proofs: https://github.com/washingtoneimae-dot/profile-moe

Would love to hear if this direction has been explored internally and what challenges you'd anticipate at production scale.

Originally created by @washingtoneimae-dot on GitHub (Jul 31, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/53 The DeepSeekMoE architecture uses fine-grained experts with a learned router (top-K selection via affinity scores). This works well but means expert pools are locked — swapping or adding an expert requires retraining the router. Has the team considered replacing the learned router with profile-based routing? Specifically: 1. Each expert carries a calibrated profile vector (benchmark scores: MMLU, HumanEval, GSM8K, etc.) 2. A prompt profiler φ(x) maps input → profile space 3. Router: cos_sim(φ(x), expert_profiles) → softmax → top-K This makes experts swappable (update one array, no retraining) and routing decisions fully interpretable ("Expert 3 was chosen because its math profile score of 0.97 matched the input's math profile of 0.94"). We built a working implementation proving this at small scale (99.9% routing accuracy, 38× swap isolation, identical speed to learned routing). The router has zero learned parameters — cosine similarity is pure math — reducing routing parameters from 106M (V3 scale) to ~500K (shared profiler). Code + proofs: https://github.com/washingtoneimae-dot/profile-moe Would love to hear if this direction has been explored internally and what challenges you'd anticipate at production scale.

No labels

pull-request

WIP

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepSeek-MoE#43

No description provided.

Rows
Columns