[GH-ISSUE #8] Clarification on ESFT Formulas, Expert Selection, and Code Consistency #4

New issue

Closed

opened 2026-10-26 21:06:36 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:06:36 +00:00

Owner

Originally created by @dog14230pp on GitHub (Sep 21, 2024).
Original GitHub issue: https://github.com/deepseek-ai/ESFT/issues/8

Dear authors,

I have been studying your excellent ESFT paper and the accompanying codebase, and I have a few questions that I hope you can help clarify.

(1) ESFT Formula:
In your paper, both the ESFT-Gate and ESFT-Token formulas include multiplication by 1/N_s and 1/L_j. However, in the generate_expert_config.py script, I did not find any corresponding code. Is there a specific reason these terms were omitted? For instance, could their inclusion potentially not affect the final results?

(2) Expert Selection Threshold:
Your paper mentions selecting task-relevant experts, where the scores based on both ESFT-Gate and ESFT-Token should range between 0 and 1, allowing for the application of a threshold, p, for expert selection. In the codebase, values are normalized across corresponding experts at the same position or index across different layers (link). I am curious why we do not normalize values across the experts within the same layer instead, for example, modifying the code to gate_scores = gate_scores / np.sum(gate_scores, axis=1, keepdims=True). It seems that normalization within the same layer could maintain the order of importance of experts, regardless of whether it is before or after normalization. Could this approach lead to significantly different results?

(3) Consistency of Expert Selection and Fine-tuning Formula:
I have concerns that the code provided in this section may not be consistent with the formula described in the paper. Could you please review this and confirm whether the paper or the code reflects the intended formula?

Thank you very much for your help. I greatly appreciate your time and am looking forward to your response.

Originally created by @dog14230pp on GitHub (Sep 21, 2024). Original GitHub issue: https://github.com/deepseek-ai/ESFT/issues/8 Dear authors, I have been studying your excellent ESFT paper and the accompanying codebase, and I have a few questions that I hope you can help clarify. **(1) ESFT Formula:** In your paper, both the ESFT-Gate and ESFT-Token formulas include multiplication by 1/N_s and 1/L_j. However, in the [generate_expert_config.py script](https://github.com/deepseek-ai/ESFT/blob/main/scripts/expert/generate_expert_config.py#L27), I did not find any corresponding code. Is there a specific reason these terms were omitted? For instance, could their inclusion potentially not affect the final results? **(2) Expert Selection Threshold:** Your paper mentions selecting task-relevant experts, where the scores based on both ESFT-Gate and ESFT-Token should range between 0 and 1, allowing for the application of a threshold, p, for expert selection. In the codebase, values are normalized across corresponding experts at the same position or index across different layers ([link](https://github.com/deepseek-ai/ESFT/blob/main/scripts/expert/generate_expert_config.py#L34)). I am curious why we do not normalize values across the experts within the same layer instead, for example, modifying the code to `gate_scores = gate_scores / np.sum(gate_scores, axis=1, keepdims=True)`. It seems that normalization within the same layer could maintain the order of importance of experts, regardless of whether it is before or after normalization. Could this approach lead to significantly different results? **(3) Consistency of Expert Selection and Fine-tuning Formula:** I have concerns that the code provided in [this section](https://github.com/deepseek-ai/ESFT/blob/main/scripts/expert/generate_expert_config.py#L82) may not be consistent with the formula described in the paper. Could you please review this and confirm whether the paper or the code reflects the intended formula? Thank you very much for your help. I greatly appreciate your time and am looking forward to your response.

bsakaguchi closed this issue

2026-10-26 21:06:36 +00:00

bsakaguchi commented

2026-10-26 21:06:37 +00:00

Author

Owner

@ZihanWang314 commented on GitHub (Sep 22, 2024):

Hi there - Thank you for your attention and questions.
(1) ESFT Formula: Please check generate_expert_config.py, L34-L35.

(2) Expert Selection Threshold:: It is absolutely reasonable whether or not to modify the gate value function with a normalization factor. You can try on your own dataset if you feel normalization can lead to better results. We did not adopt this since we have the top_p shared across layers, and we would like to keep the importance across all layers consistent (if normed, then the layers with a larger sum of experts' gate values will be having more experts tuned. You can verify this on your own).

(3) Consistency of Expert Selection and Fine-tuning Formula:: Could you specify your concerns? The code here looks fine to me. It selects the expert set whose aggregate score exceeds the top_p threshold.

Thank you again for your attention and questions!

Best,

@ZihanWang314 commented on GitHub (Sep 22, 2024): Hi there - Thank you for your attention and questions. **(1) ESFT Formula:** Please check [generate_expert_config.py](https://github.com/deepseek-ai/ESFT/blob/main/scripts/expert/generate_expert_config.py#L27), L34-L35. **(2) Expert Selection Threshold:**: It is absolutely reasonable whether or not to modify the gate value function with a normalization factor. You can try on your own dataset if you feel normalization can lead to better results. We did not adopt this since we have the **top_p** shared across layers, and we would like to keep the importance across all layers consistent (if normed, then the layers with a larger sum of experts' gate values will be having more experts tuned. You can verify this on your own). **(3) Consistency of Expert Selection and Fine-tuning Formula:**: Could you specify your concerns? The code here looks fine to me. It selects the expert set whose aggregate score exceeds the top_p threshold. Thank you again for your attention and questions! Best,

bsakaguchi commented

2026-10-26 21:06:37 +00:00

Author

Owner

@dog14230pp commented on GitHub (Sep 24, 2024):

Dear Zihan,

Thank you for your prompt and detailed reply!

For issues (1) and (2), I will experiment with my own dataset to determine which method yields the best results.

Regarding issue (3), your explanation has clarified my concerns! But I still have a minor suggestion: for formula (8) in the paper, it might be helpful to add a comment like, "For each layer l, we select a set of top-scored experts E^{l}_{s} whose cumulative relevance score first exceeds the threshold p." I believe this could make the explanation clearer and easier to understand, especially for readers referencing that specific passage.

Thank you again for taking the time to address my questions. :)

@dog14230pp commented on GitHub (Sep 24, 2024): Dear Zihan, Thank you for your prompt and detailed reply! For issues (1) and (2), I will experiment with my own dataset to determine which method yields the best results. Regarding issue (3), your explanation has clarified my concerns! But I still have a minor suggestion: for formula (8) in the paper, it might be helpful to add a comment like, "For each layer l, we select a set of top-scored experts E^{l}_{s} whose cumulative relevance score **first** exceeds the threshold p." I believe this could make the explanation clearer and easier to understand, especially for readers referencing that specific passage. Thank you again for taking the time to address my questions. :)

bsakaguchi commented

2026-10-26 21:06:37 +00:00

Author

Owner

@ZihanWang314 commented on GitHub (Sep 24, 2024):

Thank you for your suggestions! We will discuss and consider how to rephrase this part for better clearance.

@ZihanWang314 commented on GitHub (Sep 24, 2024): Thank you for your suggestions! We will discuss and consider how to rephrase this part for better clearance.

bsakaguchi referenced this issue

2026-10-26 21:06:50 +00:00

[PR #4] [MERGED] add (multi-gpu) training and eval #18