[GH-ISSUE #420] [疑问] 关于DeepseekV3Model类默认关闭gradient checkpointing.的原因 #333

New issue

Closed

opened 2026-10-26 21:19:34 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:19:34 +00:00

Owner

Originally created by @ShaohonChen on GitHub (Feb 16, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/420

感谢DeepSeek团队优秀的工作！

我在阅读HuggingFace上DeepSeek-V3模型的代码时（R1继承了V3的代码，因此有相同的问题），发现在modeling_deepseek.py中DeepseekV3PreTrainedModel类中声明了支持supports_gradient_checkpointing = True，但是在DeepseekV3Model类中似乎默认关闭了gradient checkpointing. （1372行）代码的切片如下：

class DeepseekV3Model(DeepseekV3PreTrainedModel):
    """
    Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`DeepseekV3DecoderLayer`]

    Args:
        config: DeepseekV3Config
    """

    def __init__(self, config: DeepseekV3Config):
        super().__init__(config)
        self.padding_idx = config.pad_token_id
        self.vocab_size = config.vocab_size

        self.embed_tokens = nn.Embedding(
            config.vocab_size, config.hidden_size, self.padding_idx
        )
        self.layers = nn.ModuleList(
            [
                DeepseekV3DecoderLayer(config, layer_idx)
                for layer_idx in range(config.num_hidden_layers)
            ]
        )
        self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
        self.norm = DeepseekV3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)

        self.gradient_checkpointing = False       # 似乎是默认关闭了gradient_checkpointing
        # Initialize weights and apply final processing
        self.post_init()

我想请教下开发者为什么要这么设置呢？期待开发者和社区伙伴的答疑。

Originally created by @ShaohonChen on GitHub (Feb 16, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/420 感谢DeepSeek团队优秀的工作！我在阅读HuggingFace上DeepSeek-V3模型的代码时（R1继承了V3的代码，因此有相同的问题），发现在`modeling_deepseek.py`中DeepseekV3PreTrainedModel类中声明了支持`supports_gradient_checkpointing = True`，但是在DeepseekV3Model类中似乎默认关闭了gradient checkpointing. （1372行）代码的切片如下： ```python class DeepseekV3Model(DeepseekV3PreTrainedModel): """ Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`DeepseekV3DecoderLayer`] Args: config: DeepseekV3Config """ def __init__(self, config: DeepseekV3Config): super().__init__(config) self.padding_idx = config.pad_token_id self.vocab_size = config.vocab_size self.embed_tokens = nn.Embedding( config.vocab_size, config.hidden_size, self.padding_idx ) self.layers = nn.ModuleList( [ DeepseekV3DecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers) ] ) self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2" self.norm = DeepseekV3RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.gradient_checkpointing = False # 似乎是默认关闭了gradient_checkpointing # Initialize weights and apply final processing self.post_init() ``` 我想请教下开发者为什么要这么设置呢？期待开发者和社区伙伴的答疑。

bsakaguchi

2026-10-26 21:19:34 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:19:35 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Mar 21, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Mar 21, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:19:35 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 25, 2025):

false

@github-actions[bot] commented on GitHub (Apr 25, 2025): false