[GH-ISSUE #17] Question about AddAuxiliaryLoss? #14

New issue

Closed

opened 2026-10-26 21:01:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:01:39 +00:00

Owner

Originally created by @KaiWU5 on GitHub (Jan 26, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/17

In the code AddAuxiliaryLoss, the loss is not stored or used in the forward function, does that mean the grad is constantly to be 1? should it be grad_output * loss?

Thanks a lot if you can straighten this out for me.

class AddAuxiliaryLoss(torch.autograd.Function):
    """
    The trick function of adding auxiliary (aux) loss,
    which includes the gradient of the aux loss during backpropagation.
    """
    @staticmethod
    def forward(ctx, x, loss):
        assert loss.numel() == 1
        ctx.dtype = loss.dtype
        ctx.required_aux_loss = loss.requires_grad
        return x

    @staticmethod
    def backward(ctx, grad_output):
        grad_loss = None
        if ctx.required_aux_loss:
            grad_loss = torch.ones(1, dtype=ctx.dtype, device=grad_output.device)
        return grad_output, grad_loss

@DeepSeekDDM @zwd003 Thanks a lot for helping.

Originally created by @KaiWU5 on GitHub (Jan 26, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/17 In the code `AddAuxiliaryLoss`, the loss is not stored or used in the forward function, does that mean the grad is constantly to be 1? should it be grad_output * loss? Thanks a lot if you can straighten this out for me. ```python class AddAuxiliaryLoss(torch.autograd.Function): """ The trick function of adding auxiliary (aux) loss, which includes the gradient of the aux loss during backpropagation. """ @staticmethod def forward(ctx, x, loss): assert loss.numel() == 1 ctx.dtype = loss.dtype ctx.required_aux_loss = loss.requires_grad return x @staticmethod def backward(ctx, grad_output): grad_loss = None if ctx.required_aux_loss: grad_loss = torch.ones(1, dtype=ctx.dtype, device=grad_output.device) return grad_output, grad_loss ``` @DeepSeekDDM @zwd003 Thanks a lot for helping.

bsakaguchi closed this issue

2026-10-26 21:01:39 +00:00

bsakaguchi commented

2026-10-26 21:01:40 +00:00

Author

Owner

@Hunter-DDM commented on GitHub (Jan 29, 2024):

In the code AddAuxiliaryLoss, the loss is not stored or used in the forward function, does that mean the grad is constantly to be 1? should it be grad_output * loss?

Thanks a lot if you can straighten this out for me.

class AddAuxiliaryLoss(torch.autograd.Function):
    """
    The trick function of adding auxiliary (aux) loss,
    which includes the gradient of the aux loss during backpropagation.
    """
    @staticmethod
    def forward(ctx, x, loss):
        assert loss.numel() == 1
        ctx.dtype = loss.dtype
        ctx.required_aux_loss = loss.requires_grad
        return x

    @staticmethod
    def backward(ctx, grad_output):
        grad_loss = None
        if ctx.required_aux_loss:
            grad_loss = torch.ones(1, dtype=ctx.dtype, device=grad_output.device)
        return grad_output, grad_loss

@DeepSeekDDM @zwd003 Thanks a lot for helping.

For the aux loss, the gradient with respect to itself is always 1. This is equivalent to adding the aux loss to the final loss.

@Hunter-DDM commented on GitHub (Jan 29, 2024): > In the code `AddAuxiliaryLoss`, the loss is not stored or used in the forward function, does that mean the grad is constantly to be 1? should it be grad_output * loss? > > Thanks a lot if you can straighten this out for me. > > ```python > class AddAuxiliaryLoss(torch.autograd.Function): > """ > The trick function of adding auxiliary (aux) loss, > which includes the gradient of the aux loss during backpropagation. > """ > @staticmethod > def forward(ctx, x, loss): > assert loss.numel() == 1 > ctx.dtype = loss.dtype > ctx.required_aux_loss = loss.requires_grad > return x > > @staticmethod > def backward(ctx, grad_output): > grad_loss = None > if ctx.required_aux_loss: > grad_loss = torch.ones(1, dtype=ctx.dtype, device=grad_output.device) > return grad_output, grad_loss > ``` > > @DeepSeekDDM @zwd003 Thanks a lot for helping. For the aux loss, the gradient with respect to itself is always 1. This is equivalent to adding the aux loss to the final loss.

No labels

pull-request

WIP

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepSeek-MoE#14

No description provided.

Rows
Columns