[GH-ISSUE #298] confusion in ParallelEmebdding layer in model.py #117

New issue

Closed

opened 2026-10-26 21:14:36 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:14:36 +00:00

Owner

Originally created by @baihuaxie on GitHub (Jan 17, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/298

hi there,

in the ParallelEmebdding layer as defined in inference/model.py, in the following code:

if world_size > 1:
            mask = (x < self.vocab_start_idx) | (x >= self.vocab_end_idx)
            x = x - self.vocab_start_idx
            x[mask] = 0

if I understand it correctly, in each rank the local emebdding matrix handles a portion of the token ids specified by the range [self.vocab_start_idx, self.vocab_end_idx), excluding rhs.

the line x = x - self.vocab_start_idx shifts the token ids to be aligned with the starting idx assigned to the current rank, so that it can be processed by the local embedding matrix (which starts from token id = 0). so for instance if the local rank handles token ids [100, 199], the token id 100 in the original input tensor x is now shifted to become 0.

the next line masks out the out-of-range token ids to also be 0.

what's confusing to me is that now how would I distinguish a token id = 0 in the input tensor x to be from 1) an originall out-of-range token id (for the current rank) or 2) an original valid token id which equals self.vocab_start_idx? it seems to me that these two lines of code have now shrinked the valid range of token ids handled by the current rank to become (self.vocab_start_idx, self.vocab_end_idx), both excluding.

I haven't run the code yet so this confusion might be due to some oversight on my part. in the given code world_size=1 is fixed so this code snippet never gets executed anyways. I'd appreciate if there could be some clarification or guidance on tests.

Originally created by @baihuaxie on GitHub (Jan 17, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/298 hi there, in the ParallelEmebdding layer as defined in inference/model.py, in the following code: ``` if world_size > 1: mask = (x < self.vocab_start_idx) | (x >= self.vocab_end_idx) x = x - self.vocab_start_idx x[mask] = 0 ``` if I understand it correctly, in each rank the local emebdding matrix handles a portion of the token ids specified by the range `[self.vocab_start_idx, self.vocab_end_idx)`, excluding rhs. the line `x = x - self.vocab_start_idx` shifts the token ids to be aligned with the starting idx assigned to the current rank, so that it can be processed by the local embedding matrix (which starts from token id = 0). so for instance if the local rank handles token ids [100, 199], the token id 100 in the original input tensor x is now shifted to become 0. the next line masks out the out-of-range token ids to also be 0. what's confusing to me is that now how would I distinguish a token id = 0 in the input tensor x to be from 1) an originall out-of-range token id (for the current rank) or 2) an original valid token id which equals self.vocab_start_idx? it seems to me that these two lines of code have now shrinked the valid range of token ids handled by the current rank to become `(self.vocab_start_idx, self.vocab_end_idx)`, both excluding. I haven't run the code yet so this confusion might be due to some oversight on my part. in the given code world_size=1 is fixed so this code snippet never gets executed anyways. I'd appreciate if there could be some clarification or guidance on tests.

bsakaguchi

2026-10-26 21:14:36 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:14:38 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Feb 18, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Feb 18, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:14:38 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Mar 5, 2025):

false

@github-actions[bot] commented on GitHub (Mar 5, 2025): false