deepseek-ai/DeepSeek-V2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-V2.git synced 2026-11-05 00:47:23 +00:00
Code Issues 90 Projects Packages Wiki Activity

[GH-ISSUE #102] Equation 7 in DeepSeek-V2 Technical Report . #92

New issue
Open
opened 2026-10-26 21:07:47 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 21:07:47 +00:00
Owner
Copy link

Originally created by @whatdhack on GitHub (Jan 3, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V2/issues/102

I am trying to understand equation 7 in the DeepSeek-V2 tech report. . Here are the confusions I am having.

  1. qti, kti, and vti are row vectors ? shapes are ( 1, dh) ?
  2. qti^T.Kji shape is (dh,1) . (1, dh) = (dh,dh) ?
  3. Softmax_j implies over the -1 dim ?
  4. qti^T.Kji . vji , would it compute as (dh, dh) . ( 1, dh) ?
  5. What is the summation over j for ?
Originally created by @whatdhack on GitHub (Jan 3, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V2/issues/102 I am trying to understand equation 7 in the [DeepSeek-V2 tech report](https://arxiv.org/html/2405.04434v5). . Here are the confusions I am having. 1. qti, kti, and vti are row vectors ? shapes are ( 1, dh) ? 2. qti^T.Kji shape is (dh,1) . (1, dh) = (dh,dh) ? 3. Softmax_j implies over the -1 dim ? 4. qti^T.Kji . vji , would it compute as (dh, dh) . ( 1, dh) ? 5. What is the summation over j for ?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-V2#92
No description provided.