[PR #234] [CLOSED] fix: remove inconsistent +1e-5 offset in entropy loss log_softmax #233

New issue

Closed

opened 2026-10-26 21:14:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:14:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/Janus/pull/234
Author: @Mr-Neutr0n
Created: 2/11/2026
Status: ❌ Closed

Base: main ← Head: fix/entropy-loss-consistent-softmax

📝 Commits (1)

277b071 fix: remove inconsistent offset in log_softmax for entropy loss

📊 Changes

1 file changed (+1 additions, -1 deletions)

View changed files

📝 janus/models/vq_model.py (+1 -1)

📄 Description

Summary

In compute_entropy_loss() in janus/models/vq_model.py, probs and log_probs should represent the same probability distribution, but they diverge because of an unnecessary + 1e-5 offset applied before log_softmax:

probs = F.softmax(flat_affinity, dim=-1)
log_probs = F.log_softmax(flat_affinity + 1e-5, dim=-1)  # +1e-5 changes the input distribution

The + 1e-5 offset shifts the logits before computing log_softmax, which means log_probs != log(probs). This causes the sample entropy term (-sum(probs * log_probs)) to not be a true entropy computation, introducing a small but systematic bias in the entropy loss.

F.log_softmax is already numerically stable (it uses the log-sum-exp trick internally), so the + 1e-5 offset is not needed for numerical safety and only serves to make probs and log_probs inconsistent.

Fix

Remove the + 1e-5 so both probs and log_probs are computed from the same input:

log_probs = F.log_softmax(flat_affinity, dim=-1)

Test plan

Verify that compute_entropy_loss produces correct entropy values by checking log_probs == log(probs) for sample inputs
Confirm no numerical instability is introduced (expected: none, since F.log_softmax is inherently stable)
Run existing VQ model training/evaluation to validate no regression

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/Janus/pull/234 **Author:** [@Mr-Neutr0n](https://github.com/Mr-Neutr0n) **Created:** 2/11/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/entropy-loss-consistent-softmax` --- ### 📝 Commits (1) - [`277b071`](https://github.com/deepseek-ai/Janus/commit/277b071009f8402492547b571228365903141878) fix: remove inconsistent offset in log_softmax for entropy loss ### 📊 Changes **1 file changed** (+1 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `janus/models/vq_model.py` (+1 -1) </details> ### 📄 Description ## Summary In `compute_entropy_loss()` in `janus/models/vq_model.py`, `probs` and `log_probs` should represent the same probability distribution, but they diverge because of an unnecessary `+ 1e-5` offset applied before `log_softmax`: ```python probs = F.softmax(flat_affinity, dim=-1) log_probs = F.log_softmax(flat_affinity + 1e-5, dim=-1) # +1e-5 changes the input distribution ``` The `+ 1e-5` offset shifts the logits before computing `log_softmax`, which means `log_probs != log(probs)`. This causes the sample entropy term (`-sum(probs * log_probs)`) to not be a true entropy computation, introducing a small but systematic bias in the entropy loss. `F.log_softmax` is already numerically stable (it uses the log-sum-exp trick internally), so the `+ 1e-5` offset is not needed for numerical safety and only serves to make `probs` and `log_probs` inconsistent. ## Fix Remove the `+ 1e-5` so both `probs` and `log_probs` are computed from the same input: ```python log_probs = F.log_softmax(flat_affinity, dim=-1) ``` ## Test plan - [ ] Verify that `compute_entropy_loss` produces correct entropy values by checking `log_probs == log(probs)` for sample inputs - [ ] Confirm no numerical instability is introduced (expected: none, since `F.log_softmax` is inherently stable) - [ ] Run existing VQ model training/evaluation to validate no regression --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:14:23 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/Janus#233

No description provided.

Rows
Columns