[GH-ISSUE #29] Clarification of training details #26

New issue

Closed

opened 2026-10-31 21:10:01 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 21:10:01 +00:00

Owner

Originally created by @jwyang on GitHub (Jan 30, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/29

Dear CLIP Authors,

Recently, I am reading the training details in the paper:

To save additional memory, gradient checkpointing (Griewank & Walther, 2000; Chen et al., 2016), half-precision Adam statistics (Dhariwal et al., 2020), and half-precision stochastically rounded text encoder weights were used. The calculation of embedding
similarities was also sharded with individual GPUs computing only the subset of the pairwise similarities necessary for their local batch of embeddings.

I have two questions:

what does "half-precision stochastically rounded text encoder weights" mean? Is there any references?
I do not quite understanding "The calculation of embedding similarities was also sharded with individual GPUs". By saying "sharded" instead of "shared" Does this mean for each sample, the similarities to negative is only from the same gpu? Or you did a gather of features from other gpus and then compute the similarities between one sample to all.

thanks,
Jianwei

Originally created by @jwyang on GitHub (Jan 30, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/29 Dear CLIP Authors, Recently, I am reading the training details in the paper: _To save additional memory, gradient checkpointing (Griewank & Walther, 2000; Chen et al., 2016), half-precision Adam statistics (Dhariwal et al., 2020), and half-precision stochastically rounded text encoder weights were used. The calculation of embedding similarities was also sharded with individual GPUs computing only the subset of the pairwise similarities necessary for their local batch of embeddings._ I have two questions: 1. what does "half-precision stochastically rounded text encoder weights" mean? Is there any references? 2. I do not quite understanding "The calculation of embedding similarities was also sharded with individual GPUs". By saying "**sharded**" instead of "**shared**" Does this mean for each sample, the similarities to negative is only from the same gpu? Or you did a gather of features from other gpus and then compute the similarities between one sample to all. thanks, Jianwei

bsakaguchi closed this issue

2026-10-31 21:10:02 +00:00

bsakaguchi commented

2026-10-31 21:10:02 +00:00

Author

Owner

@jongwook commented on GitHub (Jan 30, 2021):

When the fp32 encoder weights are cast to fp16, rather than deterministically rounding to the nearest fp16 point, stochastic rounding casts the value pseudorandomly to either ⌈x⌉ or ⌊x⌋ according to some probability. This reference mentions two modes for doing this, in (1.1) and (1.2).
Say there are N GPUs and each GPU gets M input pairs. When calculating similarities (and thus the logit scores across text inputs for each image) a GPU only needs to hold M image features, which needs to be matmul'ed with NM text features, resulting in an M×NM matrix. This computation is distributed (i.e. sharded) across N GPUs, and overall we have calculated NM×NM similarities across the GPUs. The loss we use is symmetric and the same happens w.r.t. text inputs.

@jongwook commented on GitHub (Jan 30, 2021): 1. When the fp32 encoder weights are cast to fp16, rather than deterministically rounding to the nearest fp16 point, stochastic rounding casts the value pseudorandomly to either ⌈x⌉ or ⌊x⌋ according to some probability. [This reference](https://hal.archives-ouvertes.fr/hal-02556997v2/document) mentions two modes for doing this, in (1.1) and (1.2). 2. Say there are N GPUs and each GPU gets M input pairs. When calculating similarities (and thus the logit scores across text inputs for each image) a GPU only needs to hold M image features, which needs to be matmul'ed with NM text features, resulting in an M×NM matrix. This computation is distributed (i.e. sharded) across N GPUs, and overall we have calculated NM×NM similarities across the GPUs. The loss we use is symmetric and the same happens w.r.t. text inputs.

bsakaguchi commented

2026-10-31 21:10:02 +00:00

Author

Owner

@jwyang commented on GitHub (Jan 30, 2021):

Hi, Jongwook,

thanks a lot for your quick reply! The explanations are very helpful to understand the training procedure!

@jwyang commented on GitHub (Jan 30, 2021): Hi, Jongwook, thanks a lot for your quick reply! The explanations are very helpful to understand the training procedure!

bsakaguchi commented

2026-10-31 21:10:02 +00:00

Author

Owner

@KeremTurgutlu commented on GitHub (Feb 20, 2021):

2. Say there are N GPUs and each GPU gets M input pairs. When calculating similarities (and thus the logit scores across text inputs for each image) a GPU only needs to hold M image features, which needs to be matmul'ed with NM text features, resulting in an M×NM matrix. This computation is distributed (i.e. sharded) across N GPUs, and overall we have calculated NM×NM similarities across the GPUs. The loss we use is symmetric and the same happens w.r.t. text inputs.

Thanks for the explanation, is there a sample code in pytorch on how to do that? Is it similar to this or is there a more efficient way that was used during CLIP training? Thanks!

@KeremTurgutlu commented on GitHub (Feb 20, 2021): > 2\. Say there are N GPUs and each GPU gets M input pairs. When calculating similarities (and thus the logit scores across text inputs for each image) a GPU only needs to hold M image features, which needs to be matmul'ed with NM text features, resulting in an M×NM matrix. This computation is distributed (i.e. sharded) across N GPUs, and overall we have calculated NM×NM similarities across the GPUs. The loss we use is symmetric and the same happens w.r.t. text inputs. Thanks for the explanation, is there a sample code in pytorch on how to do that? Is it similar to [this](https://discuss.pytorch.org/t/matmul-on-multiple-gpus/33122/2) or is there a more efficient way that was used during CLIP training? Thanks!

bsakaguchi referenced this issue

2026-10-31 21:15:06 +00:00

[PR #26] [MERGED] Feature/make installable #449