[GH-ISSUE #205] 反馈一个BUG

bsakaguchi commented

2026-10-31 21:12:04 +00:00

Owner

Originally created by @ninding on GitHub (Jan 4, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/205

clip.py中第133行应该修改为`model = torch.load(model_path, map_location="cpu")`

Originally created by @ninding on GitHub (Jan 4, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/205 <img width="716" alt="image" src="https://user-images.githubusercontent.com/91425172/148052527-e9fe6da2-b681-42b4-b48d-6f5a3974bd59.png"> clip.py中第133行应该修改为`model = torch.load(model_path, map_location="cpu")`

bsakaguchi closed this issue

2026-10-31 21:12:04 +00:00

bsakaguchi commented

2026-10-31 21:12:04 +00:00

Author

Owner

@haofanwang commented on GitHub (Jan 4, 2022):

当前版本已经是正确的了。
github.com/openai/CLIP@573315e83f/clip/clip.py (L132)

@haofanwang commented on GitHub (Jan 4, 2022): 当前版本已经是正确的了。 https://github.com/openai/CLIP/blob/573315e83f07b53a61ff5098757e8fc885f1703e/clip/clip.py#L132

bsakaguchi commented

2026-10-31 21:12:05 +00:00

Author

Owner

@ninding commented on GitHub (Jan 5, 2022):

如果我使用
state_dict = torch.load(model_path, map_location="cpu")
报错信息
Traceback (most recent call last):
File "/Users/zhangjiuding/Desktop/CLIP/test1.py", line 8, in
model, preprocess = clip.load('/Users/zhangjiuding/Desktop/CLIP/clip/model/model1.pkl', device)
File "/Users/zhangjiuding/Desktop/CLIP/clip/clip.py", line 138, in load
model = build_model(state_dict or model.state_dict()).to(device)
File "/Users/zhangjiuding/Desktop/CLIP/clip/model.py", line 400, in build_model
vit = "visual.proj" in state_dict
TypeError: argument of type 'CLIP' is not iterable

如果我使用
model = torch.load(model_path, map_location="cpu")
state_dict = None
不报错

@ninding commented on GitHub (Jan 5, 2022): 如果我使用 state_dict = torch.load(model_path, map_location="cpu") 报错信息 Traceback (most recent call last): File "/Users/zhangjiuding/Desktop/CLIP/test1.py", line 8, in <module> model, preprocess = clip.load('/Users/zhangjiuding/Desktop/CLIP/clip/model/model1.pkl', device) File "/Users/zhangjiuding/Desktop/CLIP/clip/clip.py", line 138, in load model = build_model(state_dict or model.state_dict()).to(device) File "/Users/zhangjiuding/Desktop/CLIP/clip/model.py", line 400, in build_model vit = "visual.proj" in state_dict TypeError: argument of type 'CLIP' is not iterable 如果我使用 model = torch.load(model_path, map_location="cpu") state_dict = None 不报错

bsakaguchi commented

2026-10-31 21:12:05 +00:00

Author

Owner

@haofanwang commented on GitHub (Jan 5, 2022):

这应该是你模型保存的问题，你是怎么训练然后保存的？

@haofanwang commented on GitHub (Jan 5, 2022): 这应该是你模型保存的问题，你是怎么训练然后保存的？

bsakaguchi commented

2026-10-31 21:12:05 +00:00

Author

Owner

@ninding commented on GitHub (Jan 5, 2022):

from torch.utils.data import Dataset, DataLoader
import torch
import clip
from torch import nn, optim
import pandas as pd
from PIL import Image
import os
import time
from torchvision.datasets import CIFAR10

device = 'cpu'

class image_caption_dataset(Dataset):
    def __init__(self, df, preprocess):
        self.images = df["image"]
        self.caption = df["caption"]
        self.preprocess = preprocess

    def __len__(self):
        return len(self.caption)

    def __getitem__(self, idx):
        images = self.preprocess(self.images[idx])
        caption = self.caption[idx]
        return images, caption



def load_data(cifar10_train, batch_size, preprocess):
    df = {'image': [], 'caption':[]}
    for i in range(int(len(cifar10_train)/10)):
        image, class_id = cifar10_train[i]
        df['image'].append(image)
        df['caption'].append(cifar10_train.classes[class_id])
        print(image)

    dataset = image_caption_dataset(df, preprocess)
    train_dataloader = DataLoader(dataset, batch_size=batch_size)
    return train_dataloader


def convert_models_to_fp32(model):
    for p in model.parameters():
        p.data = p.data.float()
        p.grad.data = p.grad.data.float()

epoch = 10
batch_size = 10
learning_rate = 5e-5

# print(Image.open("./CLIP.PNG"))<PIL.PngImagePlugin.PngImageFile image mode=RGB size=248x163 at 0x7F9F88108A00>
# 加载模型
model, preprocess = clip.load('ViT-B/32', device=device, jit=False)  # 训练时 jit必须设置为false

# 下载数据集
cifar10_test = CIFAR10(root=os.path.expanduser("~/.cache"), download=True, train=False)
cifar10_train = CIFAR10(root=os.path.expanduser("~/.cache"), download=True, train=True)

#加载数据集
train_dataloader = load_data(cifar10_train, batch_size, preprocess)

#设置参数
loss_img = nn.CrossEntropyLoss().to(device)
loss_txt = nn.CrossEntropyLoss().to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate, betas=(0.9, 0.98), eps=1e-6, weight_decay=0.2)

for i in range(epoch):
    j = 1
    print(len(train_dataloader))
    for batch in train_dataloader:
        print('start: i='+str(i+1)+' j='+str(j)+' 'str(time.localtime()))
        list_image, list_txt = batch  # list_images is list of image in numpy array(np.uint8), or list of PIL images
        texts = clip.tokenize(list_txt).to(device)
        images = list_image.to(device)
        
        logits_per_image, logits_per_text = model(images, texts)
        ground_truth = torch.arange(batch_size).long().to(device)
        
        #反向传播
        total_loss = (loss_img(logits_per_image, ground_truth) + loss_txt(logits_per_text, ground_truth)) / 2
        optimizer.zero_grad()
        total_loss.backward()
        optimizer.step()
        j += 1
    print('[%d] loss: %.3f' %(i + 1, total_loss))
torch.save(model, './model/model1.pkl')

@ninding commented on GitHub (Jan 5, 2022): ```python from torch.utils.data import Dataset, DataLoader import torch import clip from torch import nn, optim import pandas as pd from PIL import Image import os import time from torchvision.datasets import CIFAR10 device = 'cpu' class image_caption_dataset(Dataset): def __init__(self, df, preprocess): self.images = df["image"] self.caption = df["caption"] self.preprocess = preprocess def __len__(self): return len(self.caption) def __getitem__(self, idx): images = self.preprocess(self.images[idx]) caption = self.caption[idx] return images, caption def load_data(cifar10_train, batch_size, preprocess): df = {'image': [], 'caption':[]} for i in range(int(len(cifar10_train)/10)): image, class_id = cifar10_train[i] df['image'].append(image) df['caption'].append(cifar10_train.classes[class_id]) print(image) dataset = image_caption_dataset(df, preprocess) train_dataloader = DataLoader(dataset, batch_size=batch_size) return train_dataloader def convert_models_to_fp32(model): for p in model.parameters(): p.data = p.data.float() p.grad.data = p.grad.data.float() epoch = 10 batch_size = 10 learning_rate = 5e-5 # print(Image.open("./CLIP.PNG"))<PIL.PngImagePlugin.PngImageFile image mode=RGB size=248x163 at 0x7F9F88108A00> # 加载模型 model, preprocess = clip.load('ViT-B/32', device=device, jit=False) # 训练时 jit必须设置为false # 下载数据集 cifar10_test = CIFAR10(root=os.path.expanduser("~/.cache"), download=True, train=False) cifar10_train = CIFAR10(root=os.path.expanduser("~/.cache"), download=True, train=True) #加载数据集 train_dataloader = load_data(cifar10_train, batch_size, preprocess) #设置参数 loss_img = nn.CrossEntropyLoss().to(device) loss_txt = nn.CrossEntropyLoss().to(device) optimizer = optim.Adam(model.parameters(), lr=learning_rate, betas=(0.9, 0.98), eps=1e-6, weight_decay=0.2) for i in range(epoch): j = 1 print(len(train_dataloader)) for batch in train_dataloader: print('start: i='+str(i+1)+' j='+str(j)+' 'str(time.localtime())) list_image, list_txt = batch # list_images is list of image in numpy array(np.uint8), or list of PIL images texts = clip.tokenize(list_txt).to(device) images = list_image.to(device) logits_per_image, logits_per_text = model(images, texts) ground_truth = torch.arange(batch_size).long().to(device) #反向传播 total_loss = (loss_img(logits_per_image, ground_truth) + loss_txt(logits_per_text, ground_truth)) / 2 optimizer.zero_grad() total_loss.backward() optimizer.step() j += 1 print('[%d] loss: %.3f' %(i + 1, total_loss)) torch.save(model, './model/model1.pkl') ```

bsakaguchi commented

2026-10-31 21:12:05 +00:00

Author

Owner

@haofanwang commented on GitHub (Jan 5, 2022):

加载的时候，不要使用clip.load()，直接使用torch.load()。

@haofanwang commented on GitHub (Jan 5, 2022): 加载的时候，不要使用`clip.load()`，直接使用`torch.load()`。

Rows
Columns

[GH-ISSUE #205] 反馈一个BUG #178