[GH-ISSUE #82] Attention map to extract objects

bsakaguchi commented

2026-10-31 21:10:30 +00:00

Owner

Originally created by @rodrigoheck on GitHub (Apr 7, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/82

I have no idea if something like this is possible, but if I have, for example, an image of a supermarket shelve and the text "juice", could I in theory use the attention map to extract all the regions in which the model found something similar to a juice?

Originally created by @rodrigoheck on GitHub (Apr 7, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/82 I have no idea if something like this is possible, but if I have, for example, an image of a supermarket shelve and the text "juice", could I in theory use the attention map to extract all the regions in which the model found something similar to a juice?

bsakaguchi closed this issue

2026-10-31 21:10:30 +00:00

bsakaguchi commented

2026-10-31 21:10:31 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 8, 2021):

It's not as straightforward as CLIP doesn't have cross-modal attentions and the only cross-modal interaction is the final dot product, but you could start with cropping the images into square-shaped patches and finding the cosine similarity between the image features of those patches and the query text features.

@jongwook commented on GitHub (Apr 8, 2021): It's not as straightforward as CLIP doesn't have cross-modal attentions and the only cross-modal interaction is the final dot product, but you could start with cropping the images into square-shaped patches and finding the cosine similarity between the image features of those patches and the query text features.

bsakaguchi commented

2026-10-31 21:10:31 +00:00

Author

Owner

@rodrigoheck commented on GitHub (Apr 8, 2021):

Thanks @jongwook, I will try this approach.

@rodrigoheck commented on GitHub (Apr 8, 2021): Thanks @jongwook, I will try this approach.

bsakaguchi commented

2026-10-31 21:10:31 +00:00

Author

Owner

@ModMorph commented on GitHub (Apr 8, 2021):

FYI: @rodrigoheck you might be interested in the following projects: https://ganpaint.io/miniclip/ and https://github.com/hila-chefer/Transformer-MM-Explainability - these have saliency and/or gradcam maps for RN50 and ViTB CLIP respectively. (Although I think the Transformer method is new so not sure how confident to be in it.)

@ModMorph commented on GitHub (Apr 8, 2021): FYI: @rodrigoheck you might be interested in the following projects: https://ganpaint.io/miniclip/ and https://github.com/hila-chefer/Transformer-MM-Explainability - these have saliency and/or gradcam maps for RN50 and ViTB CLIP respectively. (Although I think the Transformer method is new so not sure how confident to be in it.)

bsakaguchi commented

2026-10-31 21:10:31 +00:00

Author

Owner

@g-luo commented on GitHub (Apr 22, 2021):

@ModMorph Do you have insight on which layers are best to use as the saliency layer for Grad CAM (for the visual transformer ViT models)? For the RN50 the miniclip demo uses the relu from the last visual layer, and I'm wondering what's the best for the visual transformer.

@g-luo commented on GitHub (Apr 22, 2021): @ModMorph Do you have insight on which layers are best to use as the saliency layer for Grad CAM (for the visual transformer ViT models)? For the RN50 the miniclip demo uses the relu from the last visual layer, and I'm wondering what's the best for the visual transformer.

bsakaguchi commented

2026-10-31 21:10:31 +00:00

Author

Owner

@ModMorph commented on GitHub (Apr 22, 2021):

Hi @g-luo, good timing. I may be circling back to this...but I haven't had time to
look at how Transformer-MM-Explainability's technique works yet, just that in trying
their demos, it seemed to look reasonably well.

One thing I noticed you might be interested in, is that there's seems to be some
more attention head related information you can derive by taking the full
x[:,:,:] instead of the x[:,0,:] in the last lines of the VisualTransformer forward method. I thought maybe it was de facto a vector describing
each of the 49 32x32x3 pixel patches w.r.t. to the embedding vector..but
I'm new to transformers. When I subtracted the 49 vectors from the embedding
vector currently being used, I got what seemed like a coherent heatmap description
of some basic scenes I tried. HTH. (Not exactly same as the level of detail you want,
I'm still experimenting whether I can make any use out of it.)

@ModMorph commented on GitHub (Apr 22, 2021): Hi @g-luo, good timing. I may be circling back to this...but I haven't had time to look at how Transformer-MM-Explainability's technique works yet, just that in trying their demos, it seemed to look reasonably well. One thing I noticed you might be interested in, is that there's seems to be some more attention head related information you can derive by taking the full x[:,:,:] instead of the x[:,0,:] in the last lines of the VisualTransformer forward method. I thought maybe it was de facto a vector describing each of the 49 32x32x3 pixel patches w.r.t. to the embedding vector..but I'm new to transformers. When I subtracted the 49 vectors from the embedding vector currently being used, I got what seemed like a coherent heatmap description of some basic scenes I tried. HTH. (Not exactly same as the level of detail you want, I'm still experimenting whether I can make any use out of it.) ![image](https://user-images.githubusercontent.com/7545237/115787284-2c0aa000-a37f-11eb-9005-2c396ed33705.png)

bsakaguchi commented

2026-10-31 21:10:31 +00:00

Author

Owner

@jihwanp commented on GitHub (Apr 11, 2022):

Hi @g-luo, good timing. I may be circling back to this...but I haven't had time to look at how Transformer-MM-Explainability's technique works yet, just that in trying their demos, it seemed to look reasonably well.

One thing I noticed you might be interested in, is that there's seems to be some more attention head related information you can derive by taking the full x[:,:,:] instead of the x[:,0,:] in the last lines of the VisualTransformer forward method. I thought maybe it was de facto a vector describing each of the 49 32x32x3 pixel patches w.r.t. to the embedding vector..but I'm new to transformers. When I subtracted the 49 vectors from the embedding vector currently being used, I got what seemed like a coherent heatmap description of some basic scenes I tried. HTH. (Not exactly same as the level of detail you want, I'm still experimenting whether I can make any use out of it.)

Did that similarity map just measure the similarity between local feature and text(apple)? I tried similar things, but the local similarity of the text-related parts was rather low.

@jihwanp commented on GitHub (Apr 11, 2022): > Hi @g-luo, good timing. I may be circling back to this...but I haven't had time to look at how Transformer-MM-Explainability's technique works yet, just that in trying their demos, it seemed to look reasonably well. > > One thing I noticed you might be interested in, is that there's seems to be some more attention head related information you can derive by taking the full x[:,:,:] instead of the x[:,0,:] in the last lines of the VisualTransformer forward method. I thought maybe it was de facto a vector describing each of the 49 32x32x3 pixel patches w.r.t. to the embedding vector..but I'm new to transformers. When I subtracted the 49 vectors from the embedding vector currently being used, I got what seemed like a coherent heatmap description of some basic scenes I tried. HTH. (Not exactly same as the level of detail you want, I'm still experimenting whether I can make any use out of it.) > > ![image](https://user-images.githubusercontent.com/7545237/115787284-2c0aa000-a37f-11eb-9005-2c396ed33705.png) Did that similarity map just measure the similarity between local feature and text(apple)? I tried similar things, but the local similarity of the text-related parts was rather low.

bsakaguchi commented

2026-10-31 21:10:31 +00:00

Author

Owner

@gordinmitya commented on GitHub (Feb 8, 2023):

now there's such a thing https://huggingface.co/blog/clipseg-zero-shot

@gordinmitya commented on GitHub (Feb 8, 2023): now there's such a thing https://huggingface.co/blog/clipseg-zero-shot

bsakaguchi commented

2026-10-31 21:10:32 +00:00

Author

Owner

@chinmay5 commented on GitHub (Mar 30, 2023):

Here is a very simple code snippet in case someone finds it useful.

def generate_crop_dict(img, preprocess):
    # Convert image into 56 by 56 crops
    offset = 56
    image_crop_dict = {}
    for h in range(0, 224, offset):
        for w in range(0, 224, offset):
            area = (w, h, w + offset, h + offset)
            crop = img.crop(area)
            crop = crop.resize((224, 224), Image.ANTIALIAS)
            # Include the batch dimension
            image_crop_dict[area] = torch.tensor(preprocess(crop)).unsqueeze(0)
    return image_crop_dict


def highlight_area(img, region, factor, outline_color=None, outline_width=1):
    """ Highlight specified rectangular region of image by `factor` with an
        optional colored  boarder drawn around its edges and return the result.
    """
    img = img.copy()  # Avoid changing original image.
    img_crop = img.crop(region)

    brightner = ImageEnhance.Brightness(img_crop)
    img_crop = brightner.enhance(factor)
    img.paste(img_crop, region)
    # Optionally draw a colored outline around the edge of the rectangular region.
    if outline_color:
        draw = ImageDraw.Draw(img)  # Create a drawing context.
        left, upper, right, lower = region  # Get bounds.
        coords = [(left, upper), (right, upper), (right, lower), (left, lower),
                  (left, upper)]
        draw.line(coords, fill=outline_color, width=outline_width)
    return img


def viz_pseudo_attn_maps(model, img, text, preprocess):
    image_dict = generate_crop_dict(img, preprocess)
    sim_dict = {}
    for crop_loc, image in image_dict.items():
        with torch.no_grad():
            image_features = model.encode_image(image).float()
            text_features = model.encode_text(text).float()
            image_features /= image_features.norm(dim=-1, keepdim=True)
            text_features /= text_features.norm(dim=-1, keepdim=True)
            similarity = text_features.cpu().numpy() @ image_features.cpu().numpy().T
            sim_dict[crop_loc] = similarity.item()
    bbox = max(sim_dict, key=sim_dict.get)
    print(f"Max patch is {max(sim_dict, key=sim_dict.get)}")
    # We pass the original image and the rectangle.
    red = ImageColor.getrgb('red')
    img2 = highlight_area(img, bbox, 2.5, outline_color=red, outline_width=2)
    plt.imshow(img2)
    plt.show()

@chinmay5 commented on GitHub (Mar 30, 2023): Here is a very simple code snippet in case someone finds it useful. ``` def generate_crop_dict(img, preprocess): # Convert image into 56 by 56 crops offset = 56 image_crop_dict = {} for h in range(0, 224, offset): for w in range(0, 224, offset): area = (w, h, w + offset, h + offset) crop = img.crop(area) crop = crop.resize((224, 224), Image.ANTIALIAS) # Include the batch dimension image_crop_dict[area] = torch.tensor(preprocess(crop)).unsqueeze(0) return image_crop_dict def highlight_area(img, region, factor, outline_color=None, outline_width=1): """ Highlight specified rectangular region of image by `factor` with an optional colored boarder drawn around its edges and return the result. """ img = img.copy() # Avoid changing original image. img_crop = img.crop(region) brightner = ImageEnhance.Brightness(img_crop) img_crop = brightner.enhance(factor) img.paste(img_crop, region) # Optionally draw a colored outline around the edge of the rectangular region. if outline_color: draw = ImageDraw.Draw(img) # Create a drawing context. left, upper, right, lower = region # Get bounds. coords = [(left, upper), (right, upper), (right, lower), (left, lower), (left, upper)] draw.line(coords, fill=outline_color, width=outline_width) return img def viz_pseudo_attn_maps(model, img, text, preprocess): image_dict = generate_crop_dict(img, preprocess) sim_dict = {} for crop_loc, image in image_dict.items(): with torch.no_grad(): image_features = model.encode_image(image).float() text_features = model.encode_text(text).float() image_features /= image_features.norm(dim=-1, keepdim=True) text_features /= text_features.norm(dim=-1, keepdim=True) similarity = text_features.cpu().numpy() @ image_features.cpu().numpy().T sim_dict[crop_loc] = similarity.item() bbox = max(sim_dict, key=sim_dict.get) print(f"Max patch is {max(sim_dict, key=sim_dict.get)}") # We pass the original image and the rectangle. red = ImageColor.getrgb('red') img2 = highlight_area(img, bbox, 2.5, outline_color=red, outline_width=2) plt.imshow(img2) plt.show() ```

Rows
Columns

[GH-ISSUE #82] Attention map to extract objects #74