[GH-ISSUE #56] tokenizer.encode vs tokenizer.encoder #49

New issue

Closed

opened 2026-10-31 21:10:15 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:10:15 +00:00

Owner

Originally created by @normster on GitHub (Mar 10, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/56

Hi,

I noticed that in the code and the colab notebooks you alternate between calling tokenizer.encode() and indexing into the tokenizer.encoder dict. What's the difference between the two? tokenizer.encode('the') returns [518] but tokenizer.encoder['the'] returns 599.

Thanks!

Originally created by @normster on GitHub (Mar 10, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/56 Hi, I noticed that in the code and the colab notebooks you alternate between calling tokenizer.encode() and indexing into the tokenizer.encoder dict. What's the difference between the two? `tokenizer.encode('the')` returns `[518]` but `tokenizer.encoder['the']` returns `599`. Thanks!

bsakaguchi closed this issue

2026-10-31 21:10:15 +00:00

bsakaguchi commented

2026-10-31 21:10:15 +00:00

Author

Owner

@crowsonkb commented on GitHub (Mar 19, 2021):

If you look at the tokenizer.encoder dict, you can see that 599 is 'the' but 518 is 'the' followed by a word boundary. So .encode('the p') gives [518, 335] while .encode('thep') gives [599, 335]. It's probably correct to use the encode() method.

@crowsonkb commented on GitHub (Mar 19, 2021): If you look at the tokenizer.encoder dict, you can see that 599 is 'the' but 518 is 'the' followed by a word boundary. So `.encode('the p')` gives `[518, 335]` while `.encode('thep')` gives `[599, 335]`. It's probably correct to use the `encode()` method.

bsakaguchi referenced this issue

2026-10-31 21:10:26 +00:00

[GH-ISSUE #71] Doesn't work on PyTorch 1.8 #65

bsakaguchi referenced this issue

2026-10-31 21:10:29 +00:00

[GH-ISSUE #79] TypeError: 'torch._C.Node' object is not subscriptable #71