This is an automated archive made by the Lemmit Bot.

The original was posted on /r/stablediffusion by /u/lostinspaz on 2024-10-04 16:04:02+00:00.

A while ago, I did some blackbox analysis of CLIP (L,G) to learn more about them.

Now I’m starting to do similar things with T5 (specifically, t5xxl-enconly)

One odd thing I have discovered so far: It uses SentencePiece as its tokenizer, and from a human perspective, it can be stupid/wasteful.

Not as bad as the CLIP-L used in SD(xl), but still…

It is case sensitive. Which in some limited contexts I could see as a benefit, but its stupid for the following specific examples:

It has a fixed number of unique token IDs. around 32,000.

Of those, 9000 of them are tied to explicit Uppercase use.

Some of them make sense. But then there are things like this:

“Title” and “title” have their own unique token IDs

“Cushion” and “cushion” have their own unique token IDs.

???

I havent done a comprehensive analysis, but I would guess somewhere between 200 and 900 would be like this. The waste makes me sad.

Why does this matter?

Because any time a word doesnt have its own unique token id, it then has to be represented by multiple tokens. Multiple tokens, means multiple encodings (note: CLIP coalesces multiple tokens into a single text embedding. T5 does NOT!) , which means more work, which means calculations and generations take longer.

PS: my ongoing tools will be updated at