Details about text generation

[theseventhflow]

Hi lucidrains,
Thank you for your excellent code.
I am curious about the generation scripts. Could you tell me how to generate text with the compressive transformer? Because it has the compressive memory, maybe we cannot use the current predicted word as the input for the next generation (input length ==1). In addition, if the prompt has 100 words and we use tokens [0:100], tokens[1:101], tokens[2:102]... as the input for the following timesteps, the tokens[1:100] may overlap with the memory, because the memory already contains hidden states for tokens[1:100].

I would be very appeciated if you can provide the generation scripts!

Thank you

[theseventhflow]

I am sorry that I found your generation scripts just now. When the input length reaches the maximum, the next input will be only the last word. However, I still have some confusion about the overlapping of input and memory because the left-side part of the current input may have already been stored in the memory. Could you tell me why you generate in this way?

[lucidrains]

@theseventhflow Hello! This confused me as well initially, but after some research, I gathered that you always decode up until the maximum sequence length until actually doing the action of shifting to memory (and compressed memory, if it overflows)

[lucidrains]

so, for example, if you maximum sequence length is 4, and your prime is say 2 (t h), you would do:

1. t h e
2. t h e y -> filled, so shift to hidden states [ t h e y ]
3. [t h e y] c
4. [t h e y] c a
5. [t h e y] c a m e -> filled, so shift hidden states, compressed [t h e y] into compressed memory {x} assuming compression ratio of 4
6. {x} [c a m e] t
7. {x} [c a m e] t o

and so on