what's the function for tokens with prefix null_?

[KK666-AI]

In class Alphabet, i notice that the authors define some tokens with prefix null_, however, i don't understand their exact functions. Any insights for such null_ tokens?

see detailed implementation in the following codes (refers to this code):

for i in range((8 - (len(self.all_toks) % 8)) % 8):
    self.all_toks.append(f"<null_{i  + 1}>") 

[liujas000]

Hi @lihuiknight , the null_ tokens are an engineering solution to ensure the vocabulary size is a multiple of 8, which is helpful for any model-parallel runs this was an intentional choice (having all dimensions in linear layers to be multiples of 8) for training efficiency.

https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#tensorop

thanks to @joshim5!