Token-1B

Byte models show higher performance ceiling than token models in large-scale study

Researchers have explored the performance differences between token-based and byte-based language models, particularly in the context of distillation. They introduced two methods, Marginalize-It and End-Of-Token, to con…