OlmPool

Minor architectural choices severely impact LLM long-context extension, study finds

A new research paper published on arXiv details how seemingly minor architectural choices in transformer models can significantly impact their ability to extend context length. The study found that combining three or mo…