Growing to See

Hybrid LLMs suffer from "Competing Learning"; staged training offers solution

Researchers have identified a phenomenon called "Competing Learning" in hybrid language models, where recurrence and attention layers interfere with each other's learning processes. They found that recurrence layers oft…