Caracal

Caracal architecture uses Fourier transforms for efficient long-sequence modeling

Researchers have introduced Caracal, a new architecture designed to improve the scalability of large language models for processing long sequences. Caracal replaces the computationally expensive attention mechanism with…