Abrarqasim

AirLLM enables 70B model inference on 4GB GPU by streaming layers from disk

AirLLM is a new project that enables running large language models, such as a 70B parameter model, on hardware with very limited VRAM, like a 4GB GPU. It achieves this by loading model layers sequentially from disk to t…