AnyBCQ

MatGPTQ enables efficient inference for nested quantized LLMs

Researchers have developed MatGPTQ, a new method for efficiently running large language models (LLMs) that have been quantized to use fewer bits. This approach allows a single model checkpoint to serve multiple precisio…