New MLX LM Server From Apple
Apple has released MLX LM Server, a new tool designed to enhance the performance of large language models on Mac hardware. It leverages the M5 chip's neural accelerators for faster prompt processing and employs continuous batching to manage multiple requests concurrently. For extremely large models, the server supports distributed inference across multiple Macs using Thunderbolt RDMA. AI
IMPACT Enhances LLM inference capabilities on Apple hardware, potentially improving local AI development and deployment.