B300s

Modal launches Auto Endpoints for state-of-the-art inference latency

Modal has launched Modal Auto Endpoints, a new service designed to achieve state-of-the-art inference latencies. The service leverages speculative decoding, a technique that allows for parallel processing of multiple to…