Perplexity 详细介绍了其新的服务基础设施，旨在提高嵌入式工作负载的延迟和吞吐量。该系统包含三个关键组件：Ivy，用于请求准备的 HTTP 网关；Tulip，一个使用 Rust 的 gRPC 推理服务器，它会批处理请求并利用 CUDA グラフ和 LazyTensors 来提高效率；以及 ROSE，模型引擎，负责处理前向传播并管理像 Ragged Attention 这样的注意力后端，用于嵌入式。这种集成方法旨在与现有解决方案相比，以更低的成本提供更快的搜索结果。 AI
影响 优化了 AI 搜索基础设施，可能带来更快、更具成本效益的 AI 驱动的搜索体验。
排序理由 Perplexity 详解了其内部服务基础设施组件和技术。
AI 生成摘要 · Google Gemini · 来自 8 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
Our serving infrastructure lowers latency and improves throughput across both online and batch embedding workloads. Combining Ivy, Tulip, and ROSE results in faster search at a reduced cost compared to off-the-shelf solutions. https://t.co/kAxbAz168Y
Ivy is the HTTP gateway. It handles CPU-side request prep: parsing, tokenization, templating, and splitting large batches before sending them to Tulip over gRPC. It allows us to tune request formatting and tokenization without touching the heavier inference servers.
ROSE is the model engine. It reuses the same kernels for LLMs and embeddings. For embeddings, it skips the KV cache and uses ragged attention instead of paged attention. ROSE supports multiple attention backends, so kernel choice depends on model shape and sequence length. http…
Tulip uses two techniques to lower latency and boost throughput. CUDA graphs pre‑record GPU work so it launches in one call, reducing CPU overhead. LazyTensors tracks results asynchronously, letting the CPU prep the next batch while the GPU finishes the current one. https://t.c…
Tulip is Perplexity’s lightweight Rust gRPC inference server that sits between Ivy and the ROSE engine. It collects incoming requests, batches them, and sends them to the GPU. For small embedding models, runtime depends on tokens, not query count, so ~512 tokens fills the GPU. h…
Perplexity exposes inference via standardized APIs. Ivy (Rust HTTP) does CPU work—parsing, tokenization, templating—and translates to gRPC. Tulip (Rust gRPC) schedules batches for the ROSE engine. ROSE (Python) implements forward passes and CUDA graph management. https://t.co/…
Perplexity embeds queries and documents into one vector space, then searches by nearest vectors. This creates two workloads: bulk batch embedding for indexing and scoring (throughput‑focused) and per‑query online embedding for live search (latency‑focused).
Every answer in Perplexity starts with embedding and ranking models picking the most relevant results for the query. Today we published research on how we built SoTA serving infrastructure behind those models. Read the research: https://t.co/6Rt2XlxoXA https://t.co/ORGt9TjkR0