PulseAugur
EN
LIVE 22:21:06

Perplexity details new serving infrastructure for faster, cheaper AI search

By PulseAugur Editorial · [8 sources] ·

Perplexity has detailed its new serving infrastructure, designed to enhance both latency and throughput for embedding workloads. The system comprises three key components: Ivy, the HTTP gateway for request preparation; Tulip, a Rust gRPC inference server that batches requests and utilizes CUDA graphs and LazyTensors for efficiency; and ROSE, the model engine that handles forward passes and manages attention backends like ragged attention for embeddings. This integrated approach aims to provide faster search results at a reduced cost compared to existing solutions. AI

IMPACT Optimizes AI search infrastructure, potentially leading to faster and more cost-effective AI-powered search experiences.

RANK_REASON Perplexity details its internal serving infrastructure components and techniques.

Read on X — Perplexity →

AI-generated summary · Google Gemini · from 8 sources. How we write summaries →

Perplexity details new serving infrastructure for faster, cheaper AI search

How we ranked this

Signal score
10 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Perplexity details its internal serving infrastructure components and techniques.
Source corroboration
8 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
infra, product
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [8]

  1. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Our serving infrastructure lowers latency and improves throughput across both online and batch embedding workloads.

    Our serving infrastructure lowers latency and improves throughput across both online and batch embedding workloads. Combining Ivy, Tulip, and ROSE results in faster search at a reduced cost compared to off-the-shelf solutions. https://t.co/kAxbAz168Y

  2. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Ivy is the HTTP gateway.

    Ivy is the HTTP gateway. It handles CPU-side request prep: parsing, tokenization, templating, and splitting large batches before sending them to Tulip over gRPC. It allows us to tune request formatting and tokenization without touching the heavier inference servers.

  3. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    ROSE is the model engine.

    ROSE is the model engine. It reuses the same kernels for LLMs and embeddings. For embeddings, it skips the KV cache and uses ragged attention instead of paged attention. ROSE supports multiple attention backends, so kernel choice depends on model shape and sequence length. http…

  4. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Tulip uses two techniques to lower latency and boost throughput.

    Tulip uses two techniques to lower latency and boost throughput. CUDA graphs pre‑record GPU work so it launches in one call, reducing CPU overhead. LazyTensors tracks results asynchronously, letting the CPU prep the next batch while the GPU finishes the current one. https://t.c…

  5. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Tulip is Perplexity’s lightweight Rust gRPC inference server that sits between Ivy and the ROSE engine. It collects incoming requests, batches them, and sends t

    Tulip is Perplexity’s lightweight Rust gRPC inference server that sits between Ivy and the ROSE engine. It collects incoming requests, batches them, and sends them to the GPU. For small embedding models, runtime depends on tokens, not query count, so ~512 tokens fills the GPU. h…

  6. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Perplexity exposes inference via standardized APIs.

    Perplexity exposes inference via standardized APIs. Ivy (Rust HTTP) does CPU work—parsing, tokenization, templating—and translates to gRPC. Tulip (Rust gRPC) schedules batches for the ROSE engine. ROSE (Python) implements forward passes and CUDA graph management. https://t.co/…

  7. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Perplexity embeds queries and documents into one vector space, then searches by nearest vectors.

    Perplexity embeds queries and documents into one vector space, then searches by nearest vectors. This creates two workloads: bulk batch embedding for indexing and scoring (throughput‑focused) and per‑query online embedding for live search (latency‑focused).

  8. X — Perplexity TIER_1 English(EN) · perplexity_ai ·

    Every answer in Perplexity starts with embedding and ranking models picking the most relevant results for the query.

    Every answer in Perplexity starts with embedding and ranking models picking the most relevant results for the query. Today we published research on how we built SoTA serving infrastructure behind those models. Read the research: https://t.co/6Rt2XlxoXA https://t.co/ORGt9TjkR0

RELATED ENTITIES

RELATED TOPICS