Inference Optimization

Optimizing deployed model serving to reduce latency, improve throughput, and control GPU use in production inference.

Business impact

• Token generation speed — OpenAI reports GPT-6 Astra Ultrafast as up to 8x faster token generation than Astra Standard mode.
• Response latency — The evidence says the work makes model responses faster, especially in latency-sensitive workflows.
• Infrastructure utilization — The article says the programmable platform helps repurpose compute as demand changes and avoid overprovisioning.

Data requirements

• Model inference requests (text) — The serving systems handle LLM requests and outputs.
• Workflow context and prompts (text) — The OpenAI example refers to tool calls, coding agents, and session context.
• GPU workload and scheduling data (tabular) — The Netflix example includes GPU scheduling for production workloads.

AI methods and techniques

• Inference optimization — Used to refine deployed model-serving software and make responses faster.
• GPU scheduling — Used in the internal serving platform to allocate GPUs for real-time and batch workloads.
• Model deployment — Used to run LLM inference in production serving systems.
• Prompt caching — Used by Uber to control agent-work costs while usage rises.
• Model routing — Used by Uber to manage which model handles a request and reduce cost.
• Session context limiting — Used by Uber to control the cost of AI sessions.

AI models and model families

Triton, vLLM, NVIDIA Blackwell GPUs

Industries

Real-world evidence

3 documented case studies on record.

Companies using this: Netflix, Uber, OpenAI.