LiteLLM, Portkey, and cloud AI gateways

Overview

• LiteLLM provides model-access abstraction and proxy/gateway capabilities.
• Portkey provides an AI gateway and operational controls.
• Cloud gateways can apply access and model traffic policies within an existing cloud API-management environment; exact AI features vary by product and edition.

Why this matters

Existing tools can provide central controls without building the entire gateway yourself.

An everyday example

Choose a managed travel desk, operate your own desk, or extend the one already in your company.

A technical example

Proxy multiple model providers behind an internal endpoint while preserving capability differences in your adapters.

How it works

1. Configure routes/credentials
2. Define budgets and allowlists
3. Integrate client
4. Test quotas/failure/streaming
5. Monitor and audit.

Architecture and flow

Internal model contract → chosen gateway → approved route → provider.

The insurance assistant example

A Microsoft-centered insurer may evaluate its existing API-management stack; another team may use a model-specific proxy.

When to use it

When the chosen product meets concrete policy and operating requirements.

When to avoid it

Do not assume an OpenAI-compatible endpoint means identical schema, tool, reasoning, or multimodal behavior.

Interview answer

“I compare gateways by capability fidelity, policy controls, latency, privacy, availability, and operating cost.”

Benefits, trade-offs, and production considerations

• Validate support for each required feature and provider version.
• Compare self-hosted operation with managed cost and retention controls.
• LiteLLM, Portkey.