Dynamo Request Planes User Guide #

Overview #

Dynamo supports multiple transport mechanisms for its request plane (the communication layer between services). You can choose from three different request plane modes based on your deployment requirements:

• TCP (default): Direct TCP connection for optimal performance

• NATS: Message broker-based request plane

• HTTP: HTTP/2-based request plane

This guide explains how to configure and use request plane in your Dynamo deployment.

What is a Request Plane? #

The request plane is the transport layer that handles communication between Dynamo services (e.g., frontend to backend, worker to worker). Different request planes offer different trade-offs:

Request Plane vs KV Event Plane #

Dynamo has two independent communication planes:

• Request plane ( DYN_REQUEST_PLANE): how RPC requests flow between components (frontend → router → worker), via tcp, http, or nats.

• KV event plane (currently only NATS is supported): how KV cache events (and optional router replica sync) are distributed/persisted for KV-aware routing.

Note: If you are using tcp or http request plane with KV events enabled (default), NATS is automatically initialized. You can optionally configure NATS_SERVER environment variable (e.g., NATS_SERVER=nats://nats-hostname:port) to specify a custom NATS server; otherwise, it defaults to localhost:4222. To completely disable NATS, use --no-kv-events on the frontend.

Because they are independent, you can mix them.

For example, a deployment with TCP request plane can use different KV event planes:

• JetStream KV events: requests use TCP, KV routing still uses NATS JetStream + object store for persistence.

• NATS Core KV events (local indexer): requests use TCP, KV events use NATS Core pub/sub and persistence lives on workers.

• no KV events: requests use TCP and KV routing runs without events (no NATS required, but no event-backed persistence).

Configuration #

Environment Variable #

Set the request plane mode using the DYN_REQUEST_PLANE environment variable:

export DYN_REQUEST_PLANE=<mode>

Copy to clipboard

Where <mode> is one of:

• tcp (default)

• nats

• http

The value is case-insensitive.

Default Behavior #

If DYN_REQUEST_PLANE is not set or contains an invalid value, Dynamo defaults to tcp.

Usage Examples #

Using TCP (Default) #

TCP is the default request plane and provides direct, low-latency communication between services.

Configuration:

# TCP is the default, so no need to set DYN_REQUEST_PLANE explicitly # But you can explicitly set it if desired: export DYN_REQUEST_PLANE=tcp # Optional: Configure TCP server host and port export DYN_TCP_RPC_HOST=0.0.0.0 # Default host # export DYN_TCP_RPC_PORT=9999 # Optional: specify a fixed port # Run your Dynamo service DYN_REQUEST_PLANE=tcp python -m dynamo.frontend --http-port=8000 & DYN_REQUEST_PLANE=tcp python -m dynamo.vllm --model Qwen/Qwen3-0.6B

Copy to clipboard

Note: By default, TCP uses an OS-assigned free port (port 0). This is ideal for environments where multiple services may run on the same machine or when you want to avoid port conflicts. If you need a specific port (e.g., for firewall rules), set DYN_TCP_RPC_PORT explicitly.

When to use TCP:

• Simple deployments with direct service-to-service communication (e.g. frontend to backend)

• Minimal infrastructure requirements (NATS is initialized by default for KV events but can be disabled with --no-kv-events)

• Low-latency requirements

TCP Configuration Options:

Additional TCP-specific environment variables:

• DYN_TCP_RPC_HOST: Server host address (default: auto-detected)

• DYN_TCP_RPC_PORT: Server port. If not set, the OS assigns a free port automatically (recommended for most deployments). Set explicitly only if you need a specific port for firewall rules.

• DYN_TCP_MAX_MESSAGE_SIZE: Maximum message size for TCP client (default: 32MB)

• DYN_TCP_REQUEST_TIMEOUT: Request timeout for TCP client (default: 10 seconds)

• DYN_TCP_POOL_SIZE: Connection pool size for TCP client (default: 50)

• DYN_TCP_CONNECT_TIMEOUT: Connect timeout for TCP client (default: 3 seconds)

• DYN_TCP_CHANNEL_BUFFER: Request channel buffer size for TCP client (default: 100)

Using HTTP #

HTTP/2 provides a standards-based request plane that’s easy to debug and widely compatible.

Configuration:

# Optional: Configure HTTP server host and port export DYN_HTTP_RPC_HOST=0.0.0.0 # Default host export DYN_HTTP_RPC_PORT=8888 # Default port export DYN_HTTP_RPC_ROOT_PATH=/v1/rpc # Default path # Run your Dynamo service DYN_REQUEST_PLANE=http python -m dynamo.frontend --http-port=8000 & DYN_REQUEST_PLANE=http python -m dynamo.vllm --model Qwen/Qwen3-0.6B

Copy to clipboard

When to use HTTP:

• Standard deployments requiring HTTP compatibility

• Debugging scenarios (use curl, browser tools, etc.)

• Integration with HTTP-based infrastructure

• Load balancers and proxies that work with HTTP

HTTP Configuration Options:

Additional HTTP-specific environment variables:

• DYN_HTTP_RPC_HOST: Server host address (default: auto-detected)

• DYN_HTTP_RPC_PORT: Server port (default: 8888)

• DYN_HTTP_RPC_ROOT_PATH: Root path for RPC endpoints (default: /v1/rpc)

DYN_HTTP2_*: Various HTTP/2 client configuration options

• DYN_HTTP2_MAX_FRAME_SIZE: Maximum frame size for HTTP client (default: 1MB)

• DYN_HTTP2_MAX_CONCURRENT_STREAMS: Maximum concurrent streams for HTTP client (default: 1000)

• DYN_HTTP2_POOL_MAX_IDLE_PER_HOST: Maximum idle connections per host for HTTP client (default: 100)

• DYN_HTTP2_POOL_IDLE_TIMEOUT_SECS: Idle timeout for HTTP client (default: 90 seconds)

• DYN_HTTP2_KEEP_ALIVE_INTERVAL_SECS: Keep-alive interval for HTTP client (default: 30 seconds)

• DYN_HTTP2_KEEP_ALIVE_TIMEOUT_SECS: Keep-alive timeout for HTTP client (default: 10 seconds)

• DYN_HTTP2_ADAPTIVE_WINDOW: Enable adaptive flow control (default: true)

Using NATS #

NATS provides durable jetstream messaging for request plane and can be used for KV events (and router replica sync).

Prerequisites:

• NATS server must be running and accessible

• Configure NATS connection via standard Dynamo NATS environment variables

# Explicitly set to NATS export DYN_REQUEST_PLANE=nats # Run your Dynamo service DYN_REQUEST_PLANE=nats python -m dynamo.frontend --http-port=8000 & DYN_REQUEST_PLANE=nats python -m dynamo.vllm --model Qwen/Qwen3-0.6B

Copy to clipboard

When to use NATS:

• Production deployments with service discovery

• KV-aware routing with accurate cache state tracking (requires NATS for event transport). Note: approximate mode (--no-kv-events) provides KV routing without NATS but with reduced accuracy.

• Need for message replay and persistence features

Limitations:

• NATS does not support payloads beyond 16MB (use TCP for larger payloads)

Complete Example #

Here’s a complete example showing how to launch a Dynamo deployment with different request planes:

See examples/backends/vllm/launch/agg_request_planes.sh for a complete working example that demonstrates launching Dynamo with TCP, HTTP, or NATS request planes.

Real-World Example #

The Dynamo repository includes a complete example demonstrating all three request planes:

Location:examples/backends/vllm/launch/agg_request_planes.sh

cd examples/backends/vllm/launch # Run with TCP ./agg_request_planes.sh --tcp # Run with HTTP ./agg_request_planes.sh --http # Run with NATS ./agg_request_planes.sh --nats

Copy to clipboard

Architecture Details #

Network Manager #

The request plane implementation is centralized in the Network Manager (lib/runtime/src/pipeline/network/manager.rs), which:

1. Reads the DYN_REQUEST_PLANE environment variable at startup

2. Creates the appropriate server and client implementations

3. Provides a transport-agnostic interface to the rest of the codebase

4. Manages all network configuration and lifecycle

Transport Abstraction #

All request plane implementations conform to common trait interfaces:

• RequestPlaneServer: Server-side interface for receiving requests

• RequestPlaneClient: Client-side interface for sending requests

This abstraction means your application code doesn’t need to change when switching request planes.

Configuration Loading #

Request plane configuration is loaded from environment variables at startup and cached globally. The configuration hierarchy is:

1. Mode Selection: DYN_REQUEST_PLANE (defaults to tcp)

2. Transport-Specific Config: Mode-specific environment variables (e.g., DYN_TCP_*, DYN_HTTP2_*)

Migration Guide #

From NATS to TCP #

1. Stop your Dynamo services

2. Set environment variable DYN_REQUEST_PLANE=tcp

3. Optionally configure TCP-specific settings (e.g., DYN_TCP_RPC_HOST). Note: DYN_TCP_RPC_PORT is optional; if not set, an OS-assigned free port is used automatically.

4. Restart your services

From NATS to HTTP #

1. Stop your Dynamo services

2. Set environment variable DYN_REQUEST_PLANE=http

3. Optionally configure HTTP-specific settings (DYN_HTTP_RPC_PORT, etc.)

4. Restart your services

Testing the Migration #

After switching request planes, verify your deployment:

# Test with a simple request curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello!"}] }'

Copy to clipboard

Troubleshooting #

Issue: Services Can’t Communicate #

Symptoms: Requests timeout or fail to reach the backend

Solutions:

• Verify all services use the same DYN_REQUEST_PLANE setting

• Check that server ports are not blocked by k8s network policies or firewalls

• For TCP/HTTP: Ensure host/port configurations are correct and accessible

• For NATS: Verify NATS server is running and accessible

Issue: “Invalid request plane mode” Error #

Symptoms: Service fails to start with configuration error

Solutions:

• Check DYN_REQUEST_PLANE spelling (valid values: nats, tcp, http)

• Value is case-insensitive but must be one of the three options

• If not set, defaults to tcp

Issue: Port Conflicts #

Symptoms: Server fails to start due to “address already in use”

Solutions:

• TCP: By default, TCP uses an OS-assigned free port, so port conflicts should be rare. If you explicitly set DYN_TCP_RPC_PORT to a specific port and get conflicts, either change the port or remove the setting to use automatic port assignment.

• HTTP default port: 8888 (adjust environment variable DYN_HTTP_RPC_PORT)

Performance Considerations #

Latency #

• TCP: Lowest latency due to direct connections and binary serialization

• HTTP: Moderate latency with HTTP/2 overhead

• NATS: Moderate latency due to nats jet stream persistence

Resource Usage #

• TCP: Minimal infrastructure (NATS required only if using KV events, can disable with --no-kv-events)

• HTTP: Minimal infrastructure (NATS required only if using KV events, can disable with --no-kv-events)

• NATS: Requires running NATS server (additional memory/CPU)

Related Articles

• KV Router — NVIDIA Dynamo Documentation
• SLA-Driven Profiling and Planner Deployment Quick Start Guide — NVIDIA Dynamo Documentation
• Event Plane Architecture — NVIDIA Dynamo Documentation