Dynamo Logging — NVIDIA Dynamo Documentation
Title: Dynamo Logging — NVIDIA Dynamo Documentation
URL Source: https://docs.nvidia.com/dynamo/latest/observability/logging.html
Published Time: Fri, 07 Nov 2025 17:51:25 GMT
Markdown Content: Dynamo Logging#
Overview#
Dynamo provides structured logging in both text as well as JSONL. When JSONL is enabled logs additionally contain span creation and exit events as well as support for trace_id and span_id fields for distributed tracing.
Environment Variables for configuring Logging#
| Environment Variable | Description | Example Settings |
|---|---|---|
DYN_LOGGING_JSONL | Enable JSONL logging format (default: READABLE) | DYN_LOGGING_JSONL=true |
DYN_LOG_USE_LOCAL_TZ | Use local timezone for logging timestamps (default: UTC) | DYN_LOG_USE_LOCAL_TZ=1 |
DYN_LOG | Log levels per target <default_level>,<module_path>=<level>,<module_path>=<level> | DYN_LOG=info,dynamo_runtime::system_status_server:trace |
DYN_LOGGING_CONFIG_PATH | Path to custom TOML logging configuration file | DYN_LOGGING_CONFIG_PATH=/path/to/config.toml |
OTEL_SERVICE_NAME | Service name for OpenTelemetry traces (default: dynamo) | OTEL_SERVICE_NAME=dynamo-frontend |
OTEL_EXPORT_ENABLED | Enable OTLP trace exporting (set to 1 to enable) | OTEL_EXPORT_ENABLED=1 |
OTEL_EXPORT_ENDPOINT | OTLP exporter endpoint (default: http://localhost:4317) | OTEL_EXPORT_ENDPOINT=http://tempo:4317 |
Available Logging Levels#
| Logging Levels (Least to Most Verbose) | Description |
|---|---|
| ERROR | Critical errors (e.g., unrecoverable failures, resource exhaustion) |
| WARN | Unexpected or degraded situations (e.g., retries, recoverable errors) |
| INFO | Operational information (e.g., startup/shutdown, major events) |
| DEBUG | General debugging information (e.g., variable values, flow control) |
| TRACE | Very low-level, detailed information (e.g., internal algorithm steps) |
Example Readable Format#
Environment Setting:
export DYN_LOG="info,dynamo_runtime::system_status_server:trace" export DYN_LOGGING_JSONL="false"
Resulting Log format:
2025-09-02T15:50:01.770028Z INFO main.init: VllmWorker for Qwen/Qwen3-0.6B has been initialized 2025-09-02T15:50:01.770195Z INFO main.init: Reading Events from tcp://127.0.0.1:21555 2025-09-02T15:50:01.770265Z INFO main.init: Getting engine runtime configuration metadata from vLLM engine... 2025-09-02T15:50:01.770316Z INFO main.get_engine_cache_info: Cache config values: {'num_gpu_blocks': 24064} 2025-09-02T15:50:01.770358Z INFO main.get_engine_cache_info: Scheduler config values: {'max_num_seqs': 256, 'max_num_batched_tokens': 2048}
Example JSONL Format#
Environment Setting:
export DYN_LOG="info,dynamo_runtime::system_status_server:trace" export DYN_LOGGING_JSONL="true"
Resulting Log format:
{"time":"2025-09-02T15:53:31.943377Z","level":"INFO","target":"log","message":"VllmWorker for Qwen/Qwen3-0.6B has been initialized","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":191,"log.target":"main.init"} {"time":"2025-09-02T15:53:31.943550Z","level":"INFO","target":"log","message":"Reading Events from tcp://127.0.0.1:26771","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":212,"log.target":"main.init"} {"time":"2025-09-02T15:53:31.943636Z","level":"INFO","target":"log","message":"Getting engine runtime configuration metadata from vLLM engine...","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":220,"log.target":"main.init"} {"time":"2025-09-02T15:53:31.943701Z","level":"INFO","target":"log","message":"Cache config values: {'num_gpu_blocks': 24064}","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":267,"log.target":"main.get_engine_cache_info"} {"time":"2025-09-02T15:53:31.943747Z","level":"INFO","target":"log","message":"Scheduler config values: {'max_num_seqs': 256, 'max_num_batched_tokens': 2048}","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":268,"log.target":"main.get_engine_cache_info"}
OpenTelemetry Distributed Tracing#
When DYN_LOGGING_JSONL is enabled, Dynamo uses OpenTelemetry for distributed tracing. All logs include trace_id and span_id fields, and spans are automatically created for requests. By default, traces are not exported. To export traces to an observability backend (like Tempo, Jaeger, or Zipkin), set OTEL_EXPORT_ENABLED=1.
Behavior#
-
With
DYN_LOGGING_JSONL=trueonly: OpenTelemetry layer is active, generating trace context and span IDs for all requests. Traces appear in logs but are not exported anywhere. -
With
OTEL_EXPORT_ENABLED=1andDYN_LOGGING_JSONL=true: Same as above, plus traces are exported to an OTLP collector for visualization.
Configuration#
To enable OTLP trace exporting:
-
Set
OTEL_EXPORT_ENABLED=1to enable trace export -
Optionally configure the endpoint using
OTEL_EXPORT_ENDPOINT(default:http://localhost:4317) -
Optionally set
OTEL_SERVICE_NAMEto identify the service (useful in Kubernetes, default:dynamo)
Export Settings:
-
Protocol: gRPC (Tonic)
-
Service Name: Value of
OTEL_SERVICE_NAMEenv var, ordynamoif not set -
Endpoint: Value of
OTEL_EXPORT_ENDPOINTenv var, orhttp://localhost:4317if not set
Example: JSONL Logging Only (No Export)#
export DYN_LOGGING_JSONL=true
OpenTelemetry is active, traces appear in logs, but nothing is exported
Example: JSONL Logging + Trace Export to Tempo#
export DYN_LOGGING_JSONL=true export OTEL_EXPORT_ENABLED=1 export OTEL_EXPORT_ENDPOINT=http://tempo:4317 export OTEL_SERVICE_NAME=dynamo-frontend
OpenTelemetry is active, traces appear in logs AND are exported to Tempo
Trace and Span information#
When DYN_LOGGING_JSONL is enabled with DYN_LOG set to greater than or equal to info level, trace information is added to all logged spans along with SPAN_CREATED and SPAN_CLOSED events.
Example Request#
curl -d '{"model": "Qwen/Qwen3-0.6B", "max_completion_tokens": 2049, "messages":[{"role":"user", "content": "What is the capital of South Africa?" }]}' -H 'Content-Type: application/json' http://localhost:8080/v1/chat/completions
Example Logs#
Span Created in HTTP Frontend with trace_id and span_id
{"time":"2025-09-02T16:38:06.656503Z","level":"INFO","file":"/workspace/lib/runtime/src/logging.rs","line":248,"target":"dynamo_runtime::logging","message":"SPAN_CREATED","method":"POST","span_id":"6959a1b2d1ee41a5","span_name":"http-request","trace_id":"425ef761ca5b44c795b4c912f1d84b39","uri":"/v1/chat/completions","version":"HTTP/1.1"}
Span Created in Worker with trace_id and parent_id from frontend and new span_id
{"time":"2025-09-02T16:38:06.666672Z","level":"INFO","file":"/workspace/lib/runtime/src/pipeline/network/ingress/push_endpoint.rs","line":108,"target":"dynamo_runtime::pipeline::network::ingress::push_endpoint","message":"SPAN_CREATED","component":"backend","endpoint":"generate","instance_id":"7587888160958627596","namespace":"dynamo","parent_id":"6959a1b2d1ee41a5","span_id":"b035f33bdd5c4b50","span_name":"handle_payload","trace_id":"425ef761ca5b44c795b4c912f1d84b39"} {"time":"2025-09-02T16:38:06.685333Z","level":"WARN","target":"log","message":"cudagraph dispatching keys are not initialized. No cudagraph will be used.","log.file":"/opt/vllm/vllm/v1/cudagraph_dispatcher.py","log.line":101,"log.target":"cudagraph_dispatcher.dispatch"}
Span Closed in Worker with duration, busy, and idle information
{"time":"2025-09-02T16:38:08.787232Z","level":"INFO","file":"/workspace/lib/runtime/src/pipeline/network/ingress/push_endpoint.rs","line":108,"target":"dynamo_runtime::pipeline::network::ingress::push_endpoint","message":"SPAN_CLOSED","component":"backend","endpoint":"generate","instance_id":"7587888160958627596","namespace":"dynamo","parent_id":"6959a1b2d1ee41a5","span_id":"b035f33bdd5c4b50","span_name":"handle_payload","time.busy_us":1090,"time.duration_us":2121090,"time.idle_us":2120000,"trace_id":"425ef761ca5b44c795b4c912f1d84b39"}
Span Closed in HTTP Frontend with duration, busy and idle information
{"time":"2025-09-02T16:38:08.788268Z","level":"INFO","file":"/workspace/lib/runtime/src/logging.rs","line":248,"target":"dynamo_runtime::logging","message":"SPAN_CLOSED","method":"POST","span_id":"6959a1b2d1ee41a5","span_name":"http-request","time.busy_us":13000,"time.duration_us":2133000,"time.idle_us":2120000,"trace_id":"425ef761ca5b44c795b4c912f1d84b39","uri":"/v1/chat/completions","version":"HTTP/1.1"}
Example Request with User Supplied x-request-id#
curl -d '{"model": "Qwen/Qwen3-0.6B", "max_completion_tokens": 2049, "messages":[{"role":"user", "content": "What is the capital of South Africa?" }]}' -H 'Content-Type: application/json' -H 'x-request-id: 8372eac7-5f43-4d76-beca-0a94cfb311d0' http://localhost:8080/v1/chat/completions
Example Logs#
Span Created in HTTP Frontend with x-request-id, trace_id, and span_id
{"time":"2025-09-02T17:01:46.306801Z","level":"INFO","file":"/workspace/lib/runtime/src/logging.rs","line":248,"target":"dynamo_runtime::logging","message":"SPAN_CREATED","method":"POST","span_id":"906902a4e74b4264","span_name":"http-request","trace_id":"3924188ea88d40febdfa173afd246a3a","uri":"/v1/chat/completions","version":"HTTP/1.1","x_request_id":"8372eac7-5f43-4d76-beca-0a94cfb311d0"}
Span Created in Worker with trace_id, parent_id and x_request_id from frontend and new span_id
{"time":"2025-09-02T17:01:46.307484Z","level":"INFO","file":"/workspace/lib/runtime/src/pipeline/network/ingress/push_endpoint.rs","line":108,"target":"dynamo_runtime::pipeline::network::ingress::push_endpoint","message":"SPAN_CREATED","component":"backend","endpoint":"generate","instance_id":"7587888160958627596","namespace":"dynamo","parent_id":"906902a4e74b4264","span_id":"5a732a3721814f5e","span_name":"handle_payload","trace_id":"3924188ea88d40febdfa173afd246a3a","x_request_id":"8372eac7-5f43-4d76-beca-0a94cfb311d0"}
Span Closed in Worker with duration, busy, and idle information
{"time":"2025-09-02T17:01:47.975228Z","level":"INFO","file":"/workspace/lib/runtime/src/pipeline/network/ingress/push_endpoint.rs","line":108,"target":"dynamo_runtime::pipeline::network::ingress::push_endpoint","message":"SPAN_CLOSED","component":"backend","endpoint":"generate","instance_id":"7587888160958627596","namespace":"dynamo","parent_id":"906902a4e74b4264","span_id":"5a732a3721814f5e","span_name":"handle_payload","time.busy_us":646,"time.duration_us":1670646,"time.idle_us":1670000,"trace_id":"3924188ea88d40febdfa173afd246a3a","x_request_id":"8372eac7-5f43-4d76-beca-0a94cfb311d0"}
Span Closed in HTTP Frontend with duration, busy and idle information
{"time":"2025-09-02T17:01:47.975616Z","level":"INFO","file":"/workspace/lib/runtime/src/logging.rs","line":248,"target":"dynamo_runtime::logging","message":"SPAN_CLOSED","method":"POST","span_id":"906902a4e74b4264","span_name":"http-request","time.busy_us":2980,"time.duration_us":1672980,"time.idle_us":1670000,"trace_id":"3924188ea88d40febdfa173afd246a3a","uri":"/v1/chat/completions","version":"HTTP/1.1","x_request_id":"8372eac7-5f43-4d76-beca-0a94cfb311d0"}
Links/Buttons: