fastmcp/docs/integrations/opentelemetry.mdx

---
title: OpenTelemetry Integration
description: Instrument your FastMCP server with OpenTelemetry for distributed tracing and observability
icon: chart-line
---

import { VersionBadge } from "/snippets/version-badge.mdx"

FastMCP includes built-in OpenTelemetry instrumentation that automatically creates spans for all MCP operations. The integration provides comprehensive observability through distributed tracing, logging, and metrics with zero configuration required.

## Why OpenTelemetry?

OpenTelemetry is the industry-standard observability framework that provides:

- **Distributed Tracing**: Track MCP operations across your system with spans
- **Structured Logging**: Export FastMCP logs to observability backends
- **Metrics Collection**: Monitor performance and usage patterns
- **Vendor Agnostic**: Works with Jaeger, Zipkin, Grafana, Datadog, and more
- **Production Ready**: Battle-tested with stable APIs for tracing and metrics

## Quick Start

FastMCP includes OpenTelemetry middleware out of the box. Simply add the middleware to your server:

```python
from fastmcp import FastMCP
from fastmcp.server.middleware.opentelemetry import OpenTelemetryMiddleware

mcp = FastMCP("My Server")
mcp.add_middleware(OpenTelemetryMiddleware())

@mcp.tool()
def greet(name: str) -> str:
    return f"Hello, {name}!"
```

If you don't have OpenTelemetry installed, the middleware gracefully becomes a no-op. To enable full instrumentation:

```bash
pip install fastmcp[opentelemetry]
```

Or install the packages directly:

```bash
pip install opentelemetry-api opentelemetry-sdk
```

For production deployments with OTLP export:

```bash
pip install opentelemetry-exporter-otlp-proto-grpc
```

<Note>
OpenTelemetry supports Python 3.9 and higher. Tracing and metrics are stable, while logging is in active development.
</Note>

## Logging Integration

FastMCP uses Python's standard `logging` module, which OpenTelemetry can instrument directly using `LoggingHandler`. This sends your FastMCP logs to any OpenTelemetry-compatible backend.

### Basic Setup

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, ConsoleLogExporter

from fastmcp import FastMCP
from fastmcp.utilities.logging import get_logger

# Configure OpenTelemetry
resource = Resource(attributes={
    "service.name": "my-fastmcp-server",
    "service.version": "1.0.0",
})

# Set up tracing
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(trace_provider)

# Set up logging
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(ConsoleLogExporter()))
set_logger_provider(logger_provider)

# Attach OpenTelemetry to FastMCP's logger
fastmcp_logger = get_logger("my_server")
fastmcp_logger.addHandler(LoggingHandler(logger_provider=logger_provider))

# Create your FastMCP server
mcp = FastMCP("My Server")

@mcp.tool()
def greet(name: str) -> str:
    """Greet someone by name."""
    fastmcp_logger.info(f"Greeting {name}")
    return f"Hello, {name}!"
```

### Production OTLP Export

For production environments, replace console exporters with OTLP exporters:

```python
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

# Configure OTLP endpoint (e.g., Grafana, Jaeger, or any OTLP collector)
otlp_endpoint = "http://localhost:4317"

# Tracing
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=otlp_endpoint))
)
trace.set_tracer_provider(trace_provider)

# Logging
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(endpoint=otlp_endpoint))
)
set_logger_provider(logger_provider)
```

### Structured Logging with OpenTelemetry

FastMCP's `StructuredLoggingMiddleware` outputs JSON logs that OpenTelemetry collectors can parse and enrich:

```python
from fastmcp import FastMCP
from fastmcp.server.middleware.logging import StructuredLoggingMiddleware

mcp = FastMCP("Structured Server")

# Add structured logging middleware
mcp.add_middleware(StructuredLoggingMiddleware(
    include_payloads=True,
    max_payload_length=1000
))

# OpenTelemetry will capture these structured logs
```

The structured logs include metadata like request timestamps, method names, token estimates, and payload sizes - perfect for observability platforms.

## Built-in Tracing Middleware

FastMCP includes `OpenTelemetryMiddleware` that automatically creates spans for all MCP operations including tools, resources, prompts, and list operations.

### Configuration Options

The middleware supports several configuration options:

```python
from fastmcp import FastMCP
from fastmcp.server.middleware.opentelemetry import OpenTelemetryMiddleware

mcp = FastMCP("My Server")

# Default configuration (recommended)
mcp.add_middleware(OpenTelemetryMiddleware())

# Custom configuration
mcp.add_middleware(OpenTelemetryMiddleware(
    tracer_name="my-custom-tracer",       # Custom tracer name
    enabled=True,                          # Explicitly enable/disable
    include_arguments=False,               # Don't include arguments for privacy
    max_argument_length=1000,             # Limit argument string length in spans
    propagate_context=True                # Enable trace context propagation (default)
))
```

### Trace Context Propagation

The middleware automatically propagates trace context across MCP calls using the `_meta` field. This enables distributed tracing even when using protocols like SSE that don't support standard HTTP headers.

**How it works:**

1. **Incoming requests**: The middleware extracts trace context (W3C `traceparent` and `tracestate`) from the request's `_meta` field
2. **Span creation**: New spans are created as children of the incoming trace context
3. **Outgoing responses**: The middleware injects the current trace context into the response's `_meta` field

This means that if a client includes trace context in their request metadata, your server's spans will be linked to the client's trace. Similarly, if your server calls another MCP server, you can propagate context downstream.

**Example: Client propagating context**

```python
# Client code - sending trace context
result = await client.call_tool(
    "my_tool",
    {"arg": "value"},
    _meta={"traceparent": "00-trace_id-span_id-01", "tracestate": "vendor=value"}
)

# The server will create spans as children of this trace
# And the response will include updated trace context in result.meta
if result.meta:
    downstream_traceparent = result.meta.get("traceparent")
```

To disable context propagation:

```python
mcp.add_middleware(OpenTelemetryMiddleware(propagate_context=False))
```

### What Gets Traced

The middleware automatically creates spans for:

- **Tool Calls** (`tool.{name}`): Includes tool name, arguments, success status
- **Resource Reads** (`resource.read`): Includes resource URI
- **Prompt Retrievals** (`prompt.{name}`): Includes prompt name and arguments
- **List Operations**: Includes count of items returned
  - `tools.list`
  - `resources.list`
  - `resource_templates.list`
  - `prompts.list`

All spans include:
- MCP method name
- Source (client/server)
- Message type (request/notification)
- Success/error status
- Exception details on failure

### Custom Tracing Middleware

If you need additional custom spans beyond what the built-in middleware provides, you can extend the `Middleware` base class:

```python
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from fastmcp.server.middleware import Middleware, MiddlewareContext

class CustomTracingMiddleware(Middleware):
    """Add custom spans for specific business logic."""

    def __init__(self):
        self.tracer = trace.get_tracer("my-custom-tracer")

    async def on_call_tool(self, context: MiddlewareContext, call_next):
        """Add custom spans around tool calls."""
        tool_name = context.message.name

        # Create a child span with custom attributes
        with self.tracer.start_as_current_span(
            f"custom.{tool_name}",
            attributes={"custom.attribute": "value"}
        ) as span:
            result = await call_next(context)
            # Add custom business logic attributes
            span.set_attribute("custom.result_type", type(result).__name__)
            return result

# Stack middleware - built-in first, then custom
mcp.add_middleware(OpenTelemetryMiddleware())  # Built-in tracing
mcp.add_middleware(CustomTracingMiddleware())  # Your custom spans
```

## Complete Example

Here's a production-ready example combining logging and tracing:

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, ConsoleLogExporter

from fastmcp import FastMCP
from fastmcp.utilities.logging import get_logger
from fastmcp.server.middleware.opentelemetry import OpenTelemetryMiddleware

# Configure OpenTelemetry
resource = Resource(attributes={
    "service.name": "weather-mcp-server",
    "service.version": "1.0.0",
    "deployment.environment": "production",
})

# Tracing setup
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(trace_provider)

# Logging setup
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(ConsoleLogExporter()))
set_logger_provider(logger_provider)

# Create FastMCP server
mcp = FastMCP("Weather Server")

# Attach OpenTelemetry to FastMCP logger
logger = get_logger("weather")
logger.addHandler(LoggingHandler(logger_provider=logger_provider))

# Add built-in tracing middleware
mcp.add_middleware(OpenTelemetryMiddleware())

@mcp.tool()
def get_weather(city: str) -> dict:
    """Get weather for a city."""
    logger.info(f"Fetching weather for {city}")
    return {"city": city, "temp": 72, "condition": "sunny"}

if __name__ == "__main__":
    mcp.run()
```

## Exporting to Observability Backends

### Console Exporter (Development)

The console exporter is perfect for local development and testing:

```python
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from opentelemetry.sdk._logs.export import ConsoleLogExporter

# Already shown in examples above
trace_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
logger_provider.add_log_record_processor(BatchLogRecordProcessor(ConsoleLogExporter()))
```

### OTLP Exporter (Production)

OTLP (OpenTelemetry Protocol) works with most modern observability platforms:

```python
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

# Configure for your backend
otlp_endpoint = "http://your-collector:4317"

trace_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=otlp_endpoint))
)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(endpoint=otlp_endpoint))
)
```

Supported backends include:
- **Grafana** with Tempo and Loki
- **Jaeger** for distributed tracing
- **Zipkin** for trace visualization
- **Datadog**, **New Relic**, **Honeycomb** (commercial platforms)
- **Self-hosted** OpenTelemetry Collector

### Environment Variables

OpenTelemetry exporters can be configured via environment variables:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="my-fastmcp-server"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production"
```

Then in your code:

```python
# OpenTelemetry will automatically use environment variables
trace_provider = TracerProvider()
trace_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter())  # Uses OTEL_EXPORTER_OTLP_ENDPOINT
)
```

## Best Practices

### When to Use Logging vs Spans

- **Logging**: Discrete events, errors, diagnostic messages
- **Spans**: Operations with duration, distributed tracing across services

For FastMCP servers:
- Use **spans** for tool calls, resource reads, prompt executions
- Use **logging** for validation errors, configuration issues, business logic events

### Performance Considerations

OpenTelemetry is designed for production, but follow these guidelines:

1. **Use BatchProcessors**: Always use `BatchSpanProcessor` and `BatchLogRecordProcessor` rather than synchronous exporters
2. **Sampling**: For high-volume servers, configure sampling to reduce overhead:

```python
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# Sample 10% of traces
trace_provider = TracerProvider(
    resource=resource,
    sampler=TraceIdRatioBased(0.1)
)
```

3. **Attribute Limits**: Avoid adding large payloads as span attributes. Use `max_payload_length` in middleware:

```python
# Good - limit attribute size
span.set_attribute("tool.arguments", str(args)[:500])

# Bad - unbounded attribute size
span.set_attribute("tool.arguments", str(args))  # Could be huge!
```

### Security: Avoiding Sensitive Data

Never log sensitive information in traces or logs:

```python
async def on_call_tool(self, context: MiddlewareContext, call_next):
    tool_name = context.message.name

    # Redact sensitive arguments
    safe_args = {
        k: v if k not in ["password", "api_key", "token"] else "***REDACTED***"
        for k, v in context.message.arguments.items()
    }

    with self.tracer.start_as_current_span(
        f"tool.{tool_name}",
        attributes={"tool.arguments": str(safe_args)}
    ) as span:
        return await call_next(context)
```

### Integration with Other Middleware

OpenTelemetry middleware works seamlessly with FastMCP's other built-in middleware:

```python
from fastmcp.server.middleware.opentelemetry import OpenTelemetryMiddleware
from fastmcp.server.middleware.timing import TimingMiddleware
from fastmcp.server.middleware.logging import LoggingMiddleware

# Order matters for middleware execution
mcp.add_middleware(OpenTelemetryMiddleware())  # Tracing first for complete lifecycle
mcp.add_middleware(TimingMiddleware())          # Timing within traces
mcp.add_middleware(LoggingMiddleware())         # Logging captures everything
```

The execution order ensures:
1. OpenTelemetry captures the complete request lifecycle including timing and logging
2. Timing data is included within trace spans
3. Logs are correlated with active traces
4. Everything is properly instrumented for observability

## Additional Resources

- [OpenTelemetry Python Documentation](https://opentelemetry.io/docs/languages/python/)
- [FastMCP Middleware Guide](/servers/middleware)
- [FastMCP Logging Guide](/servers/logging)
- [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/)

<Tip>
For examples and sample code, see [`examples/opentelemetry_example.py`](https://github.com/jlowin/fastmcp/tree/main/examples/opentelemetry_example.py) in the FastMCP repository.
</Tip>