[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Security](index.md) > Integrating LiteLLM Proxy with Yandex Cloud AI Studio and Monium Traces

# Integrating LiteLLM Proxy with Yandex Cloud AI Studio and Monium Traces

[LiteLLM Proxy](https://openai.com/api/) is a proxy server that enables access to LLM providers through a single interface compatible with the [OpenAI API](https://docs.litellm.ai/docs/providers/litellm_proxy).

LiteLLM Proxy simplifies multi-model workflows by providing centralized application access management, rate limiting, cost tracking, and trace generation for request analysis.

You can monitor your requests using [Monium Traces](../../monium/traces/index.md), a distributed tracing system. A trace aggregates data about individual operations, such as their sequence, duration, and errors.

Traces help you:

* Identify the exact step, service, or application component where an error or latency occurs.
* Analyze prompts, responses, and token spend. For example, you can view the exact context provided to the model, generation time, and prompt/response token counts.

Integrating LiteLLM Proxy with the tracing system allows you to collect this data on the proxy side: you do not need to add telemetry code to your application. 

In this tutorial, you will deploy LiteLLM Proxy and configure trace export to Monium Traces via [OpenTelemetry v2](https://docs.litellm.ai/docs/observability/opentelemetry_v2). You will then run a test application simulating a conversation with the DeepSeek V4 Flash [model](https://aistudio.yandex.ru/docs/en/ai-studio/concepts/generation/models) in [Yandex Cloud AI Studio](https://aistudio.yandex.ru/docs/en/) through LiteLLM Proxy and analyze the captured traces.

```mermaid
flowchart LR
    subgraph VM["VM"]
        APP["Test app"]
        TOOLS["Local tools<br/>Knowledge base, order status, return"]
        PROXY["LiteLLM Proxy<br/>Unified API"]

        APP <-->|"Tool calls"| TOOLS
        APP <-->|"Requests and responses"| PROXY
    end

    MODEL["Yandex Cloud AI Studio<br/>DeepSeek V4 Flash"]
    TRACES["Monium Traces<br/>Analysis of delays and errors,<br/>prompts, responses and tokens"]

    PROXY <-->|"Prompts to the model"| MODEL
    PROXY -.->|"Traces · OpenTelemetry"| TRACES
```

## Required paid resources {#paid-resources}

The solution support costs include:

* VM computing resources and a disk (see [Yandex Compute Cloud pricing](../../compute/pricing.md)).
* Public IP address and outgoing traffic (see [Yandex Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Text generation (see [Yandex Cloud AI Studio pricing](https://aistudio.yandex.ru/docs/en/ai-studio/pricing)).
* Trace logging to Monium Traces (see [Yandex Monium pricing](../../monium/pricing.md)).

## Steps to follow {#procedure}

1. [Get your cloud ready](#before-you-begin).
1. [Deploy and configure the LiteLLM Proxy instance](#deploy-litellm-proxy).
1. [Send requests to the model and check the traces](#test-integration).

If you deployed this solution only for testing, you can [delete the created resources](#clear-out) after completing the tutorial.

## Getting started {#before-you-begin}


Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):

1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the [Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts) page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).



### Set up your environment {#prepare}

[Create](../../compute/operations/vm-create/create-linux-vm.md) a VM from the [Ubuntu 24.04 LTS](https://yandex.cloud/en/marketplace/products/yc/ubuntu-2404-lts-oslogin) public image.

{% note warning %}

When creating the VM, specify at least 4 GB of RAM.

{% endnote %}


### Create API keys {#create-api-keys}

Create two [service accounts](../../iam/concepts/users/service-accounts.md) with [API keys](../../iam/concepts/authorization/api-key.md) (one per service) so LiteLLM Proxy can access Yandex Cloud AI Studio and send traces to Monium Traces:

{% list tabs group=instructions %}

- Management console {#console}

  1. Create a service account to enable LiteLLM Proxy to access Yandex Cloud AI Studio:

      1. In the [management console](https://console.yandex.cloud), select the folder where you are going to create your infrastructure.
      1. [Navigate](https://console.yandex.cloud/link/iam) to **Identity and Access Management**.
      1. Click **Create service account**.
      1. In the window that opens, enter the service account name: `litellm-ai-studio-sa`.
      1. In the **Roles in the folder** field, click ![plus](../../_assets/console-icons/plus.svg) **Add role** and select [role](https://aistudio.yandex.ru/docs/en/ai-studio/security/#languageModels-user) `ai.languageModels.user`.
      1. Click **Create**.
  1. Create an API key for LiteLLM Proxy access to Yandex Cloud AI Studio:

      1. In the list of service accounts, select `litellm-ai-studio-sa`.
      1. At the top of the page, click ![ellipsis](../../_assets/console-icons/ellipsis.svg) and select ![plus](../../_assets/console-icons/plus.svg) **Create new key** → **Create API key**.
      1. In the window that opens, select a [scope](*key_scope_definition) `yc.ai.languageModels.execute` and click **Create**.
      1. Save the secret part of the API key. You will not be able to view it again after closing the dialog window.
  1. Create a service account and API key to enable LiteLLM Proxy to access Yandex Monium. Specify the following properties:

      * Service account name: `litellm-monium-sa`.
      * [Role](../../monium/security/index.md#monium-traces-writer): `monium.traces.writer`.
      * API key scope: `yc.monium.traces.write`.

{% endlist %}


## Deploy and configure the LiteLLM Proxy instance {#deploy-litellm-proxy}

To deploy LiteLLM Proxy on the [VM you created](#prepare), install Docker, prepare the configuration files, and start the container:

1. Connect to the VM [over SSH](../../compute/operations/vm-connect/ssh.md) or [via OS Login](../../compute/operations/vm-connect/os-login.md) depending on the connection method you selected when creating your resources.
1. Install and configure Docker:

    1. Install the required dependencies:

        ```bash
        sudo apt update
        sudo apt install ca-certificates curl gnupg
        ```
    1. Install the Docker [GPG](https://www.gnupg.org/) key:

        ```bash
        sudo install -m 0755 -d /etc/apt/keyrings
        sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
        sudo chmod a+r /etc/apt/keyrings/docker.asc
        ```
    1. Add the Docker repository to your `apt` package sources:

        ```bash
        echo \
          "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
          $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
          sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
        ```
    1. Install Docker:

        ```bash
        sudo apt update
        sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
        ```
    1. Add the current user to the `docker` group and open a new shell to apply the changes:

        ```bash
        sudo usermod -aG docker $USER
        newgrp docker
        ```
    1. Make sure Docker is installed and operational:

        ```bash
        docker run hello-world
        ```

        Result:

        ```text
        Unable to find image 'hello-world:latest' locally
        latest: Pulling from library/hello-world
        4f55086f7dd0: Pull complete 
        d5e71e642bf5: Download complete 
        Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe89********
        Status: Downloaded newer image for hello-world:latest

        Hello from Docker!
        This message shows that your installation appears to be working correctly.
        ...
        ```
1. In your home directory, create a project directory for Docker Compose and navigate into it:

    ```bash
    mkdir ~/litellm-proxy && cd ~/litellm-proxy
    ```
1. Create the required configuration files for the project:

    ```bash
    touch .env
    touch config.yaml
    touch docker-compose.yaml
    ```
1. In a text editor, open the `.env` file:

    ```bash
    nano .env
    ```
1. Add the following environment variables to the `.env` file:

    ```text
    YANDEX_AI_STUDIO_API_KEY=<ai_studio_key_secret_part>
    MONIUM_API_KEY=<monium_key_secret_part>
    MONIUM_PROJECT=folder__<folder_ID>
    LITELLM_MASTER_KEY=sk-local-litellm
    ```

    Where:

    * `YANDEX_AI_STUDIO_API_KEY`: Secret part of the API key for the `litellm-ai-studio-sa` service account.
    * `MONIUM_API_KEY`: Secret part of the API key for the `litellm-monium-sa` service account.
    * `MONIUM_PROJECT`: Monium project ID. It consists of the `folder__` prefix followed by your folder ID where the VM and service accounts were created. Here is an example: `folder__b1gt6g8ht345********`.
    * `LITELLM_MASTER_KEY`: Key that applications use to access LiteLLM Proxy. You define it in the `.env` file so the proxy can authenticate incoming requests. For this tutorial, keep the `sk-local-litellm` value. This value is used in the code of the `agent_example.py` test application you will create in the next step.

        {% note tip %}

        For production environments, run `openssl rand -hex 32` to generate a random string and prepend it with `sk-`.

        If you updated `LITELLM_MASTER_KEY` in the `.env` file after starting the container, recreate the container by running `docker compose up -d --force-recreate`. If using a custom key, make sure to set the exact same value in the `LITELLM_PROXY_KEY` environment variable in your terminal before running the `agent_example.py` test application:

        ```bash
        export LITELLM_PROXY_KEY='<key_from_env_file>'
        ```

        The `openai` library inside the test application automatically attaches this key to the `Authorization` header using the `Bearer` scheme. The header value must match the key configured for the proxy in `.env`.

        {% endnote %}

1. In a text editor, open the `config.yaml` file:

    ```bash
    nano config.yaml
    ```
1. Add the following configuration to the `config.yaml` file:

    ```yaml
    # LiteLLM proxy: Yandex AI Studio (OpenAI-compatible) → traces to Yandex Monium.
    # Model URI: gpt://<folder_ID>/deepseek-v4-flash

    model_list:
      - model_name: deepseek-v4-flash
        litellm_params:
          model: openai/gpt://<folder_ID>/deepseek-v4-flash
          api_base: https://ai.api.cloud.yandex.net/v1
          api_key: os.environ/YANDEX_AI_STUDIO_API_KEY

    litellm_settings:
      drop_params: true
      set_verbose: false
      # Both this callback and LITELLM_OTEL_V2=true are required.
      callbacks:
        - otel

    general_settings:
      master_key: os.environ/LITELLM_MASTER_KEY
    ```

    Where `model` is the model ID in LiteLLM: the `openai/` prefix and the Yandex Cloud AI Studio model URI containing your [folder ID](../../resource-manager/operations/folder/get-id.md). Here is an example: `openai/gpt://b1gt6g8ht345********/deepseek-v4-flash`.

    The `os.environ/` prefix indicates that the parameter value should be retrieved from the respective environment variable.
1. In a text editor, open the `docker-compose.yaml` file:

    ```bash
    nano docker-compose.yaml
    ```
1. Add the Docker Compose project configuration into the `docker-compose.yaml` file:

    ```yaml
    services:
      litellm:
        image: docker.litellm.ai/berriai/litellm:main-latest
        container_name: litellm-proxy
        restart: unless-stopped
        # Host network: on this class of host net.ipv4.ip_forward=0 and bridge NAT drops connections.
        network_mode: host
        volumes:
          - ./config.yaml:/app/config.yaml:ro
        environment:
          - LITELLM_OTEL_V2=true
          - OTEL_EXPORTER=otlp_http
          - OTEL_ENDPOINT=https://ingest.monium.yandex.cloud/otlp
          - OTEL_SERVICE_NAME=litellm-proxy
          - OTEL_ENVIRONMENT_NAME=local
          - OTEL_HEADERS=Authorization=Api-Key ${MONIUM_API_KEY:?MONIUM_API_KEY},x-monium-project=${MONIUM_PROJECT:?MONIUM_PROJECT},x-monium-cluster=${MONIUM_CLUSTER:-default},x-monium-service=litellm-proxy
          - OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
          - YANDEX_AI_STUDIO_API_KEY=${YANDEX_AI_STUDIO_API_KEY:?YANDEX_AI_STUDIO_API_KEY}
          - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-local-litellm}
        command:
          - --config
          - /app/config.yaml
          - --port
          - "4000"
          - --host
          - "0.0.0.0"
    ```

    Configuration parameters:

    * `environment`: Environment variables:

        * `OTEL_EXPORTER=otlp_http`: Configures trace delivery via OTLP over HTTP.
        * `OTEL_ENDPOINT=https://ingest.monium.yandex.cloud/otlp`: Sets the base endpoint address. LiteLLM Proxy appends `/v1/traces` to it and sends traces to `https://ingest.monium.yandex.cloud/otlp/v1/traces`.
        * `x-monium-service=litellm-proxy`: Name to assign to the service in the Monium UI.
        * `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only`: Enables capturing model request and response text inside [spans](*span_definition).
    * `network_mode: host`: Runs the container in the host's network.

        This allows the container to access Yandex Cloud AI Studio and Yandex Monium even if IP packet forwarding is disabled on the host (`net.ipv4.ip_forward = 0`). LiteLLM Proxy listens for incoming connections on VM port `4000`.
1. Run the container:

    ```bash
    docker compose up -d
    ```

    Result:

    ```text
    [+] up 27/27
    ✔ Image docker.litellm.ai/berriai/litellm:main-latest Pulled                                                      82.4s
    ✔ Container litellm-proxy                             Started                                                     43.1s
    ```
1. To check that LiteLLM Proxy is ready to accept requests, run this command in the VM terminal:

    ```bash
    for _ in $(seq 1 30); do
      curl -sf http://127.0.0.1:4000/health/liveliness >/dev/null && echo "proxy is up" && break
      sleep 1
    done
    ```

    Result:

    ```text
    proxy is up
    ```

## Send requests to the model and check the traces {#test-integration}

To test trace delivery, create and run a test application on the same VM where LiteLLM Proxy is running. The application will route requests to the model through the proxy, which in turn will send the traces to Monium Traces. You can then find these traces in Monium to review the request details.

### Create and run a test application {#run-sample-application}

The Python-based test application simulates a customer conversation with a support agent. It uses the `openai` library to send requests to the model via LiteLLM Proxy. Install this library inside a dedicated virtual environment to isolate the application dependencies.

To create and run the application:

1. Navigate to the `~/litellm-proxy` directory you created when deploying LiteLLM Proxy on the VM:

    ```bash
    cd ~/litellm-proxy
    ```
1. Install Python and the `venv` module for virtual environment management:

    ```bash
    sudo apt install python3 python3.12-venv
    ```
1. Create and activate the `litellm-venv` virtual environment:

    ```bash
    python3 -m venv ~/litellm-venv
    source ~/litellm-venv/bin/activate
    ```
1. Install the `openai` library:

    ```bash
    pip install 'openai==2.54.0'
    ```
1. Create and open the `agent_example.py` test application file in a text editor:

    ```bash
    touch agent_example.py
    nano agent_example.py
    ```
1. Add this code to the `agent_example.py` file:

    {% cut "Application code" %}

    ```python
    """Customer support agent that talks to a local LiteLLM proxy only.
    
    No client-side telemetry: tracing is handled by the proxy (OTEL v2 → Monium).
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import os
    import random
    import sys
    
    from openai import OpenAI
    
    DEFAULT_PROXY_BASE = "http://127.0.0.1:4000/v1"
    DEFAULT_PROXY_KEY = "sk-local-litellm"
    DEFAULT_MODEL = "deepseek-v4-flash"
    
    
    class KnowledgeBase:
        """Simple knowledge base for customer support."""
    
        def __init__(self) -> None:
            self.knowledge = {
                "shipping": {
                    "standard": "Standard shipping takes 5-7 business days and costs $5.99",
                    "express": "Express shipping takes 2-3 business days and costs $12.99",
                    "overnight": "Overnight shipping delivers next business day and costs $24.99",
                },
                "returns": {
                    "policy": "You can return items within 30 days of purchase for a full refund",
                    "process": "To return an item, go to your account, find the order, and click 'Return Item'",
                    "condition": "Items must be in original condition with tags attached",
                },
                "account": {
                    "password_reset": "Click 'Forgot Password' on the login page and follow the email instructions",
                    "update_info": "You can update your account information in the 'My Account' section",
                    "delete_account": "Contact customer service to delete your account permanently",
                },
                "products": {
                    "warranty": "All products come with a 1-year manufacturer warranty",
                    "availability": "Check product availability on the product page",
                    "specifications": "Product specifications are listed in the product description",
                },
            }
    
        def search(self, query: str) -> str:
            query_lower = query.lower()
    
            if any(
                word in query_lower for word in ["ship", "delivery", "shipping", "deliver"]
            ):
                if "express" in query_lower or "fast" in query_lower:
                    return self.knowledge["shipping"]["express"]
                if "overnight" in query_lower or "next day" in query_lower:
                    return self.knowledge["shipping"]["overnight"]
                return self.knowledge["shipping"]["standard"]
    
            if any(word in query_lower for word in ["return", "refund", "exchange"]):
                if "how" in query_lower or "process" in query_lower:
                    return self.knowledge["returns"]["process"]
                if "condition" in query_lower:
                    return self.knowledge["returns"]["condition"]
                return self.knowledge["returns"]["policy"]
    
            if any(
                word in query_lower for word in ["account", "password", "login", "profile"]
            ):
                if "password" in query_lower or "reset" in query_lower:
                    return self.knowledge["account"]["password_reset"]
                if "update" in query_lower or "change" in query_lower:
                    return self.knowledge["account"]["update_info"]
                if "delete" in query_lower:
                    return self.knowledge["account"]["delete_account"]
    
            if any(
                word in query_lower for word in ["product", "warranty", "available", "spec"]
            ):
                if "warranty" in query_lower:
                    return self.knowledge["products"]["warranty"]
                if "available" in query_lower or "stock" in query_lower:
                    return self.knowledge["products"]["availability"]
                return self.knowledge["products"]["specifications"]
    
            return (
                "I couldn't find specific information about that. "
                "Let me help you with a general response."
            )
    
    
    class CustomerSupportAgent:
        """Customer support agent using an OpenAI-compatible LiteLLM proxy."""
    
        def __init__(
            self,
            *,
            base_url: str,
            api_key: str,
            model: str,
        ) -> None:
            self.client = OpenAI(base_url=base_url, api_key=api_key)
            self.model = model
            self.knowledge_base = KnowledgeBase()
            self.conversation_history: list[dict] = []
            self.system_prompt = """
    You are a helpful and friendly customer support agent. Your role is to:
    1. Assist customers with their questions and concerns
    2. Use the available tools whenever they help answer the question
    3. Be polite, professional, and empathetic
    4. If you don't know something, admit it and offer to escalate to a human agent
    5. Keep responses concise but helpful
    
    Always maintain a positive and solution-oriented attitude.
    Prefer calling tools for shipping, returns, order status, refunds, accounts, and products
    instead of inventing details.
    """.strip()
    
        def search_knowledge_base(self, query: str) -> str:
            return self.knowledge_base.search(query)
    
        def get_order_status(self, order_id: str) -> str:
            statuses = ["Processing", "Shipped", "Out for Delivery", "Delivered"]
            return f"Order {order_id} status: {random.choice(statuses)}"
    
        def process_refund(self, order_id: str, reason: str) -> str:
            return (
                f"Refund request for order {order_id} has been submitted. "
                f"Reason: {reason}. You'll receive confirmation within 24 hours."
            )
    
        def get_available_tools(self) -> list[dict]:
            return [
                {
                    "type": "function",
                    "function": {
                        "name": "search_knowledge_base",
                        "description": (
                            "Search the knowledge base for shipping, returns, "
                            "accounts, and products"
                        ),
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "query": {
                                    "type": "string",
                                    "description": "The customer's question or topic",
                                }
                            },
                            "required": ["query"],
                        },
                    },
                },
                {
                    "type": "function",
                    "function": {
                        "name": "get_order_status",
                        "description": "Get the current status of a customer's order",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "order_id": {
                                    "type": "string",
                                    "description": "The order ID to check",
                                }
                            },
                            "required": ["order_id"],
                        },
                    },
                },
                {
                    "type": "function",
                    "function": {
                        "name": "process_refund",
                        "description": "Process a refund request for a customer",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "order_id": {
                                    "type": "string",
                                    "description": "The order ID to refund",
                                },
                                "reason": {
                                    "type": "string",
                                    "description": "Reason for the refund",
                                },
                            },
                            "required": ["order_id", "reason"],
                        },
                    },
                },
            ]
    
        def execute_tool_call(self, tool_call) -> str:
            function_name = tool_call.function.name
            arguments = json.loads(tool_call.function.arguments)
    
            if function_name == "search_knowledge_base":
                return self.search_knowledge_base(arguments["query"])
            if function_name == "get_order_status":
                return self.get_order_status(arguments["order_id"])
            if function_name == "process_refund":
                return self.process_refund(arguments["order_id"], arguments["reason"])
            return f"Unknown tool: {function_name}"
    
        def chat(self, user_message: str) -> str:
            self.conversation_history.append({"role": "user", "content": user_message})
            messages = [
                {"role": "system", "content": self.system_prompt}
            ] + self.conversation_history
    
            try:
                response = self.client.chat.completions.create(
                    model=self.model,
                    messages=messages,
                    tools=self.get_available_tools(),
                    tool_choice="auto",
                    temperature=0.4,
                    max_tokens=700,
                )
                assistant_message = response.choices[0].message
    
                # Allow a few tool rounds so one user turn can expand into several LLM calls.
                for _ in range(4):
                    if not assistant_message.tool_calls:
                        break
    
                    self.conversation_history.append(
                        {
                            "role": "assistant",
                            "content": assistant_message.content,
                            "tool_calls": [
                                {
                                    "id": tool_call.id,
                                    "type": "function",
                                    "function": {
                                        "name": tool_call.function.name,
                                        "arguments": tool_call.function.arguments,
                                    },
                                }
                                for tool_call in assistant_message.tool_calls
                            ],
                        }
                    )
    
                    for tool_call in assistant_message.tool_calls:
                        tool_result = self.execute_tool_call(tool_call)
                        print(f"  [tool] {tool_call.function.name} -> {tool_result}")
                        self.conversation_history.append(
                            {
                                "role": "tool",
                                "tool_call_id": tool_call.id,
                                "content": tool_result,
                            }
                        )
    
                    messages = [
                        {"role": "system", "content": self.system_prompt}
                    ] + self.conversation_history
                    response = self.client.chat.completions.create(
                        model=self.model,
                        messages=messages,
                        tools=self.get_available_tools(),
                        tool_choice="auto",
                        temperature=0.4,
                        max_tokens=700,
                    )
                    assistant_message = response.choices[0].message
    
                response_content = assistant_message.content or ""
                self.conversation_history.append(
                    {"role": "assistant", "content": response_content}
                )
                return response_content
    
            except Exception as exc:  # noqa: BLE001 - surface proxy/LLM errors to the chat
                return f"I apologize, but I'm experiencing technical difficulties. Error: {exc}"
    
        def reset_conversation(self) -> None:
            self.conversation_history = []
    
    
    DEMO_TURNS = [
        "Hi! I need express shipping info for an order I'm about to place.",
        "Also, what is the status of my existing order ORD-10482?",
        "The item arrived damaged. What is your return policy and how do I start a return?",
        "Please process a refund for order ORD-10482 because the item arrived damaged.",
        "One last thing: how long is the product warranty, and how do I reset my account password?",
    ]
    
    
    def run_demo(agent: CustomerSupportAgent) -> int:
        print("Customer Support Agent (LiteLLM proxy demo)")
        print("=" * 60)
        print(f"Proxy model: {agent.model}")
        print(f"Turns: {len(DEMO_TURNS)}\n")
    
        for idx, user_input in enumerate(DEMO_TURNS, start=1):
            print(f"[{idx}/{len(DEMO_TURNS)}] You: {user_input}")
            response = agent.chat(user_input)
            print(f"Agent: {response}\n")
    
        print("Demo finished.")
        return 0
    
    
    def run_interactive(agent: CustomerSupportAgent) -> int:
        print("Customer Support Agent")
        print("=" * 50)
        print("Type 'quit' to exit or 'reset' to start a new conversation.\n")
    
        while True:
            try:
                user_input = input("\nYou: ").strip()
            except (EOFError, KeyboardInterrupt):
                print(
                    "\nAgent: Thank you for contacting customer support. Have a great day!"
                )
                return 0
    
            if user_input.lower() in {"quit", "exit", "bye"}:
                print(
                    "\nAgent: Thank you for contacting customer support. Have a great day!"
                )
                return 0
            if user_input.lower() == "reset":
                agent.reset_conversation()
                print("\nAgent: Conversation reset. How can I help you today?")
                continue
            if not user_input:
                continue
    
            print(f"\nAgent: {agent.chat(user_input)}")
    
    
    def build_agent_from_env() -> CustomerSupportAgent:
        return CustomerSupportAgent(
            base_url=os.getenv("LITELLM_PROXY_BASE", DEFAULT_PROXY_BASE),
            api_key=os.getenv("LITELLM_PROXY_KEY", DEFAULT_PROXY_KEY),
            model=os.getenv("LITELLM_MODEL", DEFAULT_MODEL),
        )
    
    
    def main(argv: list[str] | None = None) -> int:
        parser = argparse.ArgumentParser(description=__doc__)
        parser.add_argument(
            "--interactive",
            action="store_true",
            help="Run an interactive REPL instead of the multi-step demo",
        )
        args = parser.parse_args(argv)
    
        agent = build_agent_from_env()
        if args.interactive:
            return run_interactive(agent)
        return run_demo(agent)
    
    
    if __name__ == "__main__":
        sys.exit(main())
    ```

    {% endcut %}

    Through LiteLLM Proxy, the application sends five customer messages to the model one by one: a question about delivery, an order status check, a question on return policy, a refund request, and a final message with two questions, one regarding warranty and password reset.

    To prepare a response, the model can request a tool call: a local knowledge base search, an order status check, or a refund transaction. The application processes this tool call and sends the results back to the model through the proxy in the next request. Order checks and refunds are simulated in the app code.

    The number of requests depends on which tools the model calls. For example, during an order status check, the first request allows the model to select the tool, while the second one returns the tool results so the model can generate a response for the customer. LiteLLM Proxy generates a unique trace for each request and routes it to Monium Traces. The application itself does not send any telemetry data.

1. Run the test app:

    ```bash
    python3 agent_example.py
    ```

    Result:

    ```text
    Customer Support Agent (LiteLLM proxy demo)
    ============================================================
    Proxy model: deepseek-v4-flash
    Turns: 5
    ...
    [1/5] You: Hi! I need express shipping info for an order I'm about to place.
      [tool] search_knowledge_base -> Express shipping takes 2-3 business days and costs $12.99
    Agent: Great news! Here's the express shipping information:
    ...
    ...
    Demo finished.
    ```

### Find and view traces in Monium {#check-traces}

To find the traces generated by the test application:

{% list tabs group=instructions %}

- Monium UI {#console}

  1. [Navigate](https://monium.yandex.cloud) to Monium.
  1. In the left-hand panel, select ![cube](../../_assets/console-icons/cube.svg) **LLM Observability**.
  1. Use the timeline selector at the top of the screen to choose the period when you ran the test application.
  1. Click the ![code](../../_assets/console-icons/code.svg) icon in the top-right corner of the query bar and enter the following filter:

      ```text
      {project = "folder__<folder_ID>", service = "litellm-proxy"}
      ```

      Where `<folder_id>` is the [ID of the folder](../../resource-manager/operations/folder/get-id.md) where you [created](#create-api-keys) the service accounts.
  1. Click **Run query**.

{% endlist %}

The search results will display the traces generated by your test application. To view all operations within a trace, click its row and select **Show full trace**.

The expanded trace will display these spans:

* `POST /v1/chat/completions`: HTTP request to LiteLLM Proxy.
* `POST /v1/chat/completions`: LiteLLM Proxy access key verification.
* `chat gpt://<folder_ID>/deepseek-v4-flash`: Model call. This span contains the model name, input and output token counts, and message text.

If the model requests tool calls, the model span logs their names (`search_knowledge_base`, `get_order_status`, or `process_refund`) and arguments.

The application stores the conversation history and appends it to each new request, causing the context size to increase in subsequent traces.

For more information on viewing traces, see [Viewing and analyzing LLM traces](../../monium/traces/llm/traces.md).


### Troubleshooting {#troubleshooting}

#|
|| **Issue** | **Solution** ||
|| Connection error (`Error: Connection error`) |
* During the final step of the [LiteLLM Proxy deployment](#deploy-litellm-proxy), wait for the `proxy is up` message.
* Make sure VM port `4000` is available.
||
|| Yandex Cloud AI Studio returns HTTP code `401` or `403` |
* Check that the `YANDEX_AI_STUDIO_API_KEY` variable contains only the key, without `Bearer`, `Api-Key`, or any extra characters.
* Check `config.yaml` configurations:

    * `model` (the model ID in LiteLLM): `openai/gpt://<folder_ID>/deepseek-v4-flash`
    * `api_base` (address): `https://ai.api.cloud.yandex.net/v1`
||
|| The test application runs successfully, but traces do not appear in Monium |
Make sure that:

* The `MONIUM_PROJECT` value has this format: `folder__<folder_ID>` (with two underscores).
* The `Authorization` header in `docker-compose.yaml` contains a literal space between `Api-Key` and the key (not `%20`).
* The `litellm-monium-sa` service account holds the `monium.traces.writer` role for the folder defined in `MONIUM_PROJECT`.
* The service account API key scope is set to `yc.monium.traces.write`.
* The filter in the Monium UI is [set](#check-traces) to `{project = "folder__<folder_id>", service = "litellm-proxy"}` and the selected time range covers the exact period when you ran the test application.
||
|| No container network connection (`Connection reset`) |
Run the `sysctl net.ipv4.ip_forward` command.

If it returns `net.ipv4.ip_forward = 0`, keep the `network_mode: host` line in `docker-compose.yaml`.

After you make the changes, run `docker compose up -d --force-recreate`.
||
|| Spans do not contain message text | Check that the `docker-compose.yaml` file includes the `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only` environment variable.

After you make the changes, run `docker compose up -d --force-recreate`. ||
|#

## How to delete the resources you created {#clear-out}

If you deployed this solution for testing purposes only, you can delete the resources after completing this tutorial to stop paying for them:

1. [Delete the LiteLLM Proxy VM](../../compute/operations/vm-control/vm-delete.md).
1. [Delete the disk](../../compute/operations/disk-control/delete.md) if it was not deleted along with the VM.
1. [Delete the static public IP address](../../vpc/operations/address-delete.md) if you reserved one for the VM.

Delete the `litellm-ai-studio-sa` and `litellm-monium-sa` service accounts if you no longer need them.


[*key_scope_definition]: Scope is an API key parameter that restricts the key's use to specific operations in addition to the service account's access permissions. For details, see [API keys with scope and validity limits](../../iam/concepts/authorization/api-key.md#scoped-api-keys).

[*span_definition]: A _span_ is a basic distributed tracing element representing a single operation in your system, e.g., HTTP request, database call, or function execution. For more information, see [Span](../../monium/traces/concepts.md#span).