How to Continuously Improve Your LangGraph Multi-Agent System

How to Continuously Improve Your LangGraph Multi-Agent System

Pratik Bhavsar

Evals & Leaderboards @ Galileo Labs

Erin Mikail Staples

Senior Developer Experience Engineer

Jim Bennett

Principal Developer Advocate

We've all been trapped in a chatbot loop that keeps asking, "I'm sorry, I didn't understand that. Can you rephrase?" Or worse, a bot that cheerfully claims it can help with everything, then fails at everything. You ask about a billing error, but it suggests restarting your router.

Most chatbots attempt to be generalists, but they often end up being proficient in nothing. Real customer service works differently. When you call your telecom provider, you get routed to specialists who actually understand your specific problem.

That's exactly what we're building today! A multi-agent system for ConnectTel that creates delightful customer experiences through intelligent routing to specialized agents. We'll use LangGraph to build it and Galileo for real-time agent metrics, failure insights, and performance tracking so you can continuously improve what's actually working.

Welcome to the fifth installment of the multi-agent series. Don't miss out on other pieces in the Mastering Agent series.

Agent Architecture with Observability

Our system employs a supervisor pattern, where a main coordinator routes customer queries to specialized agents based on the customer's actual needs. We will leverage Chainlit for the conversational UI, Pinecone for vector-based knowledge retrieval, and GPT-4.1 as the underlying LLM.

The beauty of this design is that each agent can be developed, tested, and improved independently. When billing logic changes, you don't touch the technical support code. When you need to add a new capability, you add a new agent without rewriting everything.

However, we don't just want to build the agent; we also want to enable continuous improvement. This will require insights into the failures that happen when our users ask questions we never anticipated and expose edge cases. That is why we are building observability into our system from day one.

Here is how our continuous improvement cycle looks:

Monitor: Track every agent decision, tool call, and routing choice. You need to see not just what agents answer, but how they arrive at those answers. Which agent handled the query? What tools did they use? How long did each step take?

Debug: When things go wrong (and they will), trace through the entire chain. Did the supervisor route to the wrong agent? Did the tool return unexpected data? Did the agent misinterpret the response? Real production failures teach you more than any synthetic test.

Improve: Make targeted fixes based on actual data. Perhaps your billing agent requires more effective prompts for international charges. Technical support may need a timeout for slow diagnostics. The improvements can be measured and you'll know immediately if it worked.

This isn't a one-time optimization. It's an ongoing practice that separates production-ready systems from demos. The architecture we're about to build supports this naturally, with clear separation between agents and comprehensive tracking at every step.

Quick Setup Guide

Let's get this running on your machine. You'll need Python 3.9+, an OpenAI API key, Pinecone and Galileo accounts for the full experience. The complete source code is available in the repository, ready for you to adapt to your own use case.

Installation Steps

First, clone and navigate to the project:

git clone https://github.com/rungalileo/sdk-examples

git checkout feature/langgraph-telecom-agent

cd sdk-examples/python/agent/langgraph-telecom-agent

Configure your environment variables:

cp .env.example .env

a# Edit .env with your keys:

OPENAI_API_KEY

PINECONE_API_KEY

GALILEO_API_KEY

GALILEO_PROJECT="Multi-Agent Telecom Chatbot"

GALILEO_LOG_STREAM="prod"

Install dependencies using uv (recommended) or pip:

# Using uv

uv sync --dev

# Or using pip

pip install -e .

In order for agent to act in line with the company guidelines and product, they need to be made aware of the right context. For this, we create a document with the guidelines.

This is the snippet from the troubleshooting guide, which will be chunked and indexed to be used by the Technical Support agent using retrieval.

Network Troubleshooting Guide

Common Network Issues and Solutions

No Signal / No Service

Symptoms

Solutions

  1. Toggle Airplane Mode
  1. Restart Device
  1. Check SIM Card
  1. Reset Network Settings
  1. Update Carrier Settings

Slow Data Speeds

Symptoms

Our Plan Advisor and Technical Support agent need the docs to be indexed in Pinecone which is our vector DB.

python ./scripts/setup_pinecone.py

or

uv run ./scripts/setup_pinecone.py

Once the agent is defined, the Galileo callback can be integrated with Langgraph in just a few lines of code.

from galileo import galileo_context

from galileo.handlers.langchain import GalileoAsyncCallback

# Initialize Galileo context first

galileo_context.init()

# Start Galileo session with unique session name

galileo_context.start_session(name=session_name, external_id=cl.context.session.id)

# Create the callback. This needs to be created in the same thread as the session
# so that it uses the same session context.
galileo_callback = GalileoAsyncCallback()

# Pass the callback to the agent instance
supervisor_agent.a.stream(input=messages, stream_mode="updates", config=RunnableConfig(callbacks=galileo_callback, **config))

Launch the Chainlit interface with our Langgraph application:

chainlit run app.py -w

or

uv run chainlit run app.py -w

The application will be available at http://localhost:8000. You'll see a chat interface where you can start asking questions about bills, technical issues, or plan recommendations.

Understanding the Core Components

The Supervisor Agent

The supervisor is the brain of our operation. It analyzes incoming queries and routes them to the right specialist. Here's how it works:

def create_supervisor_agent():
    """
    Create a supervisor agent that manages all the agents in the ConnectTel telecom application.
    """
    checkpointer = MemorySaver()
    telecom_supervisor_agent = create_supervisor(
        model=ChatOpenAI(
            model=os.environ["MODEL_NAME_SUPERVISOR"], name="Supervisor"
        ),
        agents=[
            billing_account_agent,
            technical_support_agent,
            plan_advisor_agent,
        ],
        prompt="""
        You are a supervisor managing a team of specialized telecom service agents at ConnectTel.
        Route customer queries to the appropriate agent based on their needs:
        - Billing Account Agent: Bill inquiries, payment issues, usage tracking
        - Technical Support Agent: Device troubleshooting, connectivity issues
        - Plan Advisor Agent: Plan recommendations, upgrades, comparing plans
        Guidelines:
        - Route queries to the most appropriate specialist agent
        - For complex issues spanning multiple areas, coordinate between agents
        - Be helpful and empathetic to customer concerns
        """,
        add_handoff_back_messages=True,
        output_mode="full_history",
        supervisor_name="connecttel-supervisor-agent",
    ).compile(checkpointer=checkpointer)
    return telecom_supervisor_agent

The supervisor uses LangGraph's built-in create_supervisor function which handles the complex orchestration logic. This includes message routing based on query analysis, state management across agent interactions, and memory persistence for conversation continuity.

Notice the add_handoff_back_messages=True parameter. This allows agents to return control to the supervisor when they need help from another agent. It's like a customer service rep saying "Let me transfer you to billing" but happening seamlessly in the background.

Building Specialized Agents

Each specialized agent handles specific types of inquiries. They're built using LangGraph's create_react_agent pattern, which implements reasoning and action cycles. Let's look at the Billing Account Agent:

from langchain_openai import ChatOpenAI

from langgraph.prebuilt import create_react_agent

from ..tools.billing_tool import BillingTool

def create_billing_account_agent() -> CompiledGraph:
    """
    Create an agent that handles billing inquiries, usage tracking,
    and account management.
    """
    # Create the billing tool instance
    billing_tool = BillingTool()
    # Create a ReAct agent with specialized prompt
    agent = create_react_agent(
        model=ChatOpenAI(
            model=os.environ["MODEL_NAME_WORKER"],
            name="Billing Account Agent"
        ),
        tools=[billing_tool],
        prompt="""
        You are a Billing and Account specialist for ConnectTel.
        You help customers with billing inquiries, usage tracking,
        plan details, and payment issues.
        Key responsibilities:
        - Check account balances and payment due dates
        - Track data, voice, and text usage
        - Explain charges and fees clearly
        - Suggest plan optimizations based on usage
        - Process payment-related inquiries
        - Review billing history
        When discussing charges:
        - Break down costs clearly
        - Highlight any unusual charges
        - Suggest ways to reduce bills if usage patterns show opportunity
        - Always mention auto-pay discounts if not enrolled
        Be empathetic about high bills and offer solutions.
        """,
        name="billing-account-agent",
    )
    return agent

The ReAct pattern means the agent reasons about when to use tools versus when to respond directly. The specialized prompt defines not just what the agent knows, but how it should behave. The empathy instructions are needed for scenarios where people might be frustrated about bills. But how does we measure the empathy? Galileo provides out of the box sentiment metric for measuring the tone.

Creating Effective Tools

Tools are where your agents interact with the real world. They're the bridge between conversation and action. We create a mock billing tool that responds as per the arguments query_type and customer_id. The responses are randomly generated and act like real responses for different users.

from typing import Optional

from langchain.tools import BaseTool

from datetime import datetime, timedelta

import random

class BillingTool(BaseTool):
    """Tool for retrieving customer billing and usage information."
    name: str = "billing_account"
    description: str = "Check account balance, data usage, plan details, and billing history"

def _run(self, customer_id: Optional[str] = None, query_type: str = "summary") -> str:
        """
        Get billing and usage information.
        Args:
            customer_id: Customer account number (uses default if not provided)
            query_type: Type of query (summary, usage, plan, history)
        """
        # Mock customer data - in production, this would query real databases
        customer = {
            "name": "John Doe",
            "account": customer_id or "ACC-2024-789456",
            "plan": "Premium Unlimited 5G",
            "monthly_charge": 85.00,
            "data_used": random.uniform(20,80),
            "data_limit": "Unlimited",
            "due_date": (datetime.now() + timedelta(days=15)).strftime("%Y-%m-%d")
        }

if query_type == "usage":
            return f"""
            Usage Summary for {customer['name']}:
            - Data: {customer['data_used']:.1f}GB used ({customer['data_limit']})
            - Minutes: {random.randint(300,800)} (Unlimited)
            - Texts: {random.randint(500,2000)} (Unlimited)
            - Average daily: {customer['data_used'] / 15:.2f}GB
            """
        elif query_type == "plan":
            return f"""
            Current Plan: {customer['plan']}
            - Monthly Cost: ${customer['monthly_charge']:.2f}
            - Data: {customer['data_limit']}
            - Talk & Text: Unlimited
            - 5G Access: Included
            Available Upgrades:
            - Business Elite ($120/month)
            - International Plus ($95/month)
            """
        elif query_type == "history":
            history = []
            for i in range(3):
                date = (datetime.now() - timedelta(days=30*(i+1))).strftime("%Y-%m-%d")
                amount = customer['monthly_charge'] + random.uniform(-5,15)
                history.append(f"- {date}: ${amount:.2f} (Paid)")
            return f"""
            Billing History:
            {chr(10).join(history)}
            Auto-pay: Enabled
            """

# Default summary response
        return f"""
        Account Summary for {customer['name']}:
        - Account: {customer['account']}
        - Plan: {customer['plan']}
        - Amount Due: ${customer['monthly_charge']:.2f}
        - Due Date: {customer['due_date']}
        - Data Used: {customer['data_used']:.1f}GB ({customer['data_limit']})
        """

The tool uses mock data for demonstration purposes, but the underlying structure is ready. The key is the clear description field as it helps agents understand when to use the tool. The flexible parameters support various query types, and the formatted output is easily readable by humans.

How It Works in Practice

Example 1: Multi-Domain Query

When a user says, "My internet has been slow for the past week and I'm wondering if I'm being throttled because of my data usage. Can you check my current usage and bill?" here's what happens:

The supervisor first routes to the Billing Agent, which checks data usage and plan limits. Then it routes to the Technical Support Agent, which checks for connectivity issues. Finally, the supervisor combines both responses into a coherent answer.

Here is how it looks in the Galileo console, where we can see the traces, spans and metrics of the session.

Galileo automatically captures agent routing decisions, tool invocations with inputs and outputs, all LLM interactions and response times for each step.

Debugging Agents with Metrics

The real value of Galileo comes from understanding how your agents actually perform in production. Let's look at the key metrics.

Action Completion

Action Completion indicates whether your agents are actually assisting or merely responding. It's the difference between an agent that says "I'll check your bill" and one that actually retrieves and explains the charges.

Here's what makes an action complete in our telecom system. The agent must provide a complete answer (not just acknowledge the question), confirm successful actions when handling requests, maintain factual accuracy throughout, address every aspect of the user's query, avoid contradicting tool outputs, and properly summarize all relevant information from tools.

A score below 80% usually indicates your agents are either not using their tools properly, providing generic responses instead of specific answers, or failing to follow through on multi-step processes. The beauty of tracking Action Completion is that it directly correlates with customer satisfaction.

Tool Selection Quality

Tool Selection Quality reveals whether your agents are choosing the right tools with the right parameters. It's like having a toolbox—knowing when to use a hammer versus a screwdriver makes all the difference.

If a customer asks "I want to upgrade my plan.", the Plan Advisor agent needs to select the right sequence of tools: first the billing tool to check current usage, then the plan comparison tool to find suitable upgrades, and finally the upgrade tool to process the change. If it skips straight to the upgrade tool without checking usage, it might recommend a plan that doesn't fit the customer's needs.

The metric evaluates two critical aspects: did the agent choose the correct tool, and did it use the correct parameters? In our system, we often see agents choose the right tool but set the parameters incorrectly.

A score below 80% is concerning. It means your agents are winging it instead of using their capabilities. The most common issue is that agents are either too eager or too reluctant to use tools. Too eager means calling tools for questions they can answer directly ("What are your business hours?"). Too reluctant means answering from general knowledge when they should check specific data.

Latency Breakdown

The latency trace graph reveals the execution timeline and performance characteristics of our system. Understanding where time is spent helps optimize the user experience by finding the bottlenecks. Galileo’s latency trace graph provides several valuable insights into system behavior and performance:

1. Agent Coordination Patterns The trace reveals how agents collaborate in real-time. We can see the Supervisor consistently orchestrating the conversation while specialized agents (Billing, Technical Support, Plan Advisor) activate only when their expertise is needed. This validates the efficiency of our multi-agent architecture and confirms that agents aren't running unnecessarily.

2. Bottleneck Identification By examining the duration and frequency of each operation, we can pinpoint performance bottlenecks. The call_model operations exhibit the most significant latency contributions, indicating that LLM inference is the primary factor affecting response time, rather than transfer logic or retrieval operations.

3. Decision Flow Understanding The should_continue markers illustrate how the system makes routing decisions throughout the conversation. Multiple checks ensure the conversation flows appropriately between agents, and we can trace exactly when and why transfers occur.

4. Retrieval Timing The sparse appearance of pinecone_retrieval operations shows that knowledge base queries are triggered selectively rather than on every turn, indicating intelligent retrieval logic that balances accuracy with performance.

5. System Responsiveness The overall timeline demonstrates that despite multiple agent handoffs and model calls, the system maintains reasonable end-to-end latency. This validates that our multi-agent approach doesn't introduce prohibitive overhead compared to a single-agent system.

Making Observability Actionable

Performance Benchmarks for Production

Once your multi-agent system moves from development to production, establishing clear performance targets becomes critical for maintaining user satisfaction and operational excellence. These benchmarks serve multiple purposes: they provide guardrails for deployment decisions, enable objective evaluation of system changes, and help teams identify when performance degradation requires immediate attention.

The metrics below are derived from real-world production data across multiple deployments and correlate directly with user satisfaction scores. Systems that consistently hit "Excellent" targets see significantly higher user retention and completion rates, while those falling into "Needs Improvement" typically generate support tickets and abandoned sessions.

Metric Excellent Good Needs Improvement
Action Completion > 95% 85-95% < 80%
Tool Selection Quality > 90% 85-90% < 85%
Avg Response Time < 2s 2-4s > 4s
Supervisor Routing Accuracy > 95% 90-95% < 90%

Continuous Improvement with Insights

The moment your multi-agent system hits production, it encounters edge cases you never tested. A user asks "Why is my bill higher this month?" and gets transferred between Billing and Technical Support three times. Your logs show 47 LLM calls, 12 tool invocations, and 8 agent handoffs—but which one failed?

Traditional debugging forces you to:

The Insights Engine automates this investigation. Instead of spending hours hunting through traces, Galileo analyzes your entire log stream and surfaces exactly what's broken.

Each insight includes:

Wrapping Up

Our agent system demonstrates how modern AI orchestration creates sophisticated customer service solutions. By combining LangGraph's agent management with monitoring, you build systems that are both powerful and observable.

The architecture's modularity makes it particularly suitable for enterprises where different teams manage different aspects of customer service. Yes, there's added complexity and some latency overhead. But for production systems handling diverse queries, the benefits of specialization, scalability, and comprehensive monitoring can make it worthwhile.

The complete source code is available in the repository, ready for you to adapt to your own use case. Start with two agents, add monitoring from day one, and scale based on what you learn from real usage. That's how you build AI systems that actually work in production.

You can try Galileo for free to optimize your Langgraph with ease.