Table of Contents

1. Mem0 Basics – Adding, updating, and searching memories

2. LangGraph Integration – Wiring Mem0 into a LangGraph agent

3. Vector DB Setup – Swapping the default SQLite store for Qdrant

4. Cloud Usage – Using the Mem0 Cloud Platform for scalable memory management

Why do AI agents need memory?

When an LLM‑powered agent starts a brand‑new conversation it has no context about who it’s talking to or what happened in earlier sessions. Relying on the raw chat history works only inside a single session and quickly bloats your prompt window.
Long‑term memory lets the agent:

• Remember user‑level facts (name, preferences, past actions) across sessions

• Personalise responses without re‑asking the same questions

• Stay efficient by storing distilled facts instead of the entire transcript

Chat history vs memory

1. Mem0 Basics – Adding, Updating & Searching Memories

Quick setup

from mem0 import Memory
memory = Memory.from_config({"history_db_path": "history.db"})  # local SQLite file

Why the explicit config? Mem0 defaults to a read‑only temp database, so writes will fail. Pointing it to history.db (or any path you prefer) gives the library a place to persist memories. You can extend the same config dict to

• Override the LLM (provider, model, temperature, etc.)

• Plug in a vector store for semantic search (we’ll wire up Qdrant in Section 3).

Example – switching to GPT‑4.1‑mini:

config = {
    "history_db_path": "history.db",
    "llm": {
        "provider": "openai",
        "config": {
            "model": "gpt-4.1-mini",
            "temperature": 0.2,
            "max_tokens": 2000
        }
    }
}
memory = Memory.from_config(config)

Add your first memories

memory.add([
    {"role": "user", "content": "Hi, I'm Pradip Nichite. I run FutureSmart AI, where we build custom AI solutions."}
], user_id="pradip")

memory.add([
    {"role": "user", "content": "I love building RAG and AI Agent solutions that actually work in production."}
], user_id="pradip", metadata={"category": "preferences"})

Sample response:

{'results': [{'id': '5408e326‑b26b‑4737‑a404‑299887b8d597',
  'memory': 'Loves building RAG and AI Agent solutions that work in production',
  'event': 'ADD'}]}

Mem0 distills each raw chat message into a concise fact so retrieval stays lightweight.

Search

related = memory.search("who am i", user_id="pradip")
related

Full output:

{'results': [{'id': '647935d5-f913-496d-96e3-2233d7459f38',
   'memory': 'Name is Pradip Nichite',
   'hash': 'fa942a6331bb89da286d4a9e296d1008',
   'metadata': None,
   'score': 0.2294486506181006,
   'created_at': '2025-07-12T10:58:49.132915-07:00',
   'updated_at': None,
   'user_id': 'pradip'},
  {'id': 'c763c19a-7e9f-4180-8c82-012f4da5f637',
   'memory': 'Runs FutureSmart AI',
   'hash': '68a143a88a3e67ae9ebfb9575bcf49a7',
   'metadata': None,
   'score': 0.1551292009096673,
   'created_at': '2025-07-12T10:58:49.158843-07:00',
   'updated_at': None,
   'user_id': 'pradip'},
  {'id': '5408e326-b26b-4737-a404-299887b8d597',
   'memory': 'Loves building RAG and AI Agent solutions that work in production',
......
   'user_id': 'pradip'},
  {'id': '1baa6793-507f-46b1-8e01-b90dfa1e73b6',
   'memory': 'Builds custom AI solutions',
.......
   'user_id': 'pradip'}]}

score is cosine similarity—higher means closer semantic match.

Get all memories for a user

all_memories = memory.get_all(user_id="pradip")

Returns the full list (same schema as search, without scores).

Retrieve a single memory

mem_id = "1baa6793-507f-46b1-8e01-b90dfa1e73b6"
memory.get(mem_id)

Full output:

{'id': '1baa6793-507f-46b1-8e01-b90dfa1e73b6',
 'memory': 'Builds custom Gen AI solutions',
 'hash': '502bdf5771e4ef9a812453b51870f0b2',
 'metadata': None,
 'score': None,
 'created_at': '2025-07-12T10:58:49.182159-07:00',
 'updated_at': '2025-07-12T11:02:34.594521-07:00',
 'user_id': 'pradip'}
{'id': '1baa...73b6',
 'memory': 'Builds custom Gen AI solutions',
 'created_at': ..., 'updated_at': ...}

Update a memory

memory.update(memory_id=mem_id, data="Builds custom Gen AI solutions")
# → {'message': 'Memory updated successfully!'}

View change history

history = memory.history(memory_id=mem_id)

Full output:

[{'id': 'e7242249-430a-4bf2-b4df-ca0e4b99e69a',
  'memory_id': '1baa6793-507f-46b1-8e01-b90dfa1e73b6',
  'old_memory': None,
  'new_memory': 'Builds custom AI solutions',
  'event': 'ADD',
  'created_at': '2025-07-12T10:58:49.182159-07:00',
  'updated_at': None,
  'is_deleted': False,
  'actor_id': None,
  'role': None},
 {'id': '9d2e9706-b0c2-480b-b37a-07bb6143767d',
  'memory_id': '1baa6793-507f-46b1-8e01-b90dfa1e73b6',
  'old_memory': 'Builds custom AI solutions',
  'new_memory': 'Builds custom Gen AI solutions',
  'event': 'UPDATE',
  'created_at': '2025-07-12T10:58:49.182159-07:00',
  'updated_at': '2025-07-12T11:02:34.594521-07:00',
  'is_deleted': False,
  'actor_id': None,
  'role': None}]

Each entry records the old & new value plus timestamp—handy for auditing:

[{'event': 'ADD',    'old_memory': None,                     'new_memory': 'Builds custom AI solutions'},
 {'event': 'UPDATE', 'old_memory': 'Builds custom AI solutions',
  'new_memory': 'Builds custom Gen AI solutions'}]

That wraps up the core CRUD API.

2. LangGraph Integration – Wiring Mem0 into an Agent

New to LangGraph? Watch my YouTube walkthrough that covers LangGraph basics all the way to advanced patterns.

Below we build the simplest possible LangGraph agent that:

1. Accepts user messages.

2. Retrieves relevant memories from Mem0.

3. Injects them into the system prompt for personalised replies.

4. Writes the new interaction back to Mem0.

a) Define the shared state

from typing import Annotated, TypedDict
from langgraph.graph.message import add_messages
from langchain_core.messages import BaseMessage

class State(TypedDict):
    """Conversation state passed between nodes"""
    messages: Annotated[list[BaseMessage], add_messages]  # chat history for this request
    mem0_user_id: str                                     # maps to Mem0 user record

b) Init the LLM

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4.1-mini", temperature=0.7)

c) Create the chatbot node

from langchain_core.messages import HumanMessage, AIMessage, SystemMessage

def chatbot(state: State):
    global memory  # re‑use the Mem0 instance from Section 1
    msgs = state["messages"]
    uid = state["mem0_user_id"]

    # 1️⃣ Retrieve memories relevant to the latest user msg
    mems = memory.search(msgs[-1].content, user_id=uid)
    print(f"Retrieved Memories: {mems}")

    # Build context string
    if mems["results"]:
        context = "
".join(f"- {m['memory']}" for m in mems["results"])
    else:
        context = "No relevant information found."

    system = SystemMessage(content=f"""You are a helpful assistant. Use the provided context to personalise your responses.
Relevant information from previous conversations:
{context}""")

    # 2️⃣ Invoke the LLM
    response = llm.invoke([system] + msgs)

    # 3️⃣ Persist the new turn
    memory.add([
        {"role": "user", "content": msgs[-1].content},
        {"role": "assistant", "content": response.content}
    ], user_id=uid)

    return {"messages": [response]}

How the node uses Mem0

1. Search: For every incoming user message, we call memory.search() with the text and user_id. This performs a vector‑similarity lookup and returns any facts previously stored about the user.

2. Prompt injection: Those facts are concatenated into a bullet list (context) and inserted into a system prompt so the LLM can personalise its reply.

3. Add: After the LLM responds, we persist both the latest user message and the assistant reply via memory.add(). Mem0 distils them into new memories ready for the next turn.

d) Build & compile the graph

from langgraph.graph import StateGraph, START, END

graph_builder = StateGraph(State)

graph_builder.add_node("chatbot", chatbot)

graph_builder.add_edge(START, "chatbot")

graph_builder.add_edge("chatbot", END)

graph = graph_builder.compile()
print("Graph compiled successfully ✅")

e) Command‑line loop for quick testing

from langgraph_core.messages import HumanMessage

def run_conversation(user_input: str, mem0_user_id: str):
    state = {"messages": [HumanMessage(content=user_input)], "mem0_user_id": mem0_user_id}
    result = graph.invoke(state)
    print("🤖", result["messages"][-1].content)

if __name__ == "__main__":
    uid = "customer_pradip"
    while True:
        inp = input("You: ")
        if inp.lower() in {"quit", "exit", "bye"}:
            break
        run_conversation(inp, uid)

Run it, send two or three messages, then restart the script and ask “who am I?”—you’ll see the agent recall facts from the earlier run thanks to Mem0’s long‑term store.

3. Vector DB Setup – Configuring Mem0 with Qdrant

SQLite works for quick tests, but once memories grow you’ll want a proper vector store. Qdrant Cloud offers a generous free tier and plugs straight into Mem0.

a) Spin up / locate a Qdrant Cloud cluster

Grab the cluster URL and create an API key from the Qdrant dashboard.

# Install the Python client
!pip -q install qdrant_client

b) Verify connectivity (optional)

from qdrant_client import QdrantClient

qdrant = QdrantClient(
    url="https://<cluster-id>.<region>.aws.cloud.qdrant.io:6333",
    api_key=userdata.get("Qdrant_API_KEY")
)
print(qdrant.get_collections())  # sanity‑check

c) Tell Mem0 to use Qdrant

collection_name = "mem0_yt"

config = {
    "vector_store": {
        "provider": "qdrant",
        "config": {
            "collection_name": collection_name,
            "host": "<cluster-host>",
            "port": 6333,
            "api_key": userdata.get("Qdrant_API_KEY")
        }
    }
}

memory = Memory.from_config(config)

d) One‑time payload index

Mem0 filters by user_id when searching, so Qdrant needs a keyword index on that field. If you skip this step you’ll get:

400 Bad Request – Index required but not found for "user_id" of type [keyword]

Create it once, then you’re good:

qdrant.create_payload_index(
    collection_name=collection_name,
    field_name="user_id",
    field_schema="keyword"
)

e) Insert and query as usual

messages = [
    {"role": "user", "content": "Hi, I'm Pradip Nichite. I run FutureSmart AI."},
    {"role": "user", "content": "I love building RAG and AI Agent solutions that work in production."}
]
memory.add(messages, user_id="pradip")

From here all CRUD and LangGraph logic stays exactly the same—only the storage layer has changed.

4. Cloud Usage – Using the Mem0 Cloud Platform

If you’d rather skip managing your own DBs, Mem0 offers a hosted platform with a clean UI to inspect and edit memories.

a) Authenticate

from mem0 import MemoryClient

client = MemoryClient(api_key=userdata.get("Mem0_API_KEY"))

b) Add messages (same schema as before)

messages = [
    {"role": "user",      "content": "Hi, I am Pradip. I am Founder of FutureSmart AI"},
    {"role": "assistant", "content": "Hi Pradip"}
]

client.add(messages, user_id="Pradip_Founder")

c) Inspect in the dashboard

You’ll see two distilled memories automatically extracted, complete with timestamps and editable fields.

The hosted store supports the same search/update/history API, so you can swap Memory for MemoryClient with minimal changes.

Watch the full walkthrough

Prefer video? I recorded a step‑by‑step YouTube demo that mirrors this blog, including live coding and UI tours – check it out here 👇

Need a Custom AI Solution?

At FutureSmart AI we specialise in designing and shipping production‑grade AI systems—RAG agents, document parsers, NL2SQL bots, multi‑agent workflows, and more.

→ See our case studies: https://futuresmart.ai/case-studies
→ Try the LangGraph‑powered FutureSmart Agent: https://agent.futuresmart.ai/
→ Get in touch: email us at contact@futuresmart.ai to discuss how we can build or fine‑tune an AI solution for your business.

With LangGraph, you can build stateful, agent-like flows that combine tools, memory, structured decision logic, and retrieval—all driven by LLMs.

In this blog, we’ll build up to a full LangGraph-based RAG Agent from scratch. We'll follow a practical path:

1. Start with basic LLM usage

2. Bind tools to the LLM

3. Use LangGraph to build stateful agents

4. Add memory, routing logic, and tool execution

5. Finally, combine all of it with document retrieval to create a RAG-powered agent

Each section mirrors what you’d build in a notebook, but with clear explanations to help you understand why each piece matters.

Let’s start with the first building block: invoking an LLM.

🧠 Step 1: Invoking a Language Model (LLM)

To begin, we’ll use ChatOpenAI from LangChain to invoke a language model. We’ll keep it simple:

from langchain_openai import ChatOpenAI

# Initialize the LLM
llm = ChatOpenAI(model="gpt-4.1-mini", temperature=0.7)

# Basic prompt
response = llm.invoke("What is artificial intelligence?")
print(response.content)

This returns a standard response from the LLM. But the real value comes when you treat the LLM like a conversation partner using message objects:

from langchain_core.messages import HumanMessage, SystemMessage

messages = [
    SystemMessage(content="You are a helpful AI assistant that explains complex topics simply."),
    HumanMessage(content="Explain machine learning in 2 sentences.")
]

response = llm.invoke(messages)
print(response.content)

Using SystemMessage and HumanMessage gives you more control over behavior and tone. It’s also how you’ll structure inputs later when building memory-enabled and multi-step agents.

Now that we can invoke an LLM in both simple and structured ways, we’re ready to start integrating tools.

🔧 Step 2: Extending LLMs with Tools

LLMs are powerful, but they can’t do math or fetch real-time information on their own. To make your LLM truly useful, you can bind it with external tools. Here’s how:

from langchain_core.tools import tool
from langchain_community.tools import DuckDuckGoSearchRun

@tool
def calculator(expression: str) -> str:
    """Calculate mathematical expressions. Use this for any math calculations."""
    try:
        result = eval(expression)
        return f"The result of {expression} is {result}"
    except Exception as e:
        return f"Error calculating {expression}: {str(e)}"

search_tool = DuckDuckGoSearchRun()

We now have two tools:

• calculator to perform basic arithmetic

• search_tool to fetch info from the web

To bind these tools to the LLM:

# Bind tools to the LLM
tools = [calculator, search_tool]
llm_with_tools = llm.bind_tools(tools)

Let’s test the LLM with tools:

response = llm_with_tools.invoke("What's 25 * 4 + 17?")
print(response.content)

However, when an LLM is tool-enabled, its response might include tool_calls instead of just plain text. To handle that:

def handle_tool_calls(response, tool_map):
    if not getattr(response, 'tool_calls', None):
        return

    for tool_call in response.tool_calls:
        tool_name = tool_call['name']
        args = tool_call['args']
        tool = tool_map.get(tool_name)
        if tool:
            result = tool.invoke(args)
            print(f"Tool result: {result}")

Then:

tool_map = {
    'calculator': calculator,
    'duckduckgo_search': search_tool,
}

def test_llm_tool(query):
    response = llm_with_tools.invoke(query)
    print(response.content)
    handle_tool_calls(response, tool_map)

# Run some queries
test_llm_tool("What's 25 * 4 + 17?")
test_llm_tool("Search for recent news about artificial intelligence")

With this setup, your LLM is now a tool-using agent.

Next, we’ll take this a step further by wiring everything into a LangGraph to make it stateful and multi-turn.

🧩 Step 3: Building a Basic LangGraph Chatbot

At its core, LangGraph lets you define a graph of nodes that process conversational state. Let’s begin with a minimal chatbot graph.

Define Chatbot State

from typing import Annotated, TypedDict
from langchain_core.messages import BaseMessage, HumanMessage, AIMessage
from langgraph.graph.message import add_messages

class State(TypedDict):
    messages: Annotated[list[BaseMessage], add_messages]

Here, we define a State object that will carry the conversation. The add_messages function ensures new messages are appended correctly.

Create the Chatbot Node

def chatbot_node(state: State) -> State:
    response = llm.invoke(state["messages"])
    return {"messages": [response]}

This node accepts messages and returns the updated state with the AI's response.

Build and Compile the Graph

from langgraph.graph import StateGraph, START, END

graph_builder = StateGraph(State)
graph_builder.add_node("chatbot", chatbot_node)
graph_builder.add_edge(START, "chatbot")
graph_builder.add_edge("chatbot", END)
graph = graph_builder.compile()

This sets up a simple one-node chatbot pipeline. You can now test it:

def test_chatbot(message: str):
    initial_state = {"messages": [HumanMessage(content=message)]}
    result = graph.invoke(initial_state)
    print("🤖 Assistant:", result["messages"][-1].content)

test_chatbot("Hello! My name is Pradip")
test_chatbot("Do you remember my name?")

You’ll notice it doesn’t remember past messages yet. That’s what we’ll fix in the next step—by adding memory.

🧠 Step 4: Adding Memory to the Chatbot

To make the chatbot remember previous conversations, we need to add a memory backend.
LangGraph provides MemorySaver for this purpose.

from langgraph.checkpoint.memory import MemorySaver

memory = MemorySaver()

# Compile the graph again with memory enabled
graph_with_memory = graph_builder.compile(checkpointer=memory)

We can now run the chatbot in a threaded manner, and it will retain context:

def chat_with_memory(message: str, thread_id: str):
    config = {"configurable": {"thread_id": thread_id}}
    initial_state = {"messages": [HumanMessage(content=message)]}
    result = graph_with_memory.invoke(initial_state, config)
    print("🤖 Assistant:", result["messages"][-1].content)

# Start a conversation
chat_with_memory("Hi, my name is Pradip", thread_id="thread-1")
chat_with_memory("What's my name?", thread_id="thread-1")

With memory in place, the assistant can now recall previous messages.
This forms the foundation for building multi-turn, context-aware agents.

Next, we’ll add more intelligence to the flow using routing and tools.

🛠️ Step 5 – LangGraph Agent with Tools

So far, our chatbot can talk (Step 3) and remember context (Step 4). Now we want it to recognise when a tool is needed and call it automatically.

At a high‑level we’ll add two new pieces:

1. chatbot node – decides whether it can answer directly or should call a tool.

2. tools node – actually runs the requested tool‑call and passes the result back.

The conversation state stays the same – a list of LangChain Message objects – so we just rename it to emphasise the agent role:

from typing import Annotated, TypedDict
from langgraph.graph.message import add_messages
from langchain_core.messages import BaseMessage

class AgentState(TypedDict):
    """State for our two‑node agent"""
    messages: Annotated[list[BaseMessage], add_messages]

1. Bind the LLM to our existing tools

llm = ChatOpenAI(model="gpt-4.1-mini", temperature=0.7)
llm_with_tools = llm.bind_tools(tools)  # `tools` already contains `calculator` and `search_tool`

Binding keeps the API exactly the same – we just swap llm for llm_with_tools when we need tool‑usage.

2. The chatbot node – decide answer vs. tool

from langchain_core.messages import HumanMessage, AIMessage

def chatbot_node(state: AgentState) -> AgentState:
    """Gatekeeper: answer directly or request a tool"""
    system_message = (
        "You are a helpful assistant.\n"
        "Use the `web_search` tool for real‑time facts and `calculator` for maths.\n"
        "Otherwise answer directly."
    )

    messages = [
        {"role": "system", "content": system_message},
        *state["messages"],
    ]

    response = llm_with_tools.invoke(messages)
    return {"messages": [response]}  # LangGraph merges this into the running state

Key idea: we embed the routing logic inside the prompt – the LLM decides whether tool calls are needed and, if so, emits a tool_calls entry in its JSON response.

3. The tools node – run any requested tool‑calls

Instead of re‑implementing the execution loop, we reuse the pre‑built ToolNode:

from langgraph.prebuilt import ToolNode

tool_node = ToolNode(tools)  # automatically dispatches and streams results back

4. Routing logic

We just need a small helper that checks whether the last message contains tool calls:

from typing import Literal

def should_continue(state: AgentState) -> Literal["tools", "end"]:
    last = state["messages"][-1]
    return "tools" if getattr(last, "tool_calls", None) else "end"

5. Wire it all together with StateGraph

from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver

workflow = StateGraph(AgentState)
workflow.add_node("chatbot", chatbot_node)
workflow.add_node("tools",   tool_node)

workflow.add_edge(START, "chatbot")
workflow.add_conditional_edges("chatbot", should_continue, {"tools": "tools", "end": END})
workflow.add_edge("tools", "chatbot")  # come back after tools run

app = workflow.compile(checkpointer=MemorySaver())

Why a loop back to chatbot? After a tool runs we want the LLM to integrate the tool output and craft the final answer – so the graph cycles once.

6. Quick manual test

def chat_with_agent(msg: str, thread_id="demo"):
    cfg = {"configurable": {"thread_id": thread_id}}
    state = {"messages": [HumanMessage(content=msg)]}
    result = app.invoke(state, cfg)
    print(result["messages"][-1].content)

chat_with_agent("What's 15% of 240?")
chat_with_agent("Search for recent news about artificial intelligence")

You should see the calculator and web_search tools being triggered automatically, followed by a neat, fully‑formed answer.

That’s a self‑routing, tool‑aware agent. In the next step we’ll plug a knowledge‑base retriever into the tool‑chain and teach the agent when to switch from web search to internal RAG – bringing us one step closer to a production‑ready assistant.

🔍 Step 6 – LangGraph RAG Agent

Goal: Give your agent up‑to‑date, domain‑specific knowledge so it can answer beyond the LLM’s training data.

We’ll layer retrieval, routing, and an optional web‑search fallback on top of the tool‑enabled agent from Step 5.

1️⃣ Index your documents once

# ── Build & persist a Chroma index ────────────────────────────────
from pathlib import Path
from langchain_community.document_loaders import PyPDFLoader, Docx2txtLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

SOURCE_DIR   = Path("docs")          # put your files here
INDEX_DIR    = Path("chroma_db_1")   # will be created if missing
EMBED_MODEL  = "text-embedding-3-small"

# Load docs (keep only pdf/docx for brevity)
docs = []
for f in SOURCE_DIR.glob("*.*"):
    if f.suffix == ".pdf":
        docs += PyPDFLoader(str(f)).load()
    elif f.suffix == ".docx":
        docs += Docx2txtLoader(str(f)).load()

# Split & embed
chunks     = RecursiveCharacterTextSplitter(chunk_size=1_000, chunk_overlap=200).split_documents(docs)
embeddings = OpenAIEmbeddings(model=EMBED_MODEL)

vectordb = Chroma.from_documents(
    documents         = chunks,
    embedding         = embeddings,
    persist_directory = str(INDEX_DIR),
    collection_name   = "kb_collection",
)
vectordb.persist()
print("✅ Index built →", INDEX_DIR.resolve())

Run this once; the agent will query the saved index at runtime.

2️⃣ Expose a Retriever as a LangChain Tool

retriever = vectordb.as_retriever(search_kwargs={"k": 2})

@tool
def rag_search_tool(query: str) -> str:
    """Search the knowledge‑base for relevant chunks"""
    results = retriever.invoke(query)
    return "

".join(d.page_content for d in results)

3️⃣ Optional fallback → real‑time web search

from langchain_tavily import TavilySearch

tavily = TavilySearch(max_results=3, topic="general")

@tool
def web_search_tool(query: str) -> str:
    """Up‑to‑date web info via Tavily"""
    return "

".join(r["content"] for r in tavily.invoke({"query": query})["results"])  # simplified

4️⃣ Extend the Agent State

class AgentState(State):          # add to previous `State`
    route:    str          # "rag", "answer", "web", "end"
    rag:      str | None   # KB result
    web:      str | None   # web‑search snippets

5️⃣ Decision / Execution Nodes

Key implementation points (condensed):

# ── Structured helpers ─────────────────
class RouteDecision(BaseModel):
    route: Literal["rag", "answer", "end"]
    reply: str | None = None

class RagJudge(BaseModel):
    sufficient: bool

router_llm = ChatOpenAI(model="gpt-4.1-mini", temperature=0).with_structured_output(RouteDecision)
judge_llm  = ChatOpenAI(model="gpt-4.1-mini", temperature=0).with_structured_output(RagJudge)
answer_llm = ChatOpenAI(model="gpt-4.1-mini", temperature=0.7)

# ── Router ─────────────────────────────
def router_node(state: AgentState) -> AgentState:
    q = state["messages"][-1].content
    decision = router_llm.invoke([
        ("system", "Decide route: rag / answer / end"),
        ("user", q)
    ])
    new_state = {**state, "route": decision.route}
    if decision.route == "end":
        new_state["messages"] += [AIMessage(content=decision.reply or "Hello!")]
    return new_state

# ── RAG lookup ─────────────────────────
def rag_node(state: AgentState) -> AgentState:
    q = state["messages"][-1].content
    chunks = rag_search_tool.invoke(q)
    verdict = judge_llm.invoke([("user", f"Question: {q}
Docs: {chunks[:300]}…")])
    return {**state, "rag": chunks, "route": "answer" if verdict.sufficient else "web"}

# ── Web search & Answer nodes omitted for brevity (same as notebook) ──

6️⃣ Wire up the Graph

agent_graph = StateGraph(AgentState)
agent_graph.add_node("router",      router_node)
agent_graph.add_node("rag_lookup",  rag_node)
agent_graph.add_node("web_search",  web_node)
agent_graph.add_node("answer",      answer_node)

agent_graph.set_entry_point("router")
agent_graph.add_conditional_edges("router", from_router,
        {"rag": "rag_lookup", "answer": "answer", "end": END})
agent_graph.add_conditional_edges("rag_lookup", after_rag,
        {"answer": "answer", "web": "web_search"})
agent_graph.add_edge("web_search", "answer")
agent_graph.add_edge("answer", END)

agent = agent_graph.compile(checkpointer=MemorySaver())

7️⃣ Quick CLI test

if __name__ == "__main__":
    config = {"configurable": {"thread_id": "thread‑12"}}
    while True:
        q = input("You: ").strip()
        if q in {"quit", "exit"}: break
        result = agent.invoke({"messages": [HumanMessage(content=q)]}, config)
        print(result["messages"][-1].content)

Now your LangGraph agent:

• Routes intelligently

• Retrieves domain knowledge with RAG

• Falls back to web search when KB is insufficient

• Streams multi‑turn answers with memory

In short, this is a production‑ready skeleton you can plug into any project.

🚀 Conclusion & Resources

In this tutorial we climbed the ladder from basic LLM calls ➜ tool‑aware agents ➜ memory ➜ RAG ➜ full multi‑step routing with LangGraph. You now have a production‑ready skeleton that can:

• Chat naturally across turns (memory)

• Decide when to use internal knowledge vs. external tools (router)

• Pull trusted data from your own docs (RAG)

• Fall back to real‑time web search when the KB is lacking

📂 Grab the code

• Full Notebook on GitHub: LangGraph RAG Agent Notebook

🕹 Try the live RAG Agent: https://agent.futuresmart.ai/

🎥 Watch the build walkthrough

What’s next?

1. Swap in your own docs. Point the loader at your knowledge base and rebuild the index.

2. Add streaming. LangGraph supports async generators so you can pipe partial answers to the UI.

3. Deploy. Package the graph inside a FastAPI endpoint or a serverless function and wire up a front‑end.

Got questions or improvement ideas? drop a comment under the YouTube video – I’d love to hear how you extend this skeleton!

Happy building 🛠️🤖

In the era of advanced AI applications, Retrieval-Augmented Generation (RAG) stands out as a game-changing approach. By combining retrieval techniques with generative models, RAG enhances the quality, accuracy, and relevance of generated outputs. This blog walks you through building a scalable and efficient RAG system using FastAPI, Qdrant, LangChain, and OpenAI, all while leveraging asynchronous capabilities for improved performance.

At FutureSmart AI, we are committed to pioneering innovative solutions and leveraging cutting-edge technologies. Building a RAG system with Async FastAPI, Qdrant, Langchain, and OpenAI has helped us create efficient and Highly scalable AI-powered applications for our clients. This blog primarily reflects on our dedication to empowering developers and organizations with actionable knowledge to implement high-performance systems.

While this blog focuses on an on-premise setup for a hands-on approach, drawing from our experience we can assure that these tools also support scalable cloud-based deployments, ensuring flexibility for production-ready solutions. At FutureSmart AI, we’re always exploring and refining methods to push the boundaries of what AI can achieve.

Overview of Retrieval-Augmented Generation (RAG)

RAG combines two essential components:

1. Retrieval: Find relevant documents from a large dataset. This part uses a search mechanism to identify the most relevant passages from a large text based on the input query.

2. Generation: Uses a language model to generate context-aware answers. Once relevant information is retrieved, a language model generates the final response by incorporating the retrieved context into the generated text.

This integration empowers Retrieval-Augmented Generation (RAG) to deliver more accurate and contextually relevant responses compared to standalone Large Language Models (LLMs).

For a comprehensive understanding, explore our Langchain RAG Course: From Basics to Production-Ready RAG Chatbot or, if you prefer reading, visit our detailed Blog for more insights.

The Tech Stack: What You Need & Why

Let's break down our tools and why we chose them. Each one plays a crucial role in building a powerful RAG system.

FastAPI

FastAPI enables the rapid development of performant web APIs with asynchronous capabilities. Its support for Python-type hints makes it developer-friendly and robust.

For more information, check out the FastAPI Tutorial.

Qdrant

Qdrant excels in high-dimensional vector storage and retrieval operations. In our enterprise implementations, it has proven invaluable for:

• Efficient management of large-scale vector datasets

• Optimal performance in similarity search operations

• Seamless horizontal scaling capabilities

For a detailed and in-depth explanation please refer to our Comprehensive Guide to Installing and Using Qdrant VectorDB with Docker Server and Local Setup

LangChain

LangChain and its components, such as chains, prompts, and memory, enable efficient interaction with LLMs.

OpenAI

We will use OpenAI’s language models in this tutorial. You'll also need a basic understanding of how to send queries to OpenAI’s API and interpret responses.

Prerequisites

1. Create a Python Virtual Environment

It’s recommended to use a virtual environment to isolate your dependencies.

2. Install Dependencies

Use the provided requirements.txt file to install the necessary Python packages.

pip install -r requirements.txt

3. Setting up API Keys

To connect to external services like OpenAI and Qdrant, you need to set up API keys securely.

4. Create a .env File

Create a .env file in the root of your project directory to store sensitive information like API keys and configuration details.

Example .env file:

OPENAI_API_KEY=your_openai_api_key
qdrant_db_path=http://localhost:6333  # Replace with your Qdrant URL
llm_provider="openai"
model="gpt-4o-mini"
temperature="0.1"
chunk_size = 2000
no_of_chunks = 3

5. Load Environment Variables

Use libraries like python-dotenv to load the .env file into your application.

from dotenv import load_dotenv
import os

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
QDRANT_URL = os.getenv("QDRANT_URL")

Project Structure

services/
    logger.py
    pydantic_models.py
uploads/
    xxx.txt
    yyy.pdf
    zzz.docx
utils/
    __init__.py
    db_utils.py
    langchain_utils.py
    prompts.py
    qdrant_utils.py
    utils.py
.env
api.py

• services/: This folder houses essential services that support core functionalities:

  • logger.py: Manages the logging setup for the application. Logging is critical for debugging, monitoring, and tracking the application's behavior.

  • pydantic_models.py: Defines Pydantic models used for data validation and serialization. Pydantic ensures data entering the application is valid and formatted correctly.

• uploads/: A dedicated folder for file uploads. This is where the application stores temporary or permanent files uploaded by users.

• utils/: A utility module containing helper scripts that encapsulate reusable logic:

  • __init__.py: Marks the folder as a Python package.

  • db_utils.py: Contains functions for interacting with the database.

  • langchain_utils.py: Provides utility functions for integrating LangChain, a framework for language model applications.

  • prompts.py: Stores pre-defined prompts for interacting with language models or other AI systems.

  • qdrant_utils.py: Handles operations with Qdrant, a vector search engine for similarity-based search.

  • utils.py: General-purpose utility functions used across the project.

• .env: A configuration file storing environment variables like database credentials, API keys, and other sensitive data.

• api.py: The application's entry point is where FastAPI initializes and routes are defined. This file connects all the components and defines the API endpoints.

Setting Up Qdrant for Efficient Retrieval

Imports and Configuration

import os
import time
from dotenv import load_dotenv
from uuid import uuid4
import asyncio

# Langchain imports
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
from langchain_qdrant import QdrantVectorStore
from langchain_openai import OpenAIEmbeddings

# Qdrant imports
from qdrant_client import QdrantClient, AsyncQdrantClient
from qdrant_client.http.models import Distance, VectorParams

from services.logger import logger
from uuid import uuid4

load_dotenv(override=True)

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
qdrant_db_path=os.getenv("qdrant_db_path")

The setup begins with importing essential libraries, loading environment variables (like API keys), and initializing necessary configurations. Notable imports include LangChain's RecursiveCharacterTextSplitter for chunking documents and Qdrant’s async client for vector database interactions.

DocumentIndexer Class

The DocumentIndexer class handles indexing and retrieval in Qdrant. Let’s break it down step-by-step.

Initialization

class DocumentIndexer:
    def __init__(self, qdrant_db_path):
        self.db_path = qdrant_db_path
        self.embedding_function = OpenAIEmbeddings(model="text-embedding-3-large", api_key=OPENAI_API_KEY)
        self.vector_store = None
        self.client = AsyncQdrantClient(self.db_path)

• embedding_function: Uses OpenAI’s embeddings to convert text into dense vector representations.

• client: Initializes an async Qdrant client to manage the vector database.

• vector_store: Qdrant vector store is used to add documents and manage their vector representations.

Indexing Text in Qdrant

The method index_in_qdrantdb handles the extraction and indexing of document text. Here’s how it works:

async def index_in_qdrantdb(self, extracted_text, file_name, doc_type, chunk_size):
    try:
        # Create a Document object
        doc = Document(
            page_content=extracted_text,
            metadata={
                "file_name": file_name,
                "doc_type": doc_type
            }
        )


        chunk_size = int(os.getenv("chunk_size"))
        logger.info(f"Using dynamic chunk size: {chunk_size}")

        # Split the document
        text_splitter = RecursiveCharacterTextSplitter(
            separators=['\\n\\n', '\\n', ','],
            chunk_size=chunk_size,
            chunk_overlap=200
        )
        docus = text_splitter.split_documents([doc])

        # Generate UUIDs for all chunks
        uuids = [f"{str(uuid4())}" for _ in range(len(docus))]
        collection = "rag_demo_collection"

        collections = await self.client.get_collections()

        if collection in [collection_name.name for collection_name in collections.collections]:
            logger.info(f"Collection {collection} already exists in QdrantDB")
        else:
            await self.client.create_collection(
                collection_name=collection,
                vectors_config=VectorParams(size=3072, distance=Distance.COSINE))

        self.vector_store =  QdrantVectorStore.from_existing_collection(collection_name=collection, embedding=self.embedding_function, url=self.db_path)

        await self.vector_store.aadd_documents(documents=docus, ids=uuids)

        logger.info(f"Successfully indexed document in QdrantDB")
        return True

    except Exception as e:
        logger.error(f"Error indexing document in QdrantDB: {e}")
        raise

Key Points:

1. Document Creation: Loading and splitting extracted data asynchronously for efficient processing. A Document object is created to store extracted text and metadata.

2. Chunking of Document: The document is divided into manageable chunks using RecursiveCharacterTextSplitter.

3. Collection Management: The Qdrant collection is created only if it doesn’t already exist.

4. Batch Indexing: Chunks are added to the Qdrant database with unique UUIDs.

5. Using openai embedding model to create vector representations of documents.

6. Asynchronously uploading documents to Qdrant for similarity search.

Retrieving Documents

To enable querying of indexed data, the get_retriever method returns a retriever:

async def get_retriever(self, top_k):
    try:
        collection = "rag_demo_collection"
        if self.vector_store is None:
            self.vector_store =  QdrantVectorStore.from_existing_collection(collection_name=collection, embedding=self.embedding_function, url=self.db_path)

        return self.vector_store.as_retriever(search_type="similarity", search_kwargs={"k": top_k})
    except Exception as e:
        logger.error(f"Error creating retriever: {e}")
        raise

Key Points:

1. Retriever Initialization: If the vector_storeobject doesn’t exist, it initializes a retriever from the existing collection.

2. Search Parameters: Supports similarity-based searches with a configurable top_k parameter.

Implementing the Asynchronous FastAPI Endpoint

Asynchronous endpoints allow the server to handle multiple requests simultaneously, which is essential for applications that process large files or perform complex computations.

Setting Up FastAPI

The first step is initializing a FastAPI application that supports asynchronous request handling. This allows the server to process multiple incoming requests without blocking other operations, essential for high-performance APIs.

app = FastAPI()

The application is initialized with the FastAPI() class, which serves as the primary entry point for defining routes and handling requests.

Defining API Routes

The code leverages async def for efficient non-blocking request handling, ensuring high performance under concurrent loads. Two main routes are implemented: the /upload-knowledge endpoint and the /chat endpoint. These routes demonstrate seamless integration of file processing, database operations, and conversational AI.

1. Document Ingestion Endpoint (/upload-knowledge)

   • Allows users to upload files containing knowledge documents.

   • Extracts text from the uploaded file and indexes it in a database for future query responses

    @app.post("/upload-knowledge")
    async def upload_knowledge(
        username: str = Form(...),
        file: Optional[UploadFile] = File(None)
    ):
        try:
            # Handle file extraction and indexing
            extracted_text = ""
            if file:
                logger.info(f"File uploaded: {file.filename}")
                file_content = await file.read()
                file_extension = file.filename.split('.')[-1].lower()
                extracted_text = await extract_text_from_file(file_content, file_extension)
                logger.info(f"Extracted text from file: {extracted_text}")
                await index_documents(username, extracted_text, file.filename, file_extension)
            return {'response': 'Indexed Documents Successfully', 'extracted_text': extracted_text}
        except ValueError as e:
            raise HTTPException(status_code=400, detail=str(e))
        except Exception as e:
            logger.error(f"Error processing indexing request: {str(e)}")
            raise HTTPException(status_code=500, detail=f"Unexpected error: {e}")

File Text Extraction

The function extract_text_from_file extracts text from different file types (e.g., TXT, PDF, DOCX).

    # Asynchronous file text extraction
    async def extract_text_from_file(file_content: bytes, file_type: str) -> str:
        """
        Extract text from different file types based on the file type.
        """
        if file_type == "txt":
            return await extract_text_from_txt(file_content)
        elif file_type == "pdf":
            return await extract_text_from_pdf(file_content)
        elif file_type == "docx":
            return await extract_text_from_docx(file_content)
        else:
            raise HTTPException(status_code=400, detail="Unsupported file type")

PDF Text Extraction

For PDF files, text extraction requires libraries like PyPDF2. Here's the asynchronous implementation:

    # Async version of the extract_text_from_pdf
    async def extract_text_from_pdf(file_content: bytes) -> str:
        """
        Extract text from a PDF file.
        """
        return await asyncio.to_thread(extract_text_from_pdf_sync, file_content)

      def extract_text_from_pdf_sync(file_content: bytes) -> str:
        """
        Extract text from a PDF file (blocking version).
        """
        content = ""
        pdf_reader = PyPDF2.PdfReader(file_content)
        num_pages = len(pdf_reader.pages)
        for i in range(num_pages):
            page = pdf_reader.pages[i]
            content += page.extract_text()
        return content

Indexing Documents

The index_documents function stores the extracted text in a Qdrant database, optimized for vector search and similarity queries.

    async def index_documents(username,extracted_text,filename,file_extension):
        try:
            indexer = DocumentIndexer(qdrant_db_path)
            start_time = time.time()
            logger.info("Searching for similar documents in Qdrant...")

            await indexer.index_in_qdrantdb(
                extracted_text=extracted_text,
                file_name=filename,
                doc_type=file_extension,
                chunk_size=1500  
            )
            logger.info(f"Document indexing completed in {time.time() - start_time:.2f} seconds")

        except Exception as e:
            logger.error(f"Error processing documents: {str(e)}")
            raise RuntimeError(f"Failed to process documents: {str(e)}")

Refer to the GitHub Code at the end of this article for Text extraction from different sources.

2. Chat Query Endpoint (/chat)

   • Accepts user queries and provides responses based on the ingested knowledge.

   • Handles previous session data to maintain conversational context.

    @app.post("/chat", response_model=ChatResponse)
    async def chat(request: ChatRequest):
        try:
            # Process chat request
            if request.session_id is not None:
                past_messages = await get_past_conversation_async(request.session_id)
            else:
                request.session_id = str(uuid4())
                past_messages = []

            response, refined_query, extracted_documents = await generate_chatbot_response(
                request.query, past_messages, request.no_of_chunks, request.username
            )
            await add_conversation_async(request.session_id, request.query, response)
            return {
                "username": request.username,
                "query": request.query,
                "refine_query": refined_query,
                "response": response,
                "session_id": request.session_id,
            }
        except ValueError as e:
            raise HTTPException(status_code=400, detail=str(e))
        except Exception as e:
            logger.error(f"Error processing chat request: {str(e)}")
            raise HTTPException(status_code=500, detail=f"Unexpected error: {e}")

Session Context Management

Fetching Past Conversations

To retain context, we retrieve past conversations from the SQLite database. Each session ID serves as a key to fetch previous interactions.

    async def get_past_conversation_async(session_id: str) -> List[dict]:
        start_time = asyncio.get_event_loop().time()
        messages = []

        try:
            # Open an async SQLite connection
            async with aiosqlite.connect("chat_log.db") as connection:
                await connection.execute('''CREATE TABLE IF NOT EXISTS chat_logs (
                    session_id TEXT,
                    user_query TEXT,
                    gpt_response TEXT
                )''')
                logger.info("Database schema ensured.")

                # Fetch chat logs for the given session_id
                async with connection.execute(
                    "SELECT user_query, gpt_response FROM chat_logs WHERE session_id=?", (session_id,)
                ) as cursor:
                    async for row in cursor:
                        message_user = {"role": "user", "content": row[0]}
                        message_assistant = {"role": "assistant", "content": row[1]}
                        messages.extend([message_user, message_assistant])

            elapsed_time = asyncio.get_event_loop().time() - start_time
            logger.info(f"History For Context (get_conversation): {messages} in {elapsed_time:.2f}s")
            return messages

        except Exception as e:
            logger.exception(f"Error occurred: {str(e)}")
            raise e

Adding New Conversations

New conversations are stored in the database after processing. This ensures the chatbot can build upon prior interactions.

    async def add_conversation_async(session_id, user_query, gpt_response):
        try:
            # Open an async SQLite connection
            async with aiosqlite.connect(":memory:") as connection:
                cursor = await connection.cursor()

                # Create table if it doesn't exist
                await cursor.execute('''CREATE TABLE IF NOT EXISTS chat_logs (
                                            session_id TEXT,
                                            user_query TEXT,
                                            gpt_response TEXT)''')

                # Insert new conversation
                await cursor.execute("INSERT INTO chat_logs (session_id, user_query, gpt_response) VALUES (?, ?, ?)",
                                    (session_id, user_query, gpt_response))

                await connection.commit()
                logger.info(f"Conversation added for session {session_id}")

        except Exception as e:
            logger.exception(f"Error occurred while adding conversation: {str(e)}")
            raise e

Request and Response Models

FastAPI leverages Pydantic models for robust data validation and serialization, ensuring input data adheres to the expected format. For example:

• ChatRequest Model

  • Defines the structure of incoming requests to the chat endpoint, including the username, query, and optional session_id.

    from pydantic import BaseModel, field_validator
    from typing import Optional, List

    class ChatRequest(BaseModel):
        username: str
        query: str
        session_id: Optional[str] = None
        no_of_chunks: Optional[int] = 3

• ChatResponse Model

  • Specifies the format of the API response, including the query, refined query, and chatbot response.

    class ChatResponse(BaseModel):
        username: str
        query: str
        refine_query: str
        response: str
        session_id: str
        debug_info: Optional[dict] = None

Orchestrating with LangChain

Implementing a RAG Chain

Combining vector search results with prompt engineering.

@ls.traceable(run_type="chain", name="Chat Pipeline")
async def generate_chatbot_response(query, past_messages, no_of_chunks,username):
    """Main function to generate chatbot responses asynchronously."""
    logger.info("Refining user query")
    refined_query = await refine_user_query(query, past_messages)  # Async call
    logger.info(f"Generated refined query: {refined_query}")

    extracted_text_data, extracted_documents = await retrieve_similar_documents(refined_query, int(no_of_chunks),username)  # Async call
    # logger.info(f"Extracted text data: {extracted_text_data}")
    logger.info(f"Extracted text data")


    llm = initialize_llm()  # Synchronous initialization
    history = create_history(past_messages)
    logger.info(f"Created history for session: {history}")

    logger.info("Fetching response")
    start_time = time.time()
    final_response, cb = await invoke_chain(query, extracted_text_data, history, llm)  # Async call
    response_time = time.time() - start_time

    # logger.info(f"Got response from chain: {final_response}")
    logger.info(f"Got response from chain:")

    return final_response, response_time, cb.prompt_tokens, cb.completion_tokens, cb.total_tokens, extracted_text_data, refined_query, extracted_documents

Query Refinement

User queries are often ambiguous, relying on prior interactions or chat history. To address this, we design a refinement mechanism to convert user input into

def get_query_refiner_prompt():
    contextualize_q_system_prompt = ("""
    "Given a chat history and the latest user question "
    "which might reference context in the chat history, "
    "formulate a standalone question which can be understood "
    "without the chat history. Do NOT answer the question, "
    "just reformulate it if needed and otherwise return it as it is."
    """)

    final_prompt = ChatPromptTemplate.from_messages(
        [
            ("system", contextualize_q_system_prompt),
            MessagesPlaceholder(variable_name="messages"),
            ("human","{query}"),
        ]
    )
    # print(final_prompt)
    return final_prompt

  async def refine_user_query(query, messages):
    """Refines the user query asynchronously."""
    llm = ChatOpenAI(temperature=0, model_name="gpt-4o")
    history = create_history(messages)
    prompt = get_query_refiner_prompt()
    refined_query_chain = prompt | llm | StrOutputParser()
    refined_query = await refined_query_chain.ainvoke({"query": query, "messages": history.messages})  # Async method
    return refined_query

Document Retrieval

The refined query serves as input for a retrieval mechanism. Using Qdrant, we extract contextually similar documents for subsequent processing.

async def retrieve_similar_documents(refined_query: str, num_of_chunks: int,username: str) -> str:
    try:
        indexer = DocumentIndexer(qdrant_db_path)
        start_time = time.time()
        logger.info("Searching for similar documents in Qdrant...")

        if num_of_chunks is None:
            num_of_chunks = os.getenv('no_of_chunks')
        if not isinstance(num_of_chunks, int) or num_of_chunks <= 0:
            raise ValueError(f"Invalid number of chunks: {num_of_chunks}")
        retriever = await indexer.get_retriever(top_k=num_of_chunks)
        if not retriever:
            raise ValueError("Failed to initialize document retriever")
        extracted_documents = await retriever.ainvoke(refined_query)
        if not extracted_documents:
            extracted_text_data=""
        else:
            extracted_text_data = await format_docs(extracted_documents)
        logger.info(f"Document retrieval and formatting completed in {time.time() - start_time:.2f} seconds")
        return extracted_text_data, extracted_documents

    except Exception as e:
        logger.error(f"Error processing documents: {str(e)}")
        raise RuntimeError(f"Failed to process documents: {str(e)}")

Prompt Engineering

Effective prompts are the backbone of any LLM-based pipeline. We design a system prompt that combines user inputs with the retrieved context.

def get_main_prompt():
    prompt = """ 
    "You are an assistant for question-answering tasks."
    "Use the following pieces of retrieved context and user information to answer the question."
    "If you don't find the answer of the query,then just say I don't have that information at hand. Please provide more details or check your sources."
    """
    prompt=prompt + "\\n\\n" + "{context}"

    final_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", prompt),
        MessagesPlaceholder (variable_name="messages"),
        ("human", "{user_query}")
    ])
    return final_prompt

async def invoke_chain(query, context, history, llm):
    """Handles the streamed response asynchronously."""
    logger.info(f"Initializing Chain using ...")
    final_chain = get_main_prompt() | llm | StrOutputParser()
    logger.info("Chain initialized.")
    input_data = {"user_query": query, "context": context, "messages": history.messages}

    with get_openai_callback() as cb:
        final_response = await final_chain.ainvoke(input_data)  # Asynchronous method

    return final_response, cb

Learn how to leverage LangChain Expression Language (LCEL) for seamless chain composition, including prompt formatting, retrieval-augmented generation (RAG), and efficient batching, with practical examples. Discover how LCEL simplifies building advanced LLM applications with features like streaming, parallelism, and async support! Checkout the video below

Future Improvements

• Enhanced Document Preprocessing: Implement advanced text chunking methods incorporating summarization, document-based chunking, semantic and agentic chunking, and multimodal support.

• Storage: Transition from in-memory to persistent storage ensures data durability across sessions. Migrating to async PostgreSQL enhances scalability and performance for larger datasets and higher user concurrency.

• Dynamic Few-Shot Learning: Automatically generate examples based on query type.

• Adaptive Retrieval: Use feedback loops to improve retrieval accuracy over time.

• Real-Time User Feedback: Allow users to fine-tune the response in real-time.

Conclusion

Through our extensive experience with asynchronous FastAPI and building RAG systems, we have successfully optimized every operation in the pipeline to work seamlessly in an asynchronous manner. From document ingestion and indexing in Qdrant to efficient retrieval of relevant context and history storage, we have made each operation highly efficient by adopting async-first principles.

By adopting this approach, developers can craft intelligent systems capable of providing contextually accurate and highly relevant responses. From document indexing to dynamic query refinement and real-time conversational AI, the RAG architecture represents a significant leap forward in harnessing the capabilities of large language models.

At FutureSmart AI, we specialize in delivering state-of-the-art AI solutions tailored to the unique needs of businesses. Leveraging advanced technologies such as RAG, NL2SQL, multi-agent architectures, LangChain, LangGraph, Qdrant, Chroma vector databases, and OpenAI, we have successfully implemented cutting-edge systems for multiple clients—from intelligent customer service automation to advanced AI-driven interview platforms.

If you want to leverage the power of AI Applications with asynchronous FastAPI, we’re here to help. Reach out to us at contact@futuresmart.ai and discover how our experience can translate into practical, cutting-edge solutions tailored for your needs.

Search engines and retrieval systems have evolved to become remarkably intelligent. They no longer rely on exact keyword matches or rigid rules to find what you're looking for. Instead, they understand the context and meaning behind your query, allowing you to retrieve information or objects from a database with just a vague hint. For example, consider an e-commerce platform where users search for products. A traditional keyword-based search might struggle if the search terms do not exactly match product descriptions. However, with semantic search, a user looking for "comfortable running shoes for long-distance" can receive relevant results, even if the product descriptions contain phrases like "marathon-ready sneakers with superior cushioning.”

The capabilities of semantic search systems extend far beyond text-based queries. By taking it a step further, you can enable users to search not just with text, but also with images, voice, and other forms of input. Semantic search relies on vector embeddings, which capture the deeper relationships between different data points to deliver highly relevant and contextually accurate results. This makes it a powerful tool for a wide range of applications, from E-commerce and Media & Entertainment to Smart Assistants. Let’s explore the key components needed to build such a system and unlock its potential across diverse industries.

Implementing a Semantic Search system

In this blog, we’ll walk you through implementing a semantic search system using Async FastAPI, Qdrant, and Sentence-Transformers to deliver efficient, context-aware search capabilities. Among the many vector database options available, we’ve chosen Qdrant for its exceptional support for fast, asynchronous retrieval, ensuring high performance and scalability—perfect for real-time applications. Let’s understand how it goes.

Technologies We’ll use

To build this system, we’ll leverage:

• Qdrant: A powerful, open-source vector database for efficient similarity searches.

• Sentence-Transformers: For generating high-quality embeddings from text data.

• Async FastAPI: To create a lightweight, asynchronous API for seamless query handling.

Qdrant

Qdrant is an advanced vector database designed to store and search high-dimensional vector embeddings. Its scalability and efficiency make it ideal for semantic search use cases, providing powerful support for similarity-based queries. We are using Qdrant, not only for it’s speed and accuracy, but also because of it’s async nature. That allows us to reduce memory usage, allow support for multiple users simultaneously, improving the number of concurrent requests that can be handled. You can read more about the Qdrant Async API and what it bring to the table in this Documentation.

After thoroughly evaluating Qdrant and adopting it into FutureSmart AI's workflows, we’ve seen the value it brings firsthand. We encourage you to explore Qdrant further and visit their official page to learn more about their offerings and capabilities.

Sentence-Transformers

Sentence-Transformers is a powerful library designed to transform text into dense vector embeddings, enabling a deeper semantic understanding of textual data. These high-dimensional vectors are essential for tasks like semantic search, similarity matching, and clustering.

Want to explore Sentence-Transformers in action? YouTube Tutorial , for a hands-on guide to dive deeper into real-world applications. 🚀

Fastapi

FastAPI is a cutting-edge, high-performance web framework for building APIs with Python. It leverages asynchronous programming for creating fast, scalable applications and uses Python type hints to ensure clarity, robustness, and developer productivity.

Want to master FastAPI? FastAPI Tutorial for step-by-step guidance to understand its core features and best practices. 🌟

Why Use Qdrant for Semantic Search?

Qdrant is a high-performance vector database designed specifically to power semantic search systems. It allows efficient and scalable storage and retrieval of embeddings, which represent the meaning of data points as vectors.

Key Advantages of Using Qdrant :

• Efficient Vector Search: 🚀 Qdrant is optimized for handling large-scale vector search queries with low latency, making it ideal for real-time semantic search applications.

• Scalable Architecture: 🌍 Built for horizontal scaling, Qdrant can support massive datasets while maintaining quick query responses, ensuring it handles growing data demands efficiently.

• Real-Time Updates: 🔄 Qdrant supports real-time indexing and updating of vector embeddings, allowing quick integration of new data points as they become available.

• Advanced Indexing Techniques: 🔍 Qdrant uses efficient indexing algorithms like HNSW (Hierarchical Navigable Small World), making vector searches fast even with a huge volume of data.

• Seamless Integration: 🔗 Qdrant integrates easily with frameworks like FastAPI, enabling smooth deployment of semantic search systems in modern applications.

Why Use Async?

Incorporating asynchronous processing enhances the efficiency and scalability of semantic search. By allowing multiple search queries to be processed concurrently without blocking the system, async offers significant benefits, particularly for large-scale applications.

Benefits of Async in Semantic Search with Qdrant:

• High-Volume Applications: 📈 Async enables high-throughput systems, allowing efficient handling of large numbers of simultaneous search queries—critical for large-scale or enterprise-level applications.

• Low Latency and Seamless User Experience: ⏱️ Non-blocking asynchronous operations reduce response times, improving the user experience by minimizing delays in real-time semantic search, especially when powered by Qdrant.

• Optimized Resource Utilization: ⚙️ Async operations maximize system efficiency by handling multiple tasks concurrently, leveraging non-blocking I/O, and minimizing resource wastage.

• Native Support in FastAPI: 🌐 FastAPI’s built-in async support makes it a natural fit for building scalable, high-performance semantic search systems.

Understanding the Workflow

1. The user submits a search query via an asynchronous FastAPI endpoint.

2. Qdrant processes the query and performs semantic similarity matching using precomputed embeddings.

3. Results are ranked by relevance, ensuring accurate information delivery.

Prerequisites

1. System Setup

Before you begin, ensure that the following components are installed:

1.1 Python

• Ensure you're using Python 3.8 or later.

1.2 Qdrant

• Qdrant should be up and running, either locally or hosted on a cloud service.

• If you're unfamiliar with Qdrant or need assistance with its setup, check out this in-depth guide we wrote as first part of our series. Comprehensive Guide to Installing and Using Qdrant VectorDB with Docker Server and Local Setup.

1.3 Required Libraries

• Install the necessary libraries using the following command:

    pip install fastapi uvicorn sentence-transformers qdrant-client
  

Project Structure

Here’s the project structure:

plaintextCopy code
semantic-search-system/
├── app/
│   ├── __init__.py
│   ├── main.py          # FastAPI application and routes
│   ├── utils.py         # Helper functions (e.g., read_and_store_data)
│   ├── qdrant_utils.py  # Helper function to upsert data to Qdrant
│   └── models.py        # Define Pydantic models
├── vehicle.csv          # CSV file with vehicle data
└── requirements.txt     # List of dependencies

Adding Data to Qdrant via CSV

1. CSV File Preparation

The data from the CSV file will populate Qdrant for semantic search vectors. Here is a sample of the vehicle.csv file, which contains the following columns:

This .csv file should be saved in your project directory for easy access during data import.

2. Initializing the Qdrant Client

Before inserting data into Qdrant, initialize the Qdrant client with the AsyncQdrantClient from the qdrant-client library. The create_qdrant_collection function will connect to Qdrant and create a collection for your data, provided one doesn't already exist. A vector size and a distance metric (Cosine similarity in this case) need to be defined for accurate semantic search.

app/qdrant_utils.py:

import qdrant_client
from qdrant_client.models import VectorParams
from app.config import QDRANT_COLLECTION

# Initialize Qdrant client
qdrant = AsyncQdrantClient("<http://localhost:6333>")

# Create collection in Qdrant vector database
async def create_qdrant_collection():
    await qdrant.create_collection(
        collection_name="vehicles",  
        vectors_config=VectorParams(size=384, distance="Cosine") 
    )

• Vector Size: The dimensionality of the embedding vector, which should match the output from the Sentence-Transformer model. Here, we choose 384 dimensions for the model.

• Cosine: A similarity measure used to compute how similar two vectors are, which is perfect for comparing semantic meaning.

3. Async Data Insertion

Now that we have set up the Qdrant collection, it's time to insert data from the CSV file. Each textual column in the CSV (name, category, description) is passed through the Sentence-Transformer model to create vector embeddings, which are then stored in Qdrant.

app/utils.py:

import pandas as pd
from sentence_transformers import SentenceTransformer

# Initialize the Sentence-Transformer model globally
model = SentenceTransformer("all-MiniLM-L6-v2")

def generate_vector(text: str):
    """Generate vector embeddings for a given text."""
    return model.encode(text).tolist()

async def upsert_data_to_qdrant(qdrant, df):
    """Upsert data from DataFrame to Qdrant vector database."""
    vectors = [
        generate_vector(f"{row['name']} {row['category']} {row['description']}")
        for _, row in df.iterrows()
    ]
    try:
        # Insert the data into Qdrant
        await qdrant.upsert(
            collection_name="vehicles", 
            points=[{
                 "id": int(row['id']),
                 "vector": vector,
                 "payload": {"name": row['name'], 
                             "category": row['category'], 
                             "description": row['description']}
                }
                for vector, (_, row) in zip(vectors, df.iterrows())
            ]
        )
    except Exception as e:
        print(f"Error inserting data into Qdrant: {e}")
        raise e

async def read_and_store_data(csv_file_path, qdrant):
    """Reads CSV and upserts into Qdrant."""
    df = pd.read_csv(csv_file_path)  # Load CSV into a DataFrame

    # Upsert to Qdrant
    await upsert_data_to_qdrant(qdrant, df)

This approach ensures efficient handling of large datasets with asynchronous processing. The generate_vector function utilizes the all-MiniLM-L6-v2 Sentence-Transformer model to create meaningful vector embeddings by combining the name, category, and description fields. These embeddings capture the semantic meaning of each vehicle, improving search accuracy.

By using Qdrant's upsert operation, we prevent duplicate entries while allowing seamless data updates. Additionally, you can manage and inspect your Qdrant collection using the Qdrant dashboard, where you can view inserted data, monitor collection health, and perform advanced searches.

For learning more about accessing the Qdrant Web UI, you can refer to our previous blog : Comprehensive Guide to Qdrant Vector DB: Installation and Setup.

Building the Semantic Search API

Pydantic Model

Pydantic models provide a robust way to validate and serialize data, and they are integral to FastAPI's request handling. We define a QueryRequest model to structure incoming data for the search endpoint. This model will specify the expected query string and the limit for the number of search results:

app/models.py:

from pydantic import BaseModel

class QueryRequest(BaseModel):
    query: str  # The search query entered by the user
    limit: int  # The number of search results to return

Qdrant Search Function

For semantic search, we leverage Qdrant, a vector database. The function generates a query vector from the input query using the SentenceTransformer model, which translates the user's query into a vector representation. We then search Qdrant for vectors similar to the query vector.

app/search.py:

from app.qdrant_utils import qdrant
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

async def qdrant_search(query: str, limit: int):
    # Convert the user's query into a vector
    vector = model.encode(query).tolist()

    search_result = await qdrant.search(
                   collection_name="vehicles",  # The collection we're searching in
                   query_vector=vector,
                   limit=limit  # Limit the number of results returned by Qdrant
                 )
    # Format the search results for consistency
    results = [{
                    "id": res.id,
                    "name": res.payload.get("name"),
                    "category": res.payload.get("category"),
                    "description": res.payload.get("description"),
                    "score": res.score  # Semantic similarity score
                } for res in search_result]

    return results

FastAPI Endpoint for Search

Now, we implement the FastAPI endpoint that ties everything together. This endpoint listens for a POST request at /semantic_search, where it receives a query and a limit for the results. It fetches results from Qdrant using vector-based semantic search.

API Definition:

app/main.py:

from fastapi import FastAPI, HTTPException
from app.search import qdrant_search
from app.models import QueryRequest
from app.qdrant_utils import qdrant
from app.utils import read_and_store_data

# FastAPI lifecycle event to initialize Qdrant
async def async_lifespan(app: FastAPI):
    await create_qdrant_collection()  # Ensures the Qdrant collection exists
    await read_and_store_data('vehicle.csv', qdrant)  # Load initial data
    yield  # Yield control, FastAPI will now handle incoming requests

app = FastAPI(lifespan=async_lifespan)

@app.post("/semantic_search")
async def search(query_request: QueryRequest):
    # Perform semantic search in Qdrant
    results = await qdrant_search(query_request.query, query_request.limit)

    # Return the results
    return {"results": results}

How This Works:

1. Qdrant Search: The function queries Qdrant for results based on semantic similarity using vector-based search.

2. Return Results: The search results from Qdrant are returned, respecting the limit set by the user.

Running and Testing the Application

To run the application, use Uvicorn, a fast ASGI server:

uvicorn main:app --reload

Once the server is running, you can use the /semantic_search endpoint to query the Qdrant database.

If everything is configured correctly, you will receive semantic search results similar to the example shown below.

Demo :

Example Query & Result Breakdown

Sample Query :

Make a POST request to/semantic_search with the following payload:

{
"query": "sporty bikes",
"limit": 3
}

Expected Output :

{
  "results": [
    {
      "id": 17,
      "name": "Ducati Monster",
      "category": "Motorcycle",
      "description": "A sporty motorcycle with a distinctive design and powerful performance.",
      "score": 0.5491828
    },
    {
      "id": 29,
      "name": "Yamaha YZF-R3",
      "category": "Motorcycle",
      "description": "A lightweight sport motorcycle ideal for beginners and experienced riders alike.",
      "score": 0.52757794
    },
    {
      "id": 9,
      "name": "Harley-Davidson Street 750",
      "category": "Motorcycle",
      "description": "A cruiser motorcycle with a classic design and powerful engine.",
      "score": 0.47901446
    }
  ]
}

Advantage with Semantic Search

Unlike traditional keyword-based search, which depends on exact keyword matches, vector search retrieves results based on semantic similarity. This means the system focuses on the intent behind queries and the contextual meaning of terms, providing a more intelligent and effective search experience for modern applications.

Example Query: "sporty bikes "

Traditional Keyword Search:

• Returns only results that explicitly include the word "bikes" in their name or description.

• Misses relevant items such as Ducati Monster, a motorcycle not explicitly labeled as "sporty bikes."

Vector Search (Semantic Approach):

• Recognizes that "bikes" and "motorcycles" are semantically similar and retrieves items that are contextually related, like sporty motorcycles, even when the word "bikes" is not directly mentioned.

• The system also ranks results using relevance scores, indicating how closely each result aligns with the intent behind the query.

Comparison of Search Approaches

Improvements

• You can enhance the performance of the semantic search system by fine-tuning the embedding model on domain-specific data to improve result relevance.

• You can experiment with multimodal search, allowing for querying with images, voice, or other media types, expanding the search functionality.

• Adding filtering capabilities would allow for more precise search results, enabling users to refine searches based on metadata like date ranges.

Conclusion

Through this step-by-step guide, you've learned how to build a semantic search system with FastAPI and Qdrant, integrating modern vector-based search techniques to create an efficient and scalable solution. By leveraging embeddings, you can enhance search relevance and deliver more accurate results for your applications.

But this is just the beginning. As you continue exploring Qdrant, consider diving into its advanced capabilities, such as hybrid search and large language model integrations, to further refine your search experience.

At FutureSmart AI, we specialize in developing AI-driven solutions tailored to your business needs, from semantic search to custom AI Solutions. If you're looking for expert guidance or custom AI solutions, reach out to us at contact@futuresmart.ai.

Let’s build the future of AI together! 🚀

In the era of AI-driven applications and unstructured data management, vector databases have become essential tools for enabling semantic search and similarity matching. Qdrant is one such cutting-edge, open-source vector database designed to simplify working with high-dimensional data embeddings. Whether you're building a recommendation system, integrating semantic search, or powering an AI chatbot, Qdrant makes it seamless to store, query, and manage vector embeddings at scale.

At FutureSmart AI, we pride ourselves on staying adaptable and embracing new technologies. If a solution has great potential and delivers value, we’re always eager to explore and adopt it. After testing Qdrant ourselves and seeing its impressive results, we’ve prepared this comprehensive guide to walk you through its installation and setup on Docker and locally. While the blog primarily focuses on on-premise deployment, it’s worth noting that Qdrant also offers a robust cloud platform that simplifies scaling and management for production use cases.

What is a Vector Database

Let us first understand what a Vector database is, and what advantages it has over traditional dbs. In simple terms, a vector database is a specialized database designed to store and work with vector embeddings. Vector embeddings are numerical representations of data that capture its meaning, features, or relationships. These embeddings are often generated by AI models and are used to process unstructured data like text, images, audio, or videos.

Unlike traditional databases, which rely on exact matches (like finding a name or ID), vector databases focus on finding similarity between data points, even when the input isn’t identical. Unlike traditional relational databases, vector databases enable semantic search, allowing advanced similarity retrieval and unstructured data management.

The above image is an example of how we can still find relevant or similar objects even if they are structurally completely different. If it were JSON objects instead of plain text, it would still be able to do it. That’s the power of similarity search or vector search. You can try out this similarity checker yourself at AI Demos Playground.

What is Qdrant

Among the top open-source vector databases, Qdrant stands out as a powerful, Rust-based vector database and similarity search engine. It supports seamless integration with LangChain for building sophisticated AI solutions. It offers robust performance, a user-friendly API, and support for Python. With its use of indexing, Qdrant delivers both speed and precision, making it a competitive choice for modern applications.

Key Features of Qdrant:

1. HNSW Indexing: Uses the Hierarchical Navigable Small World (HNSW) algorithm for fast and accurate similarity searches.

2. Distance Metrics: Supports Cosine Similarity, Dot Product, and Euclidean Distance for flexible vector search.

3. Free Vector Database: Qdrant is open-source and perfect for creating a local vector database.

4. Multi-Language APIs: Offers APIs for Python, JavaScript/TypeScript, Rust, and Go, ensuring smooth integration with various tech stacks.

5. Recommendation API: Includes a built-in API for creating efficient recommendation systems.

6. Scalability and Production-Ready: Designed for real-world applications, it scales to handle millions or billions of vectors seamlessly.

7. Hybrid Compatibility: Works well with databases like PostgreSQL for blending relational and vector data.

Semantic Vs Vector Search

Semantic Search: Interprets user intent and context to deliver relevant results beyond simple keyword matching. It utilizes vector search to provide more accurate and contextually relevant outcomes.

Vector Search: Transforms text into vectors representing semantic meaning, enabling rapid similarity comparisons within large datasets.

In essence, vector search serves as a foundational component of semantic search, facilitating the understanding and retrieval of information based on meaning rather than mere keywords.

Setting Up the Environment:

We'll cover two approaches to using Qdrant:

1. Local Setup: Create a local LangChain vector database for prototyping.

2. Docker-based Server: Use a containerized approach for scalability and production.

Installing Qdrant Using Docker:

• Pull the Qdrant Docker Image:

docker pull qdrant/qdrant

• Run the Qdrant Container:

docker run -p 6333:6333 -p 6334:6334 -v "${PWD}/qdrant_storage:/qdrant/storage:z" qdrant/qdrant

This Docker command runs the Qdrant container, exposing ports 6333 (REST API) and 6334 (gRPC API), while mapping the host directory ${PWD}/qdrant_storage to /qdrant/storage inside the container to persist data.

Python Client:

To create a vector database in Python, start by installing the Python client:

pip install qdrant-client[fastembed]

Securing Your Qdrant Instance:

When running the Qdrant container, you can enable API key authentication by setting specific environment variables. This is an Optional Step, but it helps secure your Qdrant vector database from unauthorized requests, ensuring only trusted clients can access it.

• QDRANT__SERVICE__API_KEY: Set this to the desired API key.

Here’s the command:

docker run -d  -p 6333:6333  -e QDRANT__SERVICE__API_KEY=your-api-key-here -v "${PWD}/qdrant_storage:/qdrant/storage:z"  qdrant/qdrant

Replace your-api-key-here with a strong API key. if needed.

Understanding the Data

• The Json data represents movie entries including key details like its name, description, director, and release year. In Qdrant, this information will be stored as a point, where the vector could be derived from the movie description (e.g., embedding for semantic search), and the additional fields like director, year, and name will be stored as metadata or payload for enriched querying and context.

[
    {
        "name": "Sholay",
        "description": "Two ex-convicts are hired by a retired policeman to capture a ruthless dacoit terrorizing a village.",
        "director": "Ramesh Sippy",
        "year": 1975,
    },
    {
        "name": "Lagaan",
        "description": "Villagers unite to play a cricket match against British officers to abolish oppressive taxes.",
        "director": "Ashutosh Gowariker",
        "year": 2001,
    }
        ...
        ...
]

Loading the Data

import os

def read_files_from_folder(folder_path):
    file_data = []

    for file_name in os.listdir(folder_path):
        if file_name.endswith(".json"):
            with open(os.path.join(folder_path, file_name), 'r') as file:
                # content = file.read()
                content = json.load(file)
                file_data.append({"file_name": file_name, "content": content})

    return file_data

folder_path = "data"
file_data = read_files_from_folder(folder_path)

Then, we create separate lists for documents, metadata, which we add to our collection.

documents = []
metadatas = []
ids = []

import uuid 

for file_index, data in enumerate(file_data):
    context = data["content"]
    documents.extend(movie["description"] for movie in context)
    metadatas.extend(
        {**{key: value for key, value in movie.items() if key != "description"}, "source": data["file_name"]}
        for movie in context
    )
    ids.extend(str(uuid.uuid4()) for _ in context)  # Generate a unique UUID for each movie

documets = ['Two ex-convicts are hired by a retired policeman to capture a ruthless dacoit terrorizing a village.',
...]

metadatas = [{'name': 'Sholay',
 'director': 'Ramesh Sippy',
 'year': 1975,
 'source': 'movies.json'},
...]

ids = ['7aa5f6f5-5de5-4776-8090-9a7c38a4cfcf',
...]

Initializing Qdrant Client through Local or Docker

Setting Up a Local In-Memory Qdrant Instance:

This approach is ideal for development, prototyping, or testing without the need for a running server.

from qdrant_client import QdrantClient, models
# Initialize the local client
qdrant = QdrantClient(":memory:")  # or QdrantClient(path="path/to/db")

Connecting to a Qdrant Server via Docker:

Ensure that your Qdrant server is running, typically accessible at http://localhost:6333.

from qdrant_client import QdrantClient, models
# Connect to Docker server Qdrant instance
qdrant = QdrantClient("<http://localhost:6333>", api_key=your-api-key-here)

You can choose between running Qdrant in a Docker container (qdrant) or a local setup (client), depending on your specific requirements and environment. Code is same for both approach

Adding Documents

Inserting Data into Your Collection:

You can replace qdrant variable with client for using local in-memory setup

# Use the new add method
qdrant.add(
    collection_name="new_movie_collection",
    documents = documents,
    metadata = metadatas
)

Output: List of IDs of added documents. If no ids provided, will be randomly generated
['d8984772ed664b2b8e2da23b0660989c',...,'fb32ac5bc5a74e48ba0a8fbb578fedfe']

This function adds text documents to a Qdrant collection, creating the collection with the default vector config if it doesn’t exist. Documents are embedded using the default embedding model. If you need custom embeddings or vectors, we will see a different function in a while.

Querying

Performing Vector Searches:

search_result = qdrant.query(
    collection_name="movie_collection",
    query_text="for adults",
    limit = 1
)
print(search_result)

# List of QueryResponse object
[QueryResponse(id='96d0e002-bc96-47a5-9307-16e0924ff9f6', embedding=None, sparse_embedding=None, metadata={'document': 'The heartwarming tale of a mute and deaf man and his relationships with two women.', 'name': 'Barfi!', 'director': 'Anurag Basu', 'year': 2012}, document='The heartwarming tale of a mute and deaf man and his relationships with two women.', score=0.76938045)]

In Qdrant, a QueryResponse represents the outcome of a search query, typically including the point ID, document, similarity score, associated payload (metadata), and optionally, vector embeddings. By default, the vector data is not included in the response. To include, you can adjust the parameter: with_vector as True

Using a Different Embedding Model

Qdrant’s default embedding model is BAAI/bge-small-en, used for generating vector embeddings. You can use open-source embedding models like all-MiniLM-L6-v2 for custom embeddings:

To use a different model when creating a new collection, you need to specify its configuration explicitly, including the embedding model's details, during the collection setup in the Qdrant API or dashboard.

from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("all-MiniLM-L6-v2")

This resource provides a comprehensive guide on implementing sentence embeddings, similarity measures, semantic search, and clustering using Sentence Transformers.

Create a Collection:

qdrant.create_collection(
    collection_name="my_movies",
    vectors_config=models.VectorParams(
        size=encoder.get_sentence_embedding_dimension(),  # Vector size is defined by used model
        distance=models.Distance.COSINE,
    ),
)

Format Data

documents = []
metadatas = []
ids = []

import uuid 

for file_index, data in enumerate(file_data):
    context = data["content"]
    documents.extend(movie["description"] for movie in context)
    metadatas.extend(
        {**{key: value for key, value in movie.items() }, "source": data["file_name"]}
        for movie in context
    )
    ids.extend(str(uuid.uuid4()) for _ in context)  # Generate a unique UUID for each movie

documets = ['Two ex-convicts are hired by a retired policeman to capture a ruthless dacoit terrorizing a village.',
...]

metadatas = [{'name': 'Sholay',
 'description': 'Two ex-convicts are hired by a retired policeman to capture a ruthless dacoit terrorizing a village.',
 'director': 'Ramesh Sippy',
 'year': 1975,
 'source': 'movies.json'},
...]

ids = ['7aa5f6f5-5de5-4776-8090-9a7c38a4cfcf',
...]

Insert Data:

In Qdrant, you can insert data using two methods:

• Record-Oriented Approach: Utilize the upload_points method with a list of points, each containing an id, vectorembedding, and payload.

• Column-Oriented Approach: Employ the upload_collection method providing separate lists for ids, vectors, and payload.

Both methods facilitate efficient data insertion, allowing you to choose the format that best suits your workflow.

qdrant.upload_collection(
    collection_name="my_movies",
    ids=ids,
    vectors=encoder.encode(documents),
    payload=metadatas
)

Querying:

• In the provided query, a filter is applied on the payload data to retrieve points from the my_movies collection where the year field is greater than or equal to 2005. This refines the similarity search to include only relevant results.

from qdrant_client.models import Filter, FieldCondition, MatchValue

hits = qdrant.query_points(
    collection_name="my_movies",
    query=encoder.encode("engineering student life").tolist(),
    limit=3,
    with_payload=True,
    query_filter=Filter(
        should=[FieldCondition(
            key="year",
            range=models.Range(
            gt=None,
            gte=2005,
            lt=None,
            lte=None,),
        )]
    ),
).points

for hit in hits:
    print(hit.payload, "score:", hit.score)

Output

{'name': '3 Idiots', 'description': 'Three engineering students navigate the pressures of academia while challenging societal norms.', 'director': 'Rajkumar Hirani', 'year': 2009} score: 0.6074454
{'name': 'Dangal', 'description': 'A former wrestler trains his daughters to become world-class wrestlers against societal odds.', 'director': 'Nitesh Tiwari', 'year': 2016} score: 0.23663726
{'name': 'Taare Zameen Par', 'description': 'A dyslexic boy discovers his artistic talent with the help of a compassionate teacher.', 'director': 'Aamir Khan', 'year': 2007} score: 0.1795995

In our in AI Demos Playground, we provide all AI tools like Semantic Similarity Checker that allows you to upload text or files and receive similarity scores. It utilizes models from OpenAI, Hugging Face, Google, and Mistral to provide accurate assessments of semantic similarity.

Integration with OpenAI

To utilize OpenAI embeddings, encode your text using the OpenAI API and provide the resulting vector when uploading or querying points in your database. When creating a collection in Qdrant, set the vector size to 1,536 to match the dimensionality of the embeddings generated by models like text-embedding-ada-002

import openai
import os 

embedding_model = "text-embedding-3-small"

openai_client = openai.Client(
    api_key="<YOUR_API_KEY>"
)

result = openai_client.embeddings.create(input="hey texts how are you?", model=embedding_model)
result

# Extract the embedding vector
embedding_vector = result.data[0].embedding

# Determine the dimensionality
embedding_dimension = len(embedding_vector)
print(f"The embedding dimensionality is: {embedding_dimension}")
# The embedding dimensionality is: 1536

Accessing the Qdrant Web UI:

You can manage Qdrant deployments through the WebUI, accessible at Dashboard.

The dashboard provides two primary sections: Console and Collection

1. Console: Use the REST API to interact with Qdrant for tasks like querying or managing data.

   

2. Collections: This section lets you organize, manage, and upload collections, as well as handle snapshots for backups or migrations.

   

   

The WebUI also features a Graph Tool to visualize relationships within datasets. Found under the Graph Tab in Collections, it represents data points as an interactive tree graph. Clusters of similar points are grouped, helping users uncover hidden patterns and explore connections. The tool is flexible, allowing zooming and manipulation for enhanced clarity and deeper insights into the data structure.

Key Improvements for Using Qdrant

1. Caching: Cache frequent query results using Redis or store precomputed embeddings to reduce latency.

2. LangChain Integration: Ensure embedding dimensions match Qdrant vectors and streamline query pipelines for LLM-based systems.

3. Scalability: Use Qdrant Cloud for scaling and dividing large datasets into smaller collections to enhance performance.

4. Vector Search Techniques: Experiment with distance metrics and chunk large documents to improve search relevance and precision.

Conclusion:

In this blog, we've explored how to set up and utilize Qdrant, a robust open-source vector database, using both local and Docker-based approaches. We've covered data preparation, insertion, and querying, demonstrating how Qdrant facilitates efficient similarity searches and unstructured data management. By following these steps, you can leverage Qdrant's capabilities to build sophisticated AI solutions tailored to your specific needs.

As you continue to work with Qdrant, consider exploring its advanced features, such as hybrid search capabilities and integration with large language models, to enhance your applications further. For more detailed information and documentation, visit Qdrant's official website.

At FutureSmart AI, we specialize in developing advanced AI solutions tailored to your business needs. Our expertise encompasses building state-of-the-art vector databases and integrating LangChain-powered applications, enabling efficient semantic search and unstructured data management.

Have questions or need assistance? Contact us at contact@futuresmart.ai for a consultation! Visit our website to discover how our AI technologies have delivered measurable business value.

Don’t miss our next tutorial, where we’ll dive deeper into semantic search optimizations, vector search, and more advanced LangChain-Qdrant integrations.

Let’s build the future together! 🌟

AI adoption has shifted significantly in recent years. Earlier, businesses focused on classical machine learning models for tasks like classification and Named Entity Recognition (NER). Now, almost every project we build at FutureSmart AI involves Generative AI. While traditional models still work, companies prefer LLMs because they offer higher accuracy, require little to no labeled data, and enable faster deployment.

Through our experience building custom AI solutions for clients, we have observed a clear trend: businesses are investing in AI for real-world applications that directly impact their workflows. The most in-demand solutions today include:

• Retrieval-Augmented Generation (RAG) chatbots

• Natural Language to SQL (NL2SQL) systems

• Document parsing and structured data extraction

• AI Agents that orchestrate multiple tools

In this blog, I’ll share insights from our work at FutureSmart AI, covering why these solutions are in demand and when businesses choose custom AI over off-the-shelf tools.

Want to see how AI solutions are making an impact? Check out our case studies where we showcase real-world AI implementations and their business outcomes.

The AI Shift: From Traditional ML to Generative AI

A few years ago, classical machine learning models dominated AI development. Companies built custom models for classification, sentiment analysis, or Named Entity Recognition (NER), requiring large labeled datasets and extensive tuning. While these models were effective, they came with challenges—data collection, training costs, and maintaining model performance over time.

Now, businesses are turning to LLMs (Large Language Models) like GPT-4, which offer pre-trained knowledge, adaptability, and higher accuracy with minimal data requirements. Instead of spending months curating labeled datasets, teams can leverage few-shot learning or prompt engineering to achieve comparable, if not superior, results in much less time.

At FutureSmart AI, we've worked with companies that initially built traditional ML models but switched to LLMs due to the faster implementation and better generalization. However, using LLMs effectively still requires customization, retrieval optimization, and integration with enterprise workflows, which is where custom AI solutions add value.

For those looking to go a step further, fine-tuning LLMs can unlock even more domain-specific accuracy and performance. Check out this step-by-step guide on how to Fine-Tune GPT-4o Model:

➡️ Fine-Tune GPT-4o Model Step by Step

This video walks through the process, best practices, and practical applications of fine-tuning GPT-4o for real-world business scenarios.

The Most In-Demand AI Solutions Businesses Are Adopting

Over the past year, we've observed a growing demand for AI solutions that provide tangible business value. While many AI advancements sound futuristic, companies are investing in practical AI applications that directly improve efficiency and decision-making.

1. Retrieval-Augmented Generation (RAG) Chatbots

Many businesses now require chatbots that can accurately retrieve information from internal documentation rather than relying solely on pre-trained models. RAG-based chatbots enable:

• Real-time document retrieval to provide precise answers.

• Enhanced accuracy by leveraging structured and unstructured data.

• Custom integrations with enterprise knowledge bases and APIs.

At FutureSmart AI, we've implemented RAG-based chatbots for companies needing scalable knowledge assistants. Unlike generic chatbots, these solutions provide business-specific insights and can be fine-tuned for domain-specific knowledge.

Want to learn more about RAG? Watch our YouTube video on building RAG solutions using LangChain for a hands-on guide, or read our in-depth blog on RAG chatbots to explore how they work and why they are transforming business interactions.

2. Natural Language to SQL (NL2SQL) Systems

Many enterprises struggle with querying large databases efficiently. NL2SQL solutions bridge this gap by enabling users to ask questions in plain English and receive structured SQL queries. This has proven useful in:

• Business intelligence & analytics where non-technical users need insights.

• Customer support & operations for quick data retrieval.

• Workflow automation by simplifying access to structured databases.

However, generic NL2SQL models often fail in production environments due to complex database structures and domain-specific nuances. We’ve developed custom NL2SQL models that fine-tune responses based on real-world query patterns and database structures. Additionally, some businesses prefer Text-to-SQL models, which function similarly but have different optimization strategies depending on the use case.

Interested in how NL2SQL can enhance data accessibility? Watch our YouTube video: Mastering Natural Language to SQL with LangChain for a step-by-step guide, or check out our blog on NL2SQL solutions to see practical implementations and best practices.

3. Document Parsing and Structured Data Extraction

Many businesses deal with large volumes of unstructured documents—contracts, invoices, resumes, and more. Extracting structured information from these documents manually is time-consuming and error-prone. AI-powered document parsing helps automate this process by:

• Extracting key details like names, dates, and financial figures.

• Improving accuracy over rule-based approaches using deep learning.

• Integrating seamlessly with business workflows and databases.

At FutureSmart AI, we've built custom document parsing solutions tailored to various industries. Our models handle OCR-based text extraction, entity recognition, and format normalization, ensuring businesses get clean, structured data from PDFs and scanned images.

Many businesses also combine Document Parsing + NL2SQL, enabling natural language queries over extracted data (e.g., Show me resumes of candidates with Python experience).

The Role of AI Agents in Business Automation

AI is no longer just about isolated solutions—it’s about intelligent orchestration. AI Agents are transforming how businesses automate workflows, making real-time decisions on when and how to use different AI tools. If you're interested in exploring multi-agent systems, check out our video on Building Multi-Agent Systems with OpenAI Swarm: Practical Example.

What Are AI Agents?

AI Agents are autonomous systems that can:

• Decide dynamically which AI tool to use based on user input.

• Combine multiple AI solutions like RAG, NL2SQL, and document parsing.

• Integrate seamlessly with enterprise software like HubSpot, Odoo, and Gmail.

For example, an AI Agent can:

• Retrieve documents using RAG before responding to a query.

• Generate SQL queries using NL2SQL for structured database insights.

• Extract key details from documents before feeding them into a report.

At FutureSmart AI, we specialize in designing custom AI Agents that fit specific business needs, ensuring they provide actionable, high-quality insights without human intervention.

Want to see AI Agents in action? Watch our YouTube video on AI Agents or read our blog series on AI Agents for a deeper understanding of how they work and their business applications.

Choosing the Right AI Solution for Your Business

With so many AI technologies available, choosing the right solution for your business can be challenging. Companies must consider several factors before deciding between off-the-shelf AI solutions and custom AI development.

When to Choose Off-the-Shelf AI Solutions

Pre-built AI solutions can be useful when:

• Your requirements are general (e.g., basic chatbots, automated transcription).

• Speed and cost are priorities—off-the-shelf solutions are ready to use.

• You need quick experimentation before committing to custom development.

However, these solutions often lack flexibility and may not integrate well with existing enterprise workflows.

When to Opt for Custom AI Development

Businesses choose custom AI solutions when:

• They need domain-specific accuracy (e.g., legal, financial, or medical AI solutions).

• They require seamless integration with existing software and databases.

• Scalability and long-term ownership are critical factors.

At FutureSmart AI, we help businesses evaluate whether an off-the-shelf solution meets their needs or if they require a tailored AI system that enhances efficiency and drives growth.

Want to discuss which AI solution fits your business? Contact us today at contact@futuresmart.ai

Final Thoughts

AI is rapidly transforming how businesses operate, but success depends on choosing the right solution for your needs. Whether it’s RAG-powered chatbots, NL2SQL systems, AI Agents, or document parsing, selecting the right approach can significantly impact efficiency and scalability.

At FutureSmart AI, we specialize in building custom AI solutions that align with business goals, ensuring AI is not just a tool but a competitive advantage.

If you're considering AI adoption or want to optimize your existing AI strategy, let's talk. Get in touch with us today and explore how AI can revolutionize your business.

This implementation covers different authentication endpoints, using Google OAuth2 for authentication, and secure API access through google auth. Such a setup not only enhances security but also simplifies the user experience by leveraging Google's robust authentication system. Additionally, we’ll add an endpoint that uses this authentication, demonstrating how authenticated APIs can provide tailored and secure responses.

Prerequisites

Before we begin, ensure you have the following ready:

• Google Cloud Console Account: To set up the OAuth client credentials.

• Python Environment: Installed and configured with the required libraries

Step 1: Setting Up Google OAuth2 Client on Google Cloud Console

To enable Google authentication, the first step is setting up an OAuth2 client in the Google Cloud Console:

1. Navigate to the Google Cloud Console.

2. Create a new project or select an existing one.

   

3. Once the project is opened/created, pull up the side bar and visit the APIs and Services section.

4. In that section select the “Credentials” options. The Credentials page will open.

   

5. If this is a new project or your first project, you will be asked to configure the OAuth Consent Screen with your application details. This is a simple consent where the type of users who will be using your product is required (either External or Internal). Select it as per your requirement.

   

6. Once the previous step is completed, you will be prompted to configure the consent screen of your application. This is how the consent screen looks like for the AI Demos Playground and AI Demos Tools.

   

   You can configure your application the way you want. You can put how the consent screen appears, what app name it shows, what logo to show and many more. Simply add your details and continue. For our ease we will add it as “sample-auth-app” and select myself as the developer. I will keep everything else as default and complete the setup.

   

   This is how your OAuth Consent Screen page will look like after the setup

   

7. Now that you have the Consent screen configured, head over to the Credentials page and click on “Create Credentials”. Choose the “OAuth Client ID” option

   

   This ensures that when someone tries to authenticate themselves to your app, it asks for the user consent and displays the consent screen with all the details for user to login.

8. Once you are on the OAuth Client ID page, you will be prompted with two compulsory fields. One is the Application Type and other is the Name.

   

   You can select the Application Type as per your app requirement. For this blog, we will select Web Application as we will be using it to authenticate our APIs via FastAPI. Also, I set the Name the same as our app: sample-auth-app.

   The other two options named Authorised JavaScript Origins and Authorised redirect URIs are also something useful but we will configure them later in the blog with a clear understanding. For now we will keep them empty and click “Create”.

9. We will have your Credentials created. We can now download the credentials as a secret JSON file from the home page.

   

   We have successfully setup our OAuth Client and have completed the first step.

Step 2: Install Necessary Python Packages

We will install the necessary packages required to create the FastAPI endpoint and secure it with google auth.

python-dotenv
mysql-connector-python
requests
fastapi
uvicorn
python-jose[cryptography]
authlib
pyjwt
itsdangerous
google-auth

Step 3: Set up the Project Structure

We will follow a proper project structure for building our authenticated API system. Below is the sample project structure we will follow.

The main chat endpoint will be present in the chat.py file. The authentication setup will be present in the auth.py file. These will be routers and accessed via the main FastAPI app in the api.py file.

If you want to know in detail about how FastAPI works, you can check this detailed blog on Beginner’s Guide to FastAPI and OpenAI Integration.

fastapi_auth_demo/
├── apis/
│   ├── chat.py
│   ├── auth.py
├── utils/
├── venv/
├── .env
├── .gitignore
├── api.py
├── client_secret.json
├── README.md
└── requirements.txt

Step 4: Configure Environment Variables

We will store sensitive credentials securely using environment variables. Create a .env file in the project directory and add:

GOOGLE_CLIENT_ID=<your-google-client-id>
GOOGLE_CLIENT_SECRET=<your-google-client-secret>
SECRET_KEY=<your-secret-key>
REDIRECT_URL=<your-redirect-url>
JWT_SECRET_KEY=<your-secret-key>
FRONTEND_URL=<your-frontend-url>

• You can directly get the GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET from the JSON file you downloaded in the first step.

• Keep the REDIRECT_URL and FRONTEND_URL empty for now and we will fill it with a proper value in the upcoming steps.

• The SECRET_KEY is a random string set by us, that is used to set the state parameter in the OAuth flow.

• The JWT_SECRET_KEY is a random string set by us, that is basically used as a key to encode and decode the access tokens that will be generated after authentication.

Step 5: Set Up Google OAuth Integration in FastAPI

The core functionality for Google authentication lies in integrating the OAuth2 flow into your FastAPI application. Below is the implementation:

Import Dependencies

Start by importing necessary modules and configuring OAuth:

# auth.py
from fastapi import FastAPI, Depends, HTTPException, status, Request, Cookie
from fastapi.responses import JSONResponse, RedirectResponse
from authlib.integrations.starlette_client import OAuth
from starlette.middleware.sessions import SessionMiddleware
from datetime import datetime, timedelta
from jose import jwt, ExpiredSignatureError, JWTError
from dotenv import load_dotenv
import os
import uuid
import traceback

# Load environment variables
load_dotenv(override=True)

# App Configuration
app = FastAPI()
app.add_middleware(SessionMiddleware, secret_key=os.getenv("FASTAPI_SECRET_KEY"))

# OAuth Setup
oauth = OAuth()
oauth.register(
    name="auth_demo",
    client_id=config("GOOGLE_CLIENT_ID"),
    client_secret=config("GOOGLE_CLIENT_SECRET"),
    authorize_url="https://accounts.google.com/o/oauth2/auth",
    authorize_params=None,
    access_token_url="https://accounts.google.com/o/oauth2/token",
    access_token_params=None,
    refresh_token_url=None,
    authorize_state=config("SECRET_KEY"),
    redirect_uri="http://127.0.0.1:8000/auth",
    jwks_uri="https://www.googleapis.com/oauth2/v3/certs",
    client_kwargs={"scope": "openid profile email"},
)

# JWT Configurations
SECRET_KEY = os.getenv("JWT_SECRET_KEY")
ALGORITHM = "HS256"

This what happens in this part of the code:

The oauth = OAuth() instance initializes the OAuth client. The oauth.register() method registers a new provider, in this case, Google. Here’s what each parameter does:

• name=”auth_demo”

  • This is a unique name assigned to this OAuth2 provider configuration. It can be referenced in the application when invoking authentication flows.

• client_id=config("GOOGLE_CLIENT_ID")

  • The client_id is fetched from configuration (usually environment variables). It identifies your application to Google’s OAuth2 system.

• client_secret=config("GOOGLE_CLIENT_SECRET")

  • The client_secret, also fetched from configuration, is a secret key provided by Google that verifies your application during the OAuth2 flow. It should remain confidential.

• authorize_url="https://accounts.google.com/o/oauth2/auth"

  • This URL is the endpoint where users are redirected to log in with Google. It starts the OAuth2 authorization flow.

• authorize_params=None

  • Any additional parameters for the authorization URL can be specified here. Since it’s None, no extra parameters are added.

• access_token_url="https://accounts.google.com/o/oauth2/token"

  • After the user logs in and grants permission, your application exchanges the authorization code for an access token at this endpoint.

• access_token_params=None

  • Specifies any additional parameters for obtaining the access token. Since it’s None, no extra parameters are included.

• refresh_token_url=None

  • If your application needs to refresh tokens, you would specify the refresh token endpoint. Here it’s None, indicating no token refresh functionality is configured.

• authorize_state=config("SECRET_KEY")

  • This optional parameter ensures the state parameter (used to prevent CSRF attacks) is securely managed. The SECRET_KEY provides added security.

• redirect_uri="http://127.0.0.1:8000/auth"

  • The redirect_uri is the URL to which Google redirects the user after a successful login. This must match the redirect URI configured in the Google Cloud Console. We will understand this in detail in the later part of the blog

• jwks_uri="https://www.googleapis.com/oauth2/v3/certs"

  • The JWKS URI (JSON Web Key Set URI) provides the keys needed to validate the tokens issued by Google.

• client_kwargs={"scope": "openid profile email"}

  • Defines the scopes (permissions) your application requests:

    • openid: Grants access to the user's identity.

    • profile: Provides access to the user’s basic profile details, such as name and profile picture.

    • email: Grants access to the user’s email address.

The second part of the code configures JWT, a token standard for securely transmitting information between parties.

1. SECRET_KEY = os.getenv("JWT_SECRET_KEY"):

   • Retrieves the secret key from the environment variables. This key is used to encode and decode JWT tokens.

   • It’s essential to keep this key secure because anyone with access to it can forge valid tokens.

2. ALGORITHM = "HS256":

   • Specifies the algorithm used for encoding and decoding JWT tokens. Here, HS256 (HMAC using SHA-256) is chosen, which is widely used for its balance of security and performance.

Utility Functions for JWT

We’ll use JWT (JSON Web Tokens) for managing user sessions securely:

def create_access_token(data: dict, expires_delta: timedelta = None):
    to_encode = data.copy()
    expire = datetime.utcnow() + (expires_delta or timedelta(minutes=30))
    to_encode.update({"exp": expire})
    return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)

def get_current_user(token: str = Cookie(None)):
    if not token:
        raise HTTPException(status_code=401, detail="Not authenticated")

    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
        return {"user_id": payload.get("sub"), "email": payload.get("email")}
    except ExpiredSignatureError:
        raise HTTPException(status_code=401, detail="Token expired")
    except JWTError:
        raise HTTPException(status_code=401, detail="Invalid token")

These are the two basic functions that is crucial.

1. create_access_token : This function takes the user data, and encodes it in a jwt token using the SECRET_KEY we have set in the .env file and the algorithm we have set. It also takes a parameter called expire which is the duration for which the access_token will be active. After this duration the access_token will expire and will not be a valid one to authenticate a user.

2. get_current_user: This function takes the access_token from the Cookie and decodes it to retrieve the information like user_id, email etc. Whenever an user goes through the login procedure, the access_token will be set as a browser cookie and can be directly accessed from there. We will see in the next part how this actually happens.

Login and Auth Endpoints

Create endpoints for logging in and authenticating users:

@router.get("/login")
async def login(request: Request):
    request.session.clear()
    referer = request.headers.get("referer")
    frontend_url = os.getenv("FRONTEND_URL")
    redirect_url = os.getenv("REDIRECT_URL")
    request.session["login_redirect"] = frontend_url 

    return await oauth.auth_demo.authorize_redirect(request, redirect_url, prompt="consent")

When we hit the endpoint /login, we will first fetch the referrer (the domain from where the endpoint is accessed or the request is received). Then we also fetch the FRONTEND_URL and REDIRECT_URL from the .env file. Now why are these required and what values to put in these variables is what we will discuss.

The google authentication process happens in the following way:

1. We send the request for login to google using our registered OAuth Client. This ensures the information about the user who is logging in is shared with us and they consent for the same.

2. Once that is done and the user consents and allow to share information, Google allows the login and attaches the information in an access token.

3. The control now moves to our end to fetch the access token, decode it and retrieve the information about the user and verify it with our database or system. Now the redirect_url is responsible to decide in which URL should Google forward the control to for this verification and other process. In our case, we will use another router endpoint called /auth which will handle these requirements and we will set our auth endpoint as the redirect_url in the environment variable

4. Once this is done we can safely say the user is authenticated and allow them access to the apis.

5. The FRONTEND_URL parameter decides that once the user logs in, gets verified, where can we redirect them to.

Let’s take the example of AI Demos Playground to understand this complete flow in a much better way.

When you visit, AI Demos Playground you will see the Login button at the top right

When you click the Login button, you will be redirected to the Google Auth Consent Page to consent sharing of information with this website. What actually happens in the backend is that, when the Login button is clicked, it calls the /login api endpoint that redirects you to the google consent screen.

Now once you complete the process and consent, the control moves to the /auth endpoint that fetches the access token, verifies it, fetches information and stores them in Database. After all this the the user is redirected to the FRONTEND_URL which in this case will be the AI Demos Playground page from where you did Login.

Now that you have got the idea of the entire flow, let’s move to the auth endpoint.

@router.route("/auth")
async def auth(request: Request):
    try:
        token = await oauth.auth_demo.authorize_access_token(request)
    except Exception as e:
        raise HTTPException(status_code=401, detail="Google authentication failed.")

    try:
        user_info_endpoint = "https://www.googleapis.com/oauth2/v2/userinfo"
        headers = {"Authorization": f'Bearer {token["access_token"]}'}
        google_response = requests.get(user_info_endpoint, headers=headers)
        user_info = google_response.json()
    except Exception as e:
        raise HTTPException(status_code=401, detail="Google authentication failed.")

    user = token.get("userinfo")
    expires_in = token.get("expires_in")
    user_id = user.get("sub")
    iss = user.get("iss")
    user_email = user.get("email")
    first_logged_in = datetime.utcnow()
    last_accessed = datetime.utcnow()

    user_name = user_info.get("name")
    user_pic = user_info.get("picture")

    if iss not in ["https://accounts.google.com", "accounts.google.com"]:
        raise HTTPException(status_code=401, detail="Google authentication failed.")

    if user_id is None:
        raise HTTPException(status_code=401, detail="Google authentication failed.")

    # Create JWT token
    access_token_expires = timedelta(seconds=expires_in)
    access_token = create_access_token(data={"sub": user_id, "email": user_email}, expires_delta=access_token_expires)

    session_id = str(uuid.uuid4())
    log_user(user_id, user_email, user_name, user_pic, first_logged_in, last_accessed)
    log_token(access_token, user_email, session_id)

    redirect_url = request.session.pop("login_redirect", "")
    response = RedirectResponse(redirect_url)
    response.set_cookie(
        key="access_token",
        value=access_token,
        httponly=True,
        secure=True,  # Ensure you're using HTTPS
        samesite="strict",  # Set the SameSite attribute to None
    )

    return response

1. Initial Token Exchange:

token = await oauth.auth_demo.authorize_access_token(request)

• This exchanges the authorization code (received from Google after user consent) for an access token

• It's the final step of the OAuth flow where Google confirms the user's authentication

2. Getting Additional User Info:

user_info_endpoint = "https://www.googleapis.com/oauth2/v2/userinfo"
headers = {"Authorization": f'Bearer {token["access_token"]}'}
google_response = requests.get(user_info_endpoint, headers=headers)
user_info = google_response.json()

• Makes a separate API call to Google to get more detailed user information

• Uses the access token to authorize this request

• Returns additional details like user's name and profile picture

3. Extracting User Details:

user = token.get("userinfo")
expires_in = token.get("expires_in")
user_id = user.get("sub")
iss = user.get("iss")
user_email = user.get("email")
user_name = user_info.get("name")
user_pic = user_info.get("picture")

• Extracts various user details from both the token and user info response

• sub is the unique Google user ID

• iss is the issuer (should be Google)

• Gets basic info like email, name, and profile picture

4. Security Validation:

if iss not in ["https://accounts.google.com", "accounts.google.com"]:
    raise HTTPException(status_code=401, detail="Google authentication failed.")

• Verifies that the token was actually issued by Google

• Important security check to prevent token forgery

5. Creating JWT Token:

access_token_expires = timedelta(seconds=expires_in)
access_token = create_access_token(
    data={"sub": user_id, "email": user_email}, 
    expires_delta=access_token_expires
)

• Creates a JWT token containing the user's ID and email

• Uses the same expiration time as the Google token

• This JWT will be used for subsequent API calls

6. Logging and Session Management:

session_id = str(uuid.uuid4())
log_user(user_id, user_email, user_name, user_pic, first_logged_in, last_accessed)
log_token(access_token, user_email, session_id)

• Generates a unique session ID

• Logs the user's information and token for tracking/auditing

• Records when the user first logged in and last accessed the system

Here are the log_token and log_user functions which are simple Database CRUD operations

def log_user(user_id, user_email, user_name, user_pic, first_logged_in, last_accessed):
    try:
        connection = mysql.connector.connect(host=host, database=database, user=user, password=password)

        if connection.is_connected():
            cursor = connection.cursor()
            sql_query = """SELECT COUNT(*) from users WHERE email_id = %s"""
            cursor.execute(sql_query, (user_email,))
            row_count = cursor.fetchone()[0]

            if row_count == 0:
                sql_query = """INSERT INTO users (user_id, email_id,user_name,user_pic,first_logged_in, last_accessed) VALUES (%s, %s, %s, %s, %s, %s)"""
                cursor.execute(sql_query, (user_id, user_email, user_name, user_pic, first_logged_in, last_accessed))

            # Commit changes
            connection.commit()

    except Error as e:
        print("Error while connecting to MySQL", e)
    finally:
        if connection.is_connected():
            cursor.close()
            connection.close()

def log_token(access_token, user_email, session_id):
    try:
        connection = mysql.connector.connect(host=host, database=database, user=user, password=password)

        if connection.is_connected():
            cursor = connection.cursor()

            # SQL query to insert data
            sql_query = """INSERT INTO issued_tokens (token, email_id, session_id) VALUES (%s,%s,%s)"""
            # Execute the SQL query
            cursor.execute(sql_query, (access_token, user_email, session_id))

            # Commit changes
            connection.commit()

    except Error as e:
        print("Error while connecting to MySQL", e)
    finally:
        if connection.is_connected():
            cursor.close()
            connection.close()
            logger.info("MySQL connection is closed")

7. Setting up Response:

redirect_url = request.session.pop("login_redirect", "")
response = RedirectResponse(redirect_url)
response.set_cookie(
    key="access_token",
    value=access_token,
    httponly=True,
    secure=True,
    samesite="strict",
)

• Gets the redirect URL that was stored before starting the OAuth flow

• Creates a response that will redirect the user back to the original page

• Sets the JWT token as an HTTP-only cookie with security flags:

  • httponly: Prevents JavaScript access to the cookie

  • secure: Cookie only sent over HTTPS

  • samesite="strict": Prevents CSRF attacks by only sending cookie to same site

Now that we have understood the entire login and authentication process, it is now time to revisit some of the sections above that we have kept as empty.

• While were registering the oauth client, we kept the redirect_uri parameter empty. We also kept the REDIRECT_URL parameter empty in the .env file. This will be assigned the url of our auth endpoint. The reason being after the login is successful from Google, it should again shift the control to the auth endpoint and we would be able to perform the necessary actions as discussed above. So it will be set to 127.0.0.1:8000/auth

• We will be testing the login functionality via tha FastAPI Swagger Docs. So suppose we run the server on the 8000 port. It can be accessed at http://127.0.0.1:8000/docs. Now we want that after login and authentication we should be redirected to this page only. So we will set the FRONTEND_URL as http://127.0.0.1:8000/docs.

  This is how the final .env file look like

•                   GOOGLE_CLIENT_ID=<your-google-client-id>
                    GOOGLE_CLIENT_SECRET=<your-google-client-secret>
                    REDIRECT_URL=http://127.0.0.1:8000/auth
                    JWT_SECRET_KEY="secret-key"
                    FRONTEND_URL=http://127.0.0.1:8000/docs
  

• In the Google Cloud Console we have kept two fields empty while creating our OAuth credentials: Authorised JavaScript Origins and Authorised redirect URIs.

  • The Authorised JavaScript Origins is nothing but the source from which you are trying to login and authorize with the help of Google. In our case, we will request login from our localhost so it will be set to the same: 127.0.0.1:8000

  • The Authorised redirect URIs are the list of URLs that are authorised, to which Google can hand over control after the login process is done. As we want Google to shift control to the auth endpoint, we have to explicitly mention that it is a authorised redirect url from our end, so that Google doesn’t block the request and allows it. If we don’t specify this, the request after login will be blocked as Google ensures the access token and sensitive information is handed over to authorised uris only.

    This is how your Credentials configuration should look like.

    

Step 6: Create the API endpoint and Secure it with Google Auth Dependency

For securing the endpoint, we will be required to add a Dependency on the endpoint parameters. This dependency will be the access token which will only be available post login. Let’s create the dependency first. We will create this dependency in the auth.py file.

def get_current_user(token: str = Cookie(None)):
    if not token:
        raise HTTPException(status_code=401, detail="Not authenticated")

    credentials_exception = HTTPException(
        status_code=401,
        detail="Could not validate credentials",
        headers={"WWW-Authenticate": "Bearer"},
    )
    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])

        user_id: str = payload.get("sub")
        user_email: str = payload.get("email")

        if user_id is None or user_email is None:
            raise credentials_exception

        return {"user_id": user_id, "user_email": user_email}

    except ExpiredSignatureError:
        # Specifically handle expired tokens
        traceback.print_exc()
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Session expired. Please login again.")
    except JWTError:
        # Handle other JWT-related errors
        traceback.print_exc()
        raise credentials_exception
    except Exception as e:
        traceback.print_exc()
        raise HTTPException(status_code=401, detail="Not Authenticated")

def validate_user_request(token: str = Cookie(None)):
    session_details = get_current_user(token)

    return session_details

We will now define the chat endpoint in the chat.py file.

from fastapi import Depends, APIRouter
from auth import get_current_user

router = APIRouter()

@router.get("/chat")
async def get_response(current_user: dict = Depends(get_current_user)):
    return {"message": "Welcome!", "user": current_user}

Step 7: Building the FastAPI App

Now that we have the authentication endpoint and chat endpoint ready, we can now setup our FastAPI app in the root of the project, in the api.py file.

import os
from fastapi import FastAPI, Header, HTTPException, Depends, Request
from starlette.config import Config
from fastapi.middleware.cors import CORSMiddleware
from starlette.middleware.sessions import SessionMiddleware
from apis import auth, chat
import time, requests

from dotenv import load_dotenv
load_dotenv(override=True)

config = Config(".env")

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],  
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
    expose_headers=["*"]
)

# Add Session middleware
app.add_middleware(SessionMiddleware, secret_key=config("SECRET_KEY"))

# # Logging time taken for each api request
@app.middleware("http")
async def log_response_time(request: Request, call_next):
    start_time = time.time()
    response = await call_next(request)
    process_time = time.time() - start_time
    logger.info(f"Request: {request.url.path} completed in {process_time:.4f} seconds")
    return response 

app.include_router(chat.router)
app.include_router(auth.router)

if __name__ == "__main__":
    import uvicorn
    import nest_asyncio
    nest_asyncio.apply()
    uvicorn.run(app, host="0.0.0.0", port=8000)

Step 8: Running and Testing the Application

Run your application using Uvicorn:

uvicorn api:app --reload

1. Login Endpoint: Visit /login to authenticate with Google.

2. Chat Endpoint: Use /chat to see if you can see your details after authentication.

   If everything was done correctly, you will be able to see your FastAPI function the same way as the demo below.

If you were able to setup your Google Auth, integrate it with FastAPI and authenticate your endpoints, then kudos. You have completely understood the process and created a simple FastAPI application backed by Google Authentication working!!

Step 9: Deploying your Application

Now that we have successfully setup our FastAPI application locally and able to integrated Google OAuth with it, its time to move one step ahead and deploy it on a VM. These APIs are the ones that will be consumed by the Frontend Application. So they have to be accessible and hence deploying our application is a crucial step.

You can check out this video Deploy FastAPI & Open AI ChatGPT on AWS EC2 to understand how to deploy your FastAPI application on an AWS EC2 instance. It also has a beginner-friendly YouTube tutorial that can be followed to get the FastAPI application up and running on EC2 VM.

Now that we have our FastAPI application up and running on the VM (for this tutorial assume we have the app running on 8000 port), it is time to move ahead and configure our domain and nginx that are some crucial steps in the completion of this backend deployment. As we are building this API to be consumed by the frontend, we have to make sure the frontend makes requests to our API and gets response.

Domain Mapping and SSL Certificate Generation

The first step in this configuration is to map our deployed endpoint to a domain and have a SSL certificate assigned to the same.

Why Our API Needs Domain Mapping and SSL

Modern browsers block direct IP access to protect users from potential security threats.

• Domain names provide a verifiable identity for your service. SSL certificates are issued to domains, not IPs, because domains can be validated through a chain of trust (Domain Registrar → Certificate Authority → Your Server). IPs can change hands frequently, making them unreliable for identity verification.

• Modern web browsers enforce HTTPS for security-critical features. They require:

  • A valid domain name (not an IP address)

  • An SSL certificate from a trusted authority

  • A match between the domain name and SSL certificate

Without these, browsers display warning messages and may block access entirely, especially for features like secure cookies, service workers, or HTTP/2 connections.

Consider accessing a bank's website. We trust https://mybank.com because:

• The domain ownership is verified

• The SSL certificate confirms you're connecting to the real bank

• Your data is encrypted during transmission

If the bank used just an IP (like https://193.168.1.1), we'd have no way to verify its authenticity, making it vulnerable to impersonation attacks.

Configuring this is very straightforward and fairly simple. We can follow these simple steps and we will be good to go:

1. The first step is to obtain a domain to which you will map your IP to. For example we already have a domain named, sample-auth-app.futuresmartai.com and this is mapped to the IP of our VM where the FastAPI application is hosted.

2. Now what we want is to ensure that our IP is associated with this domain name and any request coming to this domain, basically get transferred to our FastAPI application. To make this happen, we have to add some NGINX configuration in the VM where our FastAPI app is hosted.

   • Open your EC2 instance terminal and make sure to update and install nginx. You can run these two commands in the same order: sudo apt update, sudo apt install nginx . These make sure the libraries are updated and nginx is installed on your EC2 instance.

   • Move to the directory /etc/nginx/sites-available

     

   • Create a file for nginx configuration by running: sudo nano appconfig. This creates a file named appconfig and opens it in a VIM editor where we can directly put our configurations and save it.

     

   • Now we can directly put our configuration in this file. Below is the NGINX config we will be using

       server {
           listen 80;
           server_name sample-auth-app.futuresmartai.com;
     
           location / {
               proxy_pass http://127.0.0.1:8000; # Forward requests to the backend server.
               proxy_set_header Host $host; # Pass the original host header to the backend.
               proxy_set_header X-Real-IP $remote_addr; # Send the actual client IP to the backend.
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # Include client IPs in forwarded requests.
               proxy_set_header X-Forwarded-Proto $scheme; # Indicate the original request protocol (HTTP/HTTPS).
           }
       }
     

     💡

     These headers (Host, X-Real-IP, X-Forwarded-For) are critical for passing client IP addresses and cookies to the backend. Without them, backend logs will show the Nginx IP instead of the client’s IP, and client cookies may not work correctly.

   • We can put this in the appconfig file and save it using Ctrl + O and then press Enter. Finally press Ctrl + X to come out of the editor.

   • We can now see the config in our file using, cat appconfig

     

   • We now have to create a symbolic link using the nginx file in sites-available directory using: sudo ln -s /etc/nginx/sites-available/appconfig /etc/nginx/sites-enabled/. This is a best practice in Nginx server configuration because:

     1. 'sites-available' stores all our server configurations

     2. 'sites-enabled' only contains configurations that should be currently active

     3. The symbolic link allows to easily enable/disable sites without duplicating files

     4. When Nginx starts, it only reads configurations from 'sites-enabled'

   • Now once this is done, we have to test if the nginx config file has no errors. Run: sudo nginx -t

   • If the test is successful, reload the nginx configuration for the server to ensure these changes are applied using: sudo systemctl restart nginx

3. Thats it! Now we will be able to access our FastAPI application on the mapped domain: sample-auth-app.futuresmart.ai/docs

Common Mistakes

We have successfully built a FastAPI backend application using Google OAuth and deployed it to VM and mapped it to a certified domain name. When developing the frontend application that consumes these APIs, there are some common mistakes that tend to occur that can be avoided if the frontend and backend adjust these certain configurations:

Cross-Domain Cookie Access

This is one of the most trivial mistakes that tend to happen. In many of our applications, we have faced this issue and have identified it as a core mistake that needs to be taken care of during each application being developed.

As we have our FastAPI application hosted on an EC2 instance and mapped to domain. The Frontend application will similarly be deployed on an instance and mapped to a domain. But as the domain of the frontend and backend will be completely different, the cookie being shared becomes a problem. What I mean by this is that:

• When we are accessing the /login endpoint, we have seen that it verifies the identity via Google, creates an access token and sets it as a Cookie in the browser (more specifically in the domain).

• If the frontend sends a request to the /login endpoint, even after the entire authentication process, the token will be set as Cookie in the sample-auth-app.futuresmart.ai domain and not the domain where the frontend actually makes the request from

• The issue in this approach is that now the frontend has no access to the Cookie which is required to access the /chat endpoint and as well as get the user information

The solution to this is very simple and straightforward. We have to use a proxy from the frontend to ensure the Cookie is being set in the same domain from where the request is being sent.

• This has to be communicated to the frontend dev that on their end, they have to configure the nginx configuration such that the login request from the frontend is not directly made to the backend apis.

• They have to setup a proxy such that the request from frontend is sent to frontend domain/login endpoint and upon receiving that it should be sent to the actual backend api.

  • Let’s understand this thing in a simple manner. Suppose we have our frontend at sampe-auth-app-frontend.futuresmart.ai. So any request from this domain has to be made via sampe-auth-app-frontend.futuresmart.ai/login. In the proxy configuration it has to be set such that the request coming to sampe-auth-app-frontend.futuresmart.ai/login has to be then redirected to the actual backend api which is sampe-auth-app.futuresmart.ai/login.

  • In the backend, the REDIRECT_URL in the .env file as well as the oauth object also has to be changed to sampe-auth-app-frontend.futuresmart.ai/auth. This is to ensure that when the login endpoint redirects its authenticated result, it again goes via the frontend and then reaches the backend /auth route.

  • This will ensure that the communication always happens via the frontend and hence the Cookie will be set in the same domain from where the request is sent, which is sampe-auth-app-frontend.futuresmart.ai.

Cross-Origin Resource Sharing (CORS) Issue

This is another very common issue that we have faced in our projects. Mainly during testing locally, we tend to allow all origins to access the apis. But in production, we have to ensure we only allow the origins/domains that can access the apis explicitly. For example, in our case, we have to ensure that only the frontend domain can access our apis. So we have to set our CORS Middleware in the below way:

app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://sampe-auth-app-frontend.futuresmart.ai"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
    expose_headers=["*"]
)

This ensures our application is secure and prevented from any random access.

Conclusion

In this tutorial, we've built a FastAPI application and integrated it with Google Authentication. The system showcases advanced concepts like JWT in FastAPI, OAuth 2.0 Implementation, Cookie based session and user management - which we have already implemented and successfully delivered in numerous enterprise solutions.

This architecture provides several advantages:

1. Security: Combines industry-standard JWT tokens, secure cookies, and Google's robust authentication system to create a multi-layered security approach protecting user data and preventing common attack vectors.

2. Maintainability: Implements a clean, middleware-based architecture that centralizes authentication logic and makes it easy to protect new routes while keeping the codebase organized and manageable

3. Scalability: Leverages stateless JWT tokens and Google's infrastructure to create an authentication system that can easily scale across multiple servers and services without additional complexity.

If you found this guide helpful and want to explore more advanced backend development techniques, check out our other tutorials. If you want to know what skills truly matter to become a successful software developer or AI engineer, check out our blog on Must-Have Skills for Upcoming Software Developers and AI Engineers in 2025.

At FutureSmart AI, we specialize in building develop state-of-the-art AI solutions backed by scalable authentication systems and backend infrastructure for modern web applications. We've successfully implemented diverse authentication architectures across industries, from SaaS platforms to enterprise applications, while also specializing in AI integration by building numerous applications that incorporate Langchain, Langgraph, and ChatGPT through FastAPI frameworks.

For custom implementation support or consultations, feel free to reach out to us at contact@futuresmart.ai. For real-world examples of our work, visit our case studies where we showcase how our expertise in FastAPI, OAuth integration, and security best practices has helped businesses build robust authentication systems.

Stay tuned for our next tutorial in this series, where we'll dive into building the front-end for this system, accessing the API, and developing an end-to-end full-stack application!

Resources

In today’s fast-paced medical environment, healthcare professionals juggle patient care with administrative tasks that often slow them down. While tools like Cliniko have streamlined practice management—handling scheduling, patient records, and billing—there’s still room to make workflows even more efficient with custom AI-powered solutions.

At FutureSmart AI, we specialize in building AI-driven automation, NLP-based assistants, and workflow optimization solutions that integrate seamlessly with existing platforms. In this blog, we not only explore how to leverage Cliniko APIs with Python but also showcase how we helped a client build a custom AI solution on top of Cliniko, automating key processes for clinics and improving efficiency.

1. Understanding Cliniko APIs

What is Cliniko?

Cliniko is a cloud-based software designed for comprehensive medical practice management. It combines flexibility and ease of use to streamline daily operations. The platform excels in the following areas:

• Appointment scheduling via a customizable calendar.

• Patient management, including medical histories and communication records.

• Billing and invoicing with integrated payment tracking.

• Multi-location support for organizations operating across various sites.

API Overview

The Cliniko API enables developers to programmatically access and manipulate data. Here are some key capabilities:

• Retrieve, create, or update appointment records.

• Access detailed patient data.

• Manage practitioner and business information.

The API’s flexibility allows for custom integrations that cater to unique business workflows, making it a powerful tool for healthcare organizations.

2. Setting Up Access

1. Generate API Key

To generate a Cliniko API key follow below steps :

• Log into your Cliniko account.

• Navigate to My Info and then go to the Manage API Keys section.

• Click Add an API key to create a new one, which will allow you to interact with Cliniko's API.

2. Manage Permissions

After generating your API key, assign appropriate scopes based on the specific data you need access to. Each scope allows access to different parts of Cliniko’s system (appointments, patients, etc.). Be mindful to select only the required permissions for your needs.

3. Understand Shards

• Shards represent geographically isolated instances of Cliniko’s system. When you create a new account, you’ll select a region for hosting your data.

• Each shard is identified by a short code at the end of the API key (e.g., au1). This shard ID is crucial when constructing API URLs as it ensures proper routing of your requests to the correct server.

  Example API key :

MS0xLWl4SzYYYYdtR3V2HNOT..............-au1

In the above example, the shard is au1.

4. Identifying Your Application

To identify your application, you need to send the User-Agent header. In case of an issue, this will allow Cliniko to track down your requests and contact you. The format for this header should be:

APP_VENDOR_NAME (APP_VENDOR_EMAIL)

Where:

• APP_VENDOR_NAME is the name of your application.

• APP_VENDOR_EMAIL is the contact email for you or your company.

Example of a valid User-Agent header :

Really helpful app (contact@futuresmart.ai)

5. Consult Documentation

Cliniko’s https://docs.api.cliniko.com/ provides detailed information on available endpoints, parameters, and request examples. Reviewing this documentation will ensure you understand how to interact with Cliniko’s data programmatically.

3. Getting Started with Python

Setting Up the Environment

Start by preparing a Python environment. Here’s what you’ll need:

• Python 3.x

• IDE: PyCharm, VSCode, or Jupyter Notebook.

• Libraries: Install the necessary packages using pip:

    pip install requests
  

Authentication with Cliniko API

Cliniko uses API Key authentication for secure access. Below is a simple way to set it up:

import requests

key = "your_api_key_here"
shard = API_KEY[-3:]
headers={"Accept":"application/json","User-Agent":""} # Use your User-Agent
username = API_KEY  # Use API key as the username
password = ""

url = f"<https://api>.{shard}.cliniko.com/v1/bookings/"

response = requests.get(url, headers=headers, auth=(username,password))

if response.status_code == 200:
    bookings = response.json()
    print("Bookings fetched successfully:", bookings)
else:
    print(f"Failed to fetch bookings. Status code: {response.status_code}, Response: {response.text}")

4. Making Basic API Requests

Fetching Appointment Data :

url = f"<https://api>.{shard}.cliniko.com/v1/bookings"
response = requests.get(url, headers=headers, auth=(username,password))
if response.status_code == 200:
    appointments = response.json()
    print("Appointments fetched successfully:", appointments)
else:
    print("Failed to fetch appointments. Status code:", response.status_code)

Fetching Individual Appointment Data :

id = 
url = f"<https://api>.{shard}.cliniko.com/v1/bookings/{id}"
response = requests.get(url, headers=headers, auth=(username,password))
if response.status_code == 200:
    appointments = response.json()
    print("Appointments fetched successfully:", appointments)
else:
    print("Failed to fetch appointments. Status code:", response.status_code)

Retrieving Patient Records :

url = f"<https://api>.{shard}.cliniko.com/v1/patients"
response = requests.get(url, headers=headers, auth=(username,password))
if response.status_code == 200:
    patients = response.json()
    print("Patient records fetched successfully:", patients)
else:
    print("Failed to fetch patient records. Status code:", response.status_code)

5. Advanced API Operations

Filtering and Query Parameters

Cliniko allows the use of powerful query parameters to filter data based on specific conditions. By using the q[] parameter, you can fine-tune your queries for appointments, patients, and other resources. Here are some examples of how to leverage this functionality.

1. Filtering Appointments by Date Range

To fetch appointments within a specific date range, you can apply multiple filters using the q[] parameter for conditions like "greater than" or "less than." For example, retrieving appointments that start after a certain date:

query = {
    "page": "0",
    "per_page": "1",
    "sort": "created_at:desc",
    "q[]": "starts_at:>2014-03-04T20:37:17Z",  # Start date filter
    "order": "asc"
}
url = f"<https://api>.{shard}.cliniko.com/v1/individual_appointments"
response = requests.get(url, params=query, headers=headers, auth=(username,password))

if response.status_code == 200:
    appointments = response.json()
    print("Filtered appointments:", appointments)
else:
    print("Failed to fetch filtered appointments. Status code:", response.status_code)

2. Using Wildcards for Flexible Search

You can perform more complex searches using wildcards by employing the ~~ symbol. For instance, to search for patients whose last names include a variation of "johnson":

query = {
    "page": "0",
    "per_page": "10",
    "q[]": "last_name:~~ja%on%",
    "order": "asc"
}
url = f"<https://api>.{shard}.cliniko.com/v1/patients"
response = requests.get(url, params=query, headers=headers, auth=(username,password))

if response.status_code == 200:
    patients = response.json()
    print("Filtered patients:", patients)
else:
    print("Failed to fetch filtered patients. Status code:", response.status_code)

3. Filtering Archived Records

To fetch archived records, you can filter with archived_at like this:

query = {
    "page": "0",
    "per_page": "10",
    "q[]": "archived_at:*",  # Fetch all records including archived
    "order": "asc"
}
url = f"<https://api>.{shard}.cliniko.com/v1/individual_appointments"
response = requests.get(url, params=query, headers=headers, auth=(username,password))

if response.status_code == 200:
    appointments = response.json()
    print("Archived appointments:", appointments)
else:
    print("Failed to fetch archived appointments. Status code:", response.status_code)

4. Sorting and Ordering Results

By default, results are ordered by the created_at field in ascending order. If you want to sort based on a different field, such as starts_at, and specify the order, you can use the sort and order parameters:

query = {
    "page": "0",
    "per_page": "10",
    "sort": "starts_at,created_at:desc",  # Sort by start time, then creation time
    "q[]": "starts_at:>2014-03-04T20:37:17Z",  # Example filter
    "order": "desc"
}
url = f"<https://api>.{shard}.cliniko.com/v1/individual_appointments"
response = requests.get(url, params=query, headers=headers, auth=(username,password))

if response.status_code == 200:
    appointments = response.json()
    print("Sorted appointments:", appointments)
else:
    print("Failed to fetch sorted appointments. Status code:", response.status_code)

Pagination

Large datasets are paginated. Use the next and prev links to navigate:

url = f"<https://api>.{shard}.cliniko.com/v1/bookings"
while url:
    response = requests.get(url, headers=headers, auth=(username,password))
    if response.status_code == 200:
        data = response.json()
        for record in data['records']:
            print(record)
        url = data.get('next')
    else:
        print("Failed to fetch data. Status code:", response.status_code)
        break

Error Handling in API Requests

To ensure your application handles API errors smoothly, follow this simple structure:

try:
    response = requests.get(url, headers=headers, auth=(username, password))
    response.raise_for_status()  # Checks for any HTTP error
    data = response.json()
except requests.exceptions.HTTPError as err:
    print(f"HTTP error occurred: {err}")
except Exception as e:
    print(f"An error occurred: {e}")

HTTP Response Codes :

• 2xx: Success – Everything worked as expected.

• 4xx: Client error – There was an issue with the request (e.g., missing required parameters).

• 5xx: Server error – Something went wrong on the Cliniko server's side.

6. Challenges You Can Face When Using Cliniko APIs in Python

1. Authentication and Authorization Issues

Managing API keys securely is essential to avoid access issues or security vulnerabilities. Each API key is associated with a specific shard—the geographical server where your data is hosted. When making requests, ensure that the API key includes the correct shard information in the request, such as appending the shard ID to the API endpoint to direct the request to the correct server. Expired or invalid keys can lead to authentication failures, so regularly refresh and monitor your API keys.

2. API Deprecation or Changes

Cliniko may update or discontinue API endpoints, introduce new versions, or change response formats, causing compatibility issues. Regularly monitoring the API documentation and release notes helps ensure your integration stays up to date.

7. Custom Solutions for our client Abby

Fine-Tuning Intent Classification Models for Abby

Abby is an AI-powered solution designed to streamline appointment confirmations in healthcare operations through seamless integration with the Cliniko API. The client required us to develop a secure, stand-alone system that does not rely on broad-based language models, ensuring data privacy and adherence to stringent security standards.Available as a Chrome extension for Cliniko users, Abby uses a proprietary AI language model to analyze inbound responses, updating appointment statuses directly within the Cliniko calendar. With over 98% accuracy, it reliably handles most message intents, including emojis, while ongoing refinements address edge cases like conflicting responses.

Key Features:

• AI-Powered SMS Analysis: Automatically interpret patient responses and instantly update appointment statuses, focusing on what requires action.

• Enhanced Visuals for Cliniko: Temporary color-coded status markers in your Cliniko calendar make it easy to spot unconfirmed appointments.

• Integrated Appointment Notes: Abby posts SMS responses directly to appointment notes, keeping all communication in one place.

• No Learning Curve: A single click provides access to real-time appointment status without disrupting your workflow.

By automating routine tasks, Abby optimizes healthcare operations, improving patient experience while saving valuable administrative time—without compromising on security or patient confidentiality.

Check out Abby here : https://www.abby.clinic/.

8. AI-Powered Innovations for Healthcare

We specialize in developing custom AI solutions that seamlessly integrate with existing platforms like Cliniko, enabling clinics and healthcare businesses to automate processes, enhance patient interactions, and extract actionable insights from data.

Here’s how our AI expertise can benefit your practice:

✅ RAG-Based AI Chatbots for Clinics – AI-powered chatbots capable of retrieving patient information, answering FAQs, and assisting with appointment scheduling based on real-time Cliniko data.

✅ NL2SQL for Healthcare Analytics – Query your Cliniko database using natural language and extract structured insights without writing complex SQL queries.

✅ Document Parsing with Vision LLMs – Process invoices, prescriptions, medical records, and insurance forms using advanced AI models that can accurately extract key details from scanned documents.

✅ Automated Patient Reports & Summaries – Use AI to generate structured patient reports, summarizing appointment details, lab results, and doctor notes, reducing manual effort.

✅ AI Agents for Administrative Tasks – Automate routine clinic operations, such as appointment confirmations, follow-ups, and billing inquiries, reducing workload and improving patient experience.

By combining Generative AI, NLP, and advanced data processing techniques, we create solutions that improve efficiency, reduce administrative burden, and optimize patient management.

9. Why Work With Us?

At FutureSmart AI, we bring a wealth of experience in Cliniko APIs, Python development, and cutting-edge AI solutions tailored for the healthcare industry.

🚀 Cliniko Expertise – Deep understanding of Cliniko APIs, enabling seamless integrations and automation.
🔍 AI-Driven Efficiency – We build secure, privacy-compliant AI solutions that enhance decision-making and reduce operational overhead.
📈 Custom-Tailored Solutions – Whether you need chatbot automation, document parsing, or AI-driven analytics, we create bespoke AI applications to fit your unique needs.
🔒 Security & Compliance – Our solutions adhere to industry best practices, ensuring patient data confidentiality and regulatory compliance.

Want to automate your clinic operations or build a custom AI-powered healthcare solution?
📩 Contact us at contact@futuresmart.ai to discuss how we can help!

10. Conclusion

Cliniko’s API ecosystem, combined with Python and AI, unlocks powerful opportunities for automating clinic workflows, streamlining data processing, and enhancing patient engagement. Whether it's AI-powered chatbots, document automation, or analytics-driven insights, FutureSmart AI can help you build scalable, intelligent solutions tailored to your needs.

Let's transform healthcare operations with AI. 🚀

You need solid fundamental skills in coding, APIs, Databases and Communications to thrive in the rapidly evolving tech landscape. This holds true whether you aim to become a traditional software developer or you’re intrigued by the cutting edge of Generative AI. If you’re focused on learning the “shiny” new AI tools without first mastering the core foundations, you’ll struggle to deliver real impact.

Being an Expert-Vetted (Top 1%) on Upwork with over $300K+ earned and a 100% Job Success rate has further reinforced my understanding of what skills truly matter in the industry.

Below, I’ll walk through the essential skills that, in my experience, every aspiring developer or AI engineer should prioritize.

1. Why Practical Skills Matter

Many newcomers are drawn to the excitement of advanced AI concepts—like Generative AI, LLM’s, RAG (Retrieval Augmented Generation), or vector databases. While these innovations are undoubtedly game-changing, they can overshadow the day-to-day coding, debugging and deployment tasks that keep a project running smoothly.

Think about it: Even the most cutting-edge AI system is useless if it can’t be integrated into a functional application. That requires reliable code, proper database connections, and well-structured APIs. It’s why companies—when hiring interns or fresh grads—emphasize practical software development skills just as much as familiarity with new AI trends.

Key point: Whether your goal is traditional software development or specialized AI engineering, you’ll stand out by showing you can handle the basics and add value to the team and not liability. Mastering these fundamentals will make it far easier to adopt advanced AI tools effectively.

2. Embracing Self-Learning & Adaptability

In a world where ChatGPT and YouTube tutorials are just a click away, memorizing syntax is less important than knowing how to discover answers and adapt them to your specific needs. When you face a new challenge—be it connecting a database or troubleshooting an API route—your ability to learn on the fly is what truly sets you apart.

• Resourcefulness Over Rote Memorization: Whether you’re copying a snippet from Stack Overflow or ChatGPT, your real value is in customizing and debugging that code for your application.

• Hands-On Experience: Reading documentation is great, but actually breaking things and fixing them is how you learn effectively.

• Continuous Upskilling: The AI landscape evolves quickly. If you want to keep up, you need to be comfortable teaching yourself new frameworks and libraries.

3. Mastering Practical Python

While you don’t need to be a “Python guru” or a competitive programming champion, you do need to write clear, functional code that gets the job done. Here’s what matters most:

• Use the Right Data Structures: Know when to use lists for ordered data, dictionaries for key-value pairs, and tuples for lightweight groupings.

• Organize Your Code with Functions and Files: This makes your code more readable, reusable, and easier to debug. keep files small instead of writing long scripts.

• Handle Files and Data: Most real-world software and AI projects involve ingesting or cleaning data. Be comfortable working with CSV, JSON, or other common formats.

Common Pitfalls to Avoid:

❌ Writing overly complex, hard-to-read code.

❌ Ignoring error handling—always use try-except blocks.

❌ Copy-pasting from Stack Overflow or ChatGPT without understanding the solution.

❌ Writing long, unstructured Python files instead of modular, function-based code split into small files.

Hands-on Challenge:

Write a Python script that reads a JSON file, processes it, and writes an updated file. Keep the script modular by splitting different functionalities into separate functions and files.

In the professional world, clarity and reliability are often more valuable than clever but cryptic code. Once your Python basics are solid, you’ll be able to pick up new libraries or advanced frameworks with minimal fuss.

4. Working with APIs

The ability to consume and provide APIs is crucial in modern software development—whether you’re building e-commerce backends, AI-driven chatbots, or anything in between.

API Consumption

Most applications communicate via RESTful APIs. Know how to:

✅ Send requests (GET, POST, PUT, DELETE).

✅ Handle authentication (e.g., bearer tokens, API keys).

✅ Parse JSON responses (often deeply nested).

Handling Authentication & Security

Expect to encounter JWT (JSON Web Tokens), cookies, and CORS (Cross-Origin Resource Sharing). Understanding these ensures your app communicates securely and efficiently.

Working with JSON

APIs typically return JSON, so you’ll deal with Python dictionaries. Knowing how to navigate nested structures and handle edge cases will save you hours of troubleshooting.

Practical Use Cases

• Fetch data from a public APIs (e.g., weather, currency exchange), transform the response, and store it for analytics.

5. Essential SQL and Database Operations

No matter what you build, you need a reliable way to store and retrieve data. SQL databases like MySQL and PostgreSQL are still core technologies in production environments.

• Basic SQL Queries: Learn how to SELECT, INSERT, UPDATE, and DELETE. These cover most interactions with a relational database.

• Connecting with Python: Use libraries like psycopg2 or mysql-connector-python to integrate your Python application with the database.

Real-World Example

Many LLMs don’t persist previous user inputs once an API call finishes. To maintain conversation history (for a chatbot or a Generative AI assistant):

1. Assign a session ID for each conversation.

2. Store all messages (both user and AI) in a SQL database.

3. For subsequent queries, retrieve the conversation history using that session ID.

4. Pass the history to the LLM for context, then store the new response.

This simulates “memory” and vastly improves user experience.

Practice Task:

Create a simple database that stores user messages with timestamps and session ids. Design a query to retrieve past messages for a specific user session.

6. Creating Your Own APIs with FastAPI

Even if you’re not building complex AI services, knowing how to serve any functionality via an API is invaluable. FastAPI is a popular choice in Python circles due to its simplicity and asynchronous capabilities.

Why FastAPI?

• Speed and Simplicity: An asynchronous framework that makes handling concurrent requests more efficient.

• Auto-Generated Documentation: Teammates and clients can easily understand and test your endpoints.

• Widely Used in AI & ML: FastAPI has become a go-to for deploying machine learning models quickly.

Key Concepts

• Endpoints (Routes): Clearly define the URLs (e.g., /predict) for specific functionalities.

• Sync vs. Async: Leverage asynchronous functions for better performance when scaling out.

• Data Validation: Validate incoming JSON data before processing to prevent runtime errors.

Mini Project:

Build a FastAPI service that accepts text input and returns a sentiment score using an NLP model.

6. Getting Comfortable with Basic Deployment

Whether you’re a back-end engineer or a budding AI Engineer, your application needs to be accessible to others to deliver real value. That’s where deployment comes in.

Why Deployment Matters

• Accessibility: A deployed app or API can be accessed by anyone with the right permissions.

• Real-World Feedback: Live usage data and metrics guide iterative improvements.

• Collaboration: Demonstrates you understand the full development lifecycle, not just coding.

Common Deployment Options

1. Cloud VMs: AWS, Azure, or Google Cloud let you rent servers and install your software stack.

2. PaaS (Platform as a Service): Services like Heroku or Render manage much of the infrastructure for you.

3. Company Infrastructure: Internships often provide access to enterprise-grade tools, a great way to learn hands-on without personal costs.

Hands-on Task:

Deploy your FastAPI app to Render or a cloud VM and access it via a public URL.

7. Where Advanced AI Concepts Fit In

Once you’ve nailed down the basics, exploring advanced AI topics becomes far more rewarding. Whether it’s Generative AI, RAG, vector databases, or frameworks like LangChain, these cutting-edge tools are best leveraged when you can integrate them seamlessly into real applications.

Integration Over Isolation

• RAG (Retrieval Augmented Generation): Fetch relevant context from a vector database, then pass it to an LLM.

• Vector Databases: Specialized for embedding-based searches, often used in semantic search or question-answering.

• LangChain (and Similar): Frameworks that streamline building LLM-powered applications.

Real-World Example:

Suppose you're developing a document search tool for legal professionals. Using RAG and vector databases allows efficient retrieval of case laws based on user queries, improving productivity compared to traditional keyword-based search. and legal precedents for lawyers. It uses RAG to fetch the right references and LLMs to summarize them.

Applied AI Use Case:

Imagine you're building an AI-powered FAQ chatbot. Instead of just generating random responses, you: 1️⃣ Retrieve relevant FAQs from a vector database.

2️⃣ Pass that context to an LLM.

3️⃣ Generate a response using the AI model.

4️⃣ Deliver the output via an API.

The entire system depends on strong coding, API, and database skills.

Tip: Instead of jumping straight to LangChain, try manually implementing a RAG pipeline to understand how everything connects.

8. Conclusion & Next Steps

After years of working I can confidently say these foundational skills are what truly enable success—whether you’re a software developer, a data scientist, or an AI specialist. I’ve interviewed and mentored many candidates, and the ones who excel are those who can both understand advanced AI concepts and handle the essential coding and deployment work.

Key Takeaways

• Practice & Portfolio: Build small, complete projects that demonstrate your ability to write clean Python code, work with APIs, interact with a database, and (ideally) deploy your application.

• Seek Internships & Mentorship: Real-world experience accelerates learning. You’ll gain exposure to production environments and valuable feedback from peers or senior developers.

• Explore Advanced AI Topics: Once you have the basics locked in, frameworks like LangChain or vector databases can add significant value to your skill set.

• Stay Curious & Adaptable: Technology moves fast—those who keep learning and adapting will always stay relevant.

Final Thought:

The world of tech—especially AI—is growing at an unprecedented pace. Make sure you have the fundamentals in place so you can ride that wave rather than getting washed away by it. With a strong foundation, you’ll be ready to contribute meaningfully to any team, whether you’re a software developer, data engineer, or the next generative AI whiz.

For real-world examples of our work, take a look at our case studies.

At FutureSmart AI, we've helped many clients improve their support systems with AI customer service solutions. Now, we're sharing our proven approach to creating a custom AI assistant for customer service using Freshdesk. This tutorial shows you how to build an AI customer service chatbot that delivers real results – from translating customer queries into SQL commands to generating smart responses. Having deployed similar solutions across various industries, we know exactly what works. By the end of this guide, you'll have a fully functional AI customer support agent ready to streamline your support operations.

Building on the foundational concepts covered in our previous LangGraph tutorial series, where we explored creating simple AI tools for customer service using nodes and edges, this guide takes it a step further. You'll learn how to develop an agent that seamlessly:

• Translates natural language into precise SQL queries

• Interacts with Freshdesk APIs to manage tickets

• Taps into knowledge bases for accurate responses

• Retrieves real-time order status and product details

• Handles shipping and return policy inquiries

• Generates context-aware responses to customer queries

Drawing from our years of implementing AI solutions, we've distilled the essentials of building a robust Freshdesk chatbot that truly delivers results. This tutorial combines real-world applications with step-by-step guidance to help you create an AI-powered chatbot capable of retrieving ticket details, executing database queries, and enhancing customer service automation.

Setting up the Environment

!pip install -U langchain langchain-chroma pypdf sentence-transformers langgraph langchain_openai langchain_community requests os

Setting up API Keys

Before diving into building your Freshdesk AI integration services, it’s crucial to set up your API keys. These keys allow your agent to interact securely with external environments and tools like Freshdesk and OpenAI GPT models. Without them, the tools cannot function effectively.

import getpass
import os

def _set_env(var: str):
    if not os.environ.get(var):
        os.environ[var] = getpass.getpass(f"{var}: ")

_set_env("FRESHDESK_API_KEY")
_set_env("OPENAI_API_KEY")
_set_env("FESHDESK_URL")

Creating the LLM Object

Here’s how to initialize the LLM using LangChain ChatOpenAI

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model_name="gpt-4o")

Summary Tool Integration

Automated customer support often requires summarizing ticket messages quickly. This tool fetches and processes data via Freshdesk APIs to generate concise summaries.

1. Get Ticket Message: Function that gets Ticket Data through Freshdesk API’s given ticket_id

    import requests
    import os
    FRESHDESK_API_KEY = os.getenv("FRESHDESK_API_KEY")
    FESHDESK_URL = os.getenv("FESHDESK_URL")
   
    def get_messages(ticket_id):
        url = f'https://{FESHDESK_URL}.freshdesk.com/api/v2/tickets/{ticket_id}?include=conversations'
        headers = {
            'Content-Type': 'application/json'
        }
        auth = (FRESHDESK_API_KEY, 'X')  # Replace 'X' with your actual API key
        response = requests.get(url, headers=headers, auth=auth)
        return response.json()
   

2. Process Message:

    def process_messages(messages):
        subject = messages['subject']
        description = messages['description_text']
        conversation = messages['conversations']
        messages = ""
        messages += "Subject : " + subject + "\n"
        messages += "User query : \n" + description + "\n"
        for message in conversation:
            if message['incoming']:
                messages += "User Query : \n" + message['body_text'] + "\n"
            else:
                messages += "Agent Response : \n" + message['body_text'] + "\n"
        return messages
   

3. Summary Chain: This method consolidates all the input Ticket Messages into a single prompt, which the LLM then processes.

    prompt = ChatPromptTemplate.from_messages(
         [("system", """FutureSmart TechStore is committed to being the leading destination for technology enthusiasts worldwide by combining quality, value, and exceptional service. At FutureSmart TechStore, we don’t just sell products—we provide solutions that enhance your lifestyle and work environment. We aim to offer great value to our customers through a range of promotions and discounts, ensuring you get the best deals on the latest technology products.
   
        Our collection includes a wide range of tech products designed to meet the needs of various customers, Our collection includes a wide range of tech products such as Smart Home Solutions (intelligent lighting systems, advanced security devices), Wearable Technology (fitness trackers, smartwatches), Mobile Accessories (chargers, cases, screen protectors), and Computing Peripherals (keyboards, mice, storage devices), designed to meet the needs of various customers, from tech enthusiasts to everyday users.
   
        If the query is not in English, please identify the language and generate a summary of the conversation in English. The response should be generated in the same language as the query.Also mention order id and customer name if present.\\n\\n{context}. Summarize:""")]
        )
   
    # Instantiate chain
    chain = prompt | llm
    result = chain.invoke({"context": context})
   

4. Build a Summary Tool: Define a tool that generates ticket summary

    from langchain.tools import tool
    from pydantic import BaseModel
   
    class SummarizeToolSchema(BaseModel):
        ticket_id: int
   
    @tool(args_schema=SummarizeToolSchema)
    def summarize_tool(ticket_id: int):
        """Tool to Summarize Ticket information based on ticket_id provided
        `ticket_id`: int"""
        print("INSIDE SUMMARIZE NODE", ticket_id)
        # Invoke chain
        data = get_messages(ticket_id)
        context = process_messages(data)
        print(context)
        result = chain.invoke({"context": context})
        return f"ticket_id: {ticket_id}", f"Ticket Summary: {result.content}"
   

5. Testing the Tool

    summarize_tool.invoke({"ticket_id":130})
   

    # output
    INSIDE SUMMARIZE NODE 130
    'The customer is inquiring about the status of their order #12345, as UPS lost the package. They understood that a new package would be sent immediately, but nearly a month has passed with no update. They are asking for information on what is happening.'
   

Agentic RAG Tool Integration

The Retrieval-Augmented Generation (RAG) enhances your AI agent by enabling it to fetch relevant documents and deliver accurate, context-rich responses. Whether you're developing a RAG chatbot, building RAG pipelines, or exploring RAG for enterprise, this guide will walk you through the integration process step-by-step.

Why Choose RAG?

1. Local Knowledge Integration: RAG uses local knowledge to customize responses based on private datasets.

2. Post-Generation Customization: Tweak outputs with RAG post-generation techniques or set a customized reply output.

Steps to Integrate the RAG Tool

1. Load Documents: Use the PyPDFLoader and Docx2txtLoader to load documents for your RAG app.

 from langchain_community.document_loaders import PyPDFLoader, Docx2txtLoader

 def load_documents(folder_path: str) -> List[Document]:
     documents = []
     for filename in os.listdir(folder_path):
         file_path = os.path.join(folder_path, filename)
         if filename.endswith('.pdf'):
             loader = PyPDFLoader(file_path)
         elif filename.endswith('.docx'):
             loader = Docx2txtLoader(file_path)
         else:
             print(f"Unsupported file type: {filename}")
             continue
         documents.extend(loader.load())
     return documents

 folder_path = "/docs"
 documents = load_documents(folder_path)
 print(f"Loaded {len(documents)} documents from the folder.")

2. Split Text into Chunks: Prepare documents for vectorization by splitting them into manageable chunks, which is essential for rag.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
     chunk_size=1000,
     chunk_overlap=200,
     length_function=len
)

 splits = text_splitter.split_documents(documents)
 print(f"Split the documents into {len(splits)} chunks.")

3. Generate Embeddings: Use SentenceTransformers to create embeddings for efficient similarity searches in your RAG pipeline.

from langchain_community.embeddings import HuggingFaceEmbeddings

model_name = "sentence-transformers/all-MiniLM-L6-v2"
model_kwargs = {'device': 'cpu'}
encode_kwargs = {'normalize_embeddings': False}
embedding_function = HuggingFaceEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)

4. Create and Persist a Vector Store:

   Store embeddings for future retrieval with Chroma.

    from langchain_chroma import Chroma
   
    collection_name = "my_collection"
    vectorstore = Chroma.from_documents(
         collection_name=collection_name,
         documents=splits,
         embedding=embedding_function,
         persist_directory="./chroma_db"
    )
   

5. Build the Retriever Tool:

   Implement a retriever to fetch semantically similar documents for user queries. This tool powers RAG with action reply in specific format or customized responses.

   
    from langchain.tools import tool
    from pydantic import BaseModel
    from langchain_chroma import Chroma
   
    class RagToolSchema(BaseModel):
        question: str
   
    @tool(args_schema=RagToolSchema)
    def retriever_tool(question: str):
        """Tool to Retrieve Semantically Similar documents to answer User Questions
        related to FutureSmart Tech Store and its Processess.
   
        `question`: str
        """
        print("INSIDE RETRIEVER NODE", question)
   
        vectorstore = Chroma(
            collection_name="my_collection",
            embedding_function=embedding_function,
            persist_directory="./chroma_db"
        )
   
        retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
        retriever_result = retriever.invoke(question)
   
        context = "\n\n".join(doc.page_content for doc in retriever_result)
        return f"(context: {context})"
   

   This tool allows your AI agent to retrieve relevant chunks of information from your document database, making it highly effective for knowledge-based tasks.

You can test the retriever right now to see how it is performing

retriever = vectorstore.as_retriever(search_kwargs={"k": 2})
# pass question
retriever_tool.invoke({"question": "what are different payment methods supported"})
print(retriever_results)

INSIDE RETRIEVER NODE what are different payment methods supported
'At FutureSmart TechStore, weaimtomakeyourshoppingexperienceasseamlessandsecureaspossible. Wesupport avarietyof payment methodstosuit yourconvenience.\nSupportedPaymentMethods\n1. CreditandDebitCards○ Visa○ Mastercard○ AmericanExpress○ RuPay2. DigitalWallets○ Paytm○ PhonePe○ GooglePay○ AmazonPay3. NetBanking○ Supportedforall majorIndianbanksincludingHDFC, ICICI, SBI, AxisBank, andmore.4. UPIPayments○ SeamlesspaymentsusingUnifiedPaymentsInterface(UPI).5. CashonDelivery(COD)○ Availableforselect locationsinIndia.○ MaximumCODlimit: INR10,000.6. EMIOptions○ No-cost EMI andstandardEMI optionsareavailableformajorcredit cards.○ EMI optionsdependonthebankandcardtype.7. BankTransfers○ Direct banktransfersaresupportedforbulkorbusinesspurchases.\nPaymentSecurity\n● All onlinetransactionsareprocessedthroughsecureandencryptedgatewaystoensurethesafetyof yourinformation.● Wecomplywiththelatest PCI DSS(Payment CardIndustryDataSecurityStandard)guidelines.\nImportantNotes\n\nAt FutureSmart TechStore, weaimtomakeyourshoppingexperienceasseamlessandsecureaspossible. Wesupport avarietyof payment methodstosuit yourconvenience.\nSupportedPaymentMethods\n1. CreditandDebitCards○ Visa○ Mastercard○ AmericanExpress○ RuPay2. DigitalWallets○ Paytm○ PhonePe○ GooglePay○ AmazonPay3. NetBanking○ Supportedforall majorIndianbanksincludingHDFC, ICICI, SBI, AxisBank, andmore.4. UPIPayments○ SeamlesspaymentsusingUnifiedPaymentsInterface(UPI).5. CashonDelivery(COD)○ Availableforselect locationsinIndia.○ MaximumCODlimit: INR10,000.6. EMIOptions○ No-cost EMI andstandardEMI optionsareavailableformajorcredit cards.○ EMI optionsdependonthebankandcardtype.7. BankTransfers○ Direct banktransfersaresupportedforbulkorbusinesspurchases.\nPaymentSecurity\n● All onlinetransactionsareprocessedthroughsecureandencryptedgatewaystoensurethesafetyof yourinformation.● Wecomplywiththelatest PCI DSS(Payment CardIndustryDataSecurityStandard)guidelines.\nImportantNotes'

NL2SQL Tool Integration

We now have the Summarization and RAG tools ready for the AI Agent, leaving only the NL2SQL tool to be built.

The SQL Agent serves as a bridge between natural language and SQL databases by generating and executing SQL queries from user questions. This allows the AI Agent to efficiently handle and answer database-related queries.

Steps to Integrate the NL2SQL Tool

We're using a MySQL database containing an e-commerce dataset to test SQL queries. The dataset includes all Product and Order tables.

1. Initialize the Database Connection
   First, establish a connection with the LangChain SQL Database utility.

    from langchain_community.utilities import SQLDatabase
    import os
    db_user = os.getenv("DB_USER")
    db_password = os.getenv("DB_PASSWORD")
    db_host = os.getenv("DB_HOST")
    db_name = os.getenv("DB_NAME")
   
    db = SQLDatabase.from_uri(f"mysql+pymysql://{db_user}:{db_password}@{db_host}/{db_name}",sample_rows_in_table_info=1)
   

2. Clean SQL Queries
   This function is very important. We often see this problem in many of our client’s projects that the SQL query generated from the LLM consists of some unnecessary symbols, texts, backticks, etc. As a result, you will get an error while executing this query. Thus, we require a function for error-free text-to-SQL conversion.

    import re        
   
    def clean_sql_query(text: str) -> str:
   
        # Step 1: Remove code block syntax and any SQL-related tags
        # This handles variations like ```sql, ```SQL, ```SQLQuery, etc.
        block_pattern = r"```(?:sql|SQL|SQLQuery|mysql|postgresql)?\s*(.*?)\s*```"
        text = re.sub(block_pattern, r"\1", text, flags=re.DOTALL)
   
        # Step 2: Handle "SQLQuery:" prefix and similar variations
        # This will match patterns like "SQLQuery:", "SQL Query:", "MySQL:", etc.
        prefix_pattern = r"^(?:SQL\s*Query|SQLQuery|MySQL|PostgreSQL|SQL)\s*:\s*"
        text = re.sub(prefix_pattern, "", text, flags=re.IGNORECASE)
   
        # Step 3: Extract the first SQL statement if there's random text after it
        # Look for a complete SQL statement ending with semicolon
        sql_statement_pattern = r"(SELECT.*?;)"
        sql_match = re.search(sql_statement_pattern, text, flags=re.IGNORECASE | re.DOTALL)
        if sql_match:
            text = sql_match.group(1)
   
        # Step 4: Remove backticks around identifiers
        text = re.sub(r'`([^`]*)`', r'\1', text)
   
        # Step 5: Normalize whitespace
        # Replace multiple spaces with single space
        text = re.sub(r'\s+', ' ', text)
   
        # Step 6: Preserve newlines for main SQL keywords to maintain readability
        keywords = ['SELECT', 'FROM', 'WHERE', 'GROUP BY', 'HAVING', 'ORDER BY',
                   'LIMIT', 'JOIN', 'LEFT JOIN', 'RIGHT JOIN', 'INNER JOIN',
                   'OUTER JOIN', 'UNION', 'VALUES', 'INSERT', 'UPDATE', 'DELETE']
   
        # Case-insensitive replacement for keywords
        pattern = '|'.join(r'\b{}\b'.format(k) for k in keywords)
        text = re.sub(f'({pattern})', r'\n\1', text, flags=re.IGNORECASE)
   
        # Step 7: Final cleanup
        # Remove leading/trailing whitespace and extra newlines
        text = text.strip()
        text = re.sub(r'\n\s*\n', '\n', text)
   
        return text
   

3. Construct a SQL chain

    from langchain_community.tools import QuerySQLDatabaseTool
   
    def get_sql_chain():
        execute_query = QuerySQLDataBaseTool(db=db)
   
        prompt = ChatPromptTemplate.from_messages([
            ("system", """You are a MySQL expert. Given an input question, 
            create a syntactically correct MySQL query to run. 
            Use the LIKE operator for all string matching queries, 
            ensuring proper use of wildcards (% or *) for partial matches. 
            Only use = operator when quering for order_id or product_id column. 
            Here is the relevant table information: {table_info}"""),
            ("human", "{human_message}")
        ])
   
        write_query = prompt | llm | StrOutputParser()
   
        chain = (
            RunnablePassthrough.assign(
                query=write_query | RunnableLambda(clean_sql_query)
            ).assign(
                result=itemgetter("query") | execute_query
            )
        )
   
        return chain
   

4. Create the NL2SQL Tool
   Here’s how to build and integrate the NL2SQL tool:

    from langchain_community.tools import QuerySQLDatabaseTool
    from operator import itemgetter
    import re
    from langchain_core.output_parsers import StrOutputParser
    from langchain_core.prompts import PromptTemplate
    from langchain_core.runnables import RunnablePassthrough, RunnableLambda
   
    class SQLToolSchema(BaseModel):
        question: str
        order_id: str
        product_id: str = None
   
    @tool(args_schema=SQLToolSchema)
    def nl2sql_tool(question: str, order_id: str, product_id: str):
      """Tool to get Product detail or Order status
      `question`: str, `order_id`: str, `product_id`: str"""
   
      print("INSIDE NL2SQL TOOL", question)
      chain = get_sql_chain()
      table_info = db.get_table_info() 
      human_message = f"question: {question}, order id: {order_id}, product_id: {product_id}, sql_query:" 
      print("human_message:", human_message)
      response = chain.invoke({"human_message": human_message, "table_info":table_info})
      return f"Product detail and Order status: {response['result']}, SQL Query used:{response['query']}"
   

5. Test the Tool
   Use a sample query to verify functionality:

    question = "Check Order status"
    order_id = "8"
    product_id = ""
    result = nl2sql_tool.invoke({"question": question,"order_id": order_id, "product_id":product_id})
    print(f"Question: {question}")
    print(f"Answer: {result}")
   

     # output
    INSIDE NL2SQL TOOL Check Order status
    human_message: question: Check Order status, order id: 8, product_id: , sql_query:
    Question: Check Order status
    Answer: Product detail and Order status: [('Processing',)], SQL Query used:SELECT order_status 
    FROM Orders 
    WHERE order_id = 8;
   

To master NL2SQL (Natural Language to SQL) with advanced techniques like Few-Shot example prompts (dynamic and static), dynamic table selection, and enhancing SQL query generation, here's a detailed guide

Reply Tool

Let's build a Reply Tool for your AI-powered automated answering service. This tool helps your Freshdesk agent create professional automatic replies with specific messages for each customer query. It tracks every response with a unique ID, making follow-ups simple and organized. We've designed this tool to ensure your support team can deliver consistent, high-quality answers every time.

1. Create Reply Function: This function hits the POST endpoint of the specific ticket. It uses the ticket's ID and the reply content (provided as input)

    def create_reply(ticket_id, body):
        try:
            url = f'https://{FRESHDESK_URL}.freshdesk.com/api/v2/tickets/{ticket_id}/reply'
            headers = {
                'Content-Type': 'application/json'
            }
            auth = (FRESHDESK_API_KEY, 'X')  # Replace 'X' with your actual API key
            data = {
                "body": body
            }
            response = requests.post(url, headers=headers, auth=auth, json=data)
            return response.json()
        except Exception as e:
            print("Error in create_reply : ", e)
            return ""
   

2. Reply Tool: This code creates a tool that generates professional replies for customer tickets using AI. It takes ticket details and generates a customized reply, which is then sent to the customer via the Freshdesk system.

    from langchain.tools import tool
    from pydantic import BaseModel
   
    class ReplyToTicket(BaseModel):
        information: str
        ticket_id: str
   
    @tool(args_schema=ReplyToTicket)
    def reply_to_ticket_tool(information: str, ticket_id: str):
        """Tool that will send a reply to customer with a ticket id
        `information`: str, 
        `ticket_id`: str"""
   
        print("INSIDE REPLY TOOL", ticket_id)
        prompt = ChatPromptTemplate.from_messages([
            ("system", """
    **Introduction to FutureSmart TechStore:**
    FutureSmart TechStore is a leading retailer specializing in cutting-edge technology products, ensuring you have access to the latest innovations to meet your needs. We pride ourselves on our customer service and offer support in multiple languages to cater to our diverse clientele. Our inventory includes a wide range of electronics, smart home devices, and accessories, all curated to enhance your tech lifestyle.
   
    **Guidelines for Responding to Customer Queries:**
   
    **Identify Language and Respond in English:**
       Analyze the language of the customer's query to ensure accurate understanding. While responding, provide all replies in English to maintain consistency.
   
    **Order Information and Query Specifics:**
       Utilize provided order details to address specific customer queries accurately. Ensure responses are tailored to address only the aspects of the order that the customer has inquired about. Do not include unnecessary order details unless requested.
   
    By adhering to these guidelines, ensure that your responses are informative, accurate, and aligned with FutureSmart TechStore's commitment to customer satisfaction. Remember to always maintain a friendly and professional tone in all communications.
   
    You are an AI assistant tasked with crafting a professional customer reply in Gmail format. Mention FutureSmart TechStore Support, 9876543210 Phone number. Avoid placeholders or incomplete information, and provide clear, actionable steps to resolve the issue"""),
            ("human", "context: {information}" )
        ])
        chain = prompt | llm | StrOutputParser()
        body = chain.invoke({"information": information})
        res = {}
        res = create_reply(ticket_id, body)
        return body, f"reply id: {res.get('id')}"
   

3. Test the tool

    information = """
    Customer Name: John Doe
    Order Number: FS123456789
    Issue: The customer received a damaged smart home device (a smart speaker) and wants a replacement. 
    Additional Information: The order was delivered on 28th December 2024, and the damage was noted upon unboxing. The customer has attached photos of the damaged item for reference and is requesting a quick resolution.
    """
    ticket_id = "12345"
   
    reply_to_ticket_tool.invoke({"information": information, "ticket_id": ticket_id})
   

Combining the Tools

tools = [summarize_tool, retriever_tool, nl2sql_tool, reply_to_ticket_tool]
llm_with_tools = llm.bind_tools(tools)

Building the LangGraph

LangGraph enables you to define a stateful workflow for your AI agent. By structuring nodes and edges, you can define how the agent processes user inputs and transitions between tools.

Steps to Build the LangGraph

1. Define the State: Create a State dictionary to manage the agent’s inputs and outputs.

    from typing import Annotated
    from typing_extensions import TypedDict
    from langgraph.graph.message import add_messages
   
    # Setting up the graph state
    class State(TypedDict):
        messages: Annotated[list, add_messages]
   

2. Define Checkpointer: MemorySaver allow LangGraph agents to persist their state within and across multiple interactions.

    from langgraph.checkpoint.memory import MemorySaver
   
    memory = MemorySaver()
   

3. Add Nodes: Add nodes for the chatbot and tools to handle user queries and invoke the tools.

    from langgraph.graph import StateGraph 
   
    def chatbot(state: State):
         return {"messages": [llm_with_tools.invoke(state["messages"])]}
   
     graph_builder = StateGraph(State)
     graph_builder.add_node("chatbot", chatbot)
   
     tool_node = ToolNode(tools=tools)
     graph_builder.add_node("tools", tool_node)
   

4. Define Edges: Use conditional edges to determine when the agent should switch between nodes.

    from langgraph.prebuilt import tools_condition
   
     graph_builder.add_conditional_edges("chatbot", tools_condition)
     graph_builder.add_edge("tools", "chatbot")
     graph_builder.set_entry_point("chatbot")
   

5. Compile the Graph: Finalize the graph for execution.

    graph = graph_builder.compile(checkpointer=memory)
   

Testing the AI Agent

Once the LangGraph is set up, you can test the agent by simulating user inputs. This ensures the tools and workflows are functioning as expected.

Interactive Testing

Run the following code to test your AI agent interactively:

config = {"configurable": {"thread_id": "1"}}

while True:
    user_input = input("User: ")
    if user_input.lower() in ["quit", "exit", "q"]:
        print("Goodbye!")
        break

    for event in graph.stream({"messages": [("user", user_input)]}, config):
        for value in event.values():
            print("Assistant:", value["messages"][-1].content)

You can provide queries like:

• You can ask questions based on your knowledge base. Do FutureSmart TechStore Support Rupay Credit Card (Trigger Retriever or RAG tool)

• "Brief me about the ticket with ticket id 130?" (Trigger Summary Tool)

• "Tell me the delivery status of order id 10" (Trigger NL2SQL tool)

The AI agent will invoke the appropriate tool to generate responses.

A Conversation between Agent and User:

User:  reply to ticket 100
Assistant: 
INSIDE SUMMARIZE NODE 100
Subject : Order 8
User query : 
Dear Support Team,  I am deeply disappointed that my USB-C Charger (Order #8), expected by Dec 23, did not arrive in time for an important occasion. To make matters worse, I was informed I cannot cancel the order. Please provide an update on when I will receive the item.  I hope for a prompt resolution.  Thank you.

Assistant: ["ticket_id: 100", "Ticket Summary: The customer, deeply disappointed, reports that their USB-C Charger (Order #8) did not arrive by the expected date of December 23 for an important occasion. Additionally, they were informed that they cannot cancel the order. They are requesting an update on when they will receive the item and hope for a prompt resolution. Customer name is not provided."]
Assistant: 
INSIDE NL2SQL TOOL Update on delivery status for Order #8
Assistant: Product detail and Order status: [('Processing',)], SQL Query used:SELECT order_status 
FROM Orders 
WHERE order_id = 8;
Assistant: 
INSIDE REPLY TOOL 100
Assistant: ["Subject: Update on Your USB-C Charger Order\n\nDear [Customer's Name],\n\nThank you for contacting FutureSmart TechStore and sharing your concerns about the delay in your USB-C Charger delivery. We understand how crucial this item is for your upcoming occasion, and we sincerely apologize for any inconvenience caused by the delay.\n\nAs of now, your order status is still marked as 'Processing'. Please rest assured that we are actively working to expedite the process and get your charger to you as soon as possible. We are committed to keeping you updated with any changes to your order status.\n\nIn the meantime, if you have any further questions or if there's anything else we can do to assist you, please do not hesitate to reach out to us. You can contact FutureSmart TechStore Support at 9876543210.\n\nThank you for your patience and understanding. We appreciate your trust in FutureSmart TechStore and are dedicated to resolving this promptly.\n\nBest regards,\n\n[Your Name]  \nFutureSmart TechStore Support  \n9876543210", "reply id: 11000010366579"]
Assistant: I have sent a reply to the customer regarding ticket #100. The customer has been informed about the current status of their order and that it is still marked as "Processing". We have assured them that we are actively working to expedite the process and apologize for any inconvenience caused. The reply includes our contact information for further assistance.
User:  is visa credit card accepted by futuresmart techstore
Assistant: 
INSIDE RETRIEVER NODE Is Visa credit card accepted by FutureSmart TechStore?
Assistant: (context: At FutureSmart TechStore, weaimtomakeyourshoppingexperienceasseamlessandsecureaspossible. Wesupport avarietyof payment methodstosuit yourconvenience.
            SupportedPaymentMethods
            1. CreditandDebitCards○ Visa○ Mastercard○ AmericanExpress○ RuPay2. DigitalWallets○ Paytm○ PhonePe○ GooglePay○ AmazonPay3. NetBanking○ Supportedforall majorIndianbanksincludingHDFC, ICICI, SBI, AxisBank, andmore.4. UPIPayments○ SeamlesspaymentsusingUnifiedPaymentsInterface(UPI).5. CashonDelivery(COD)○ Availableforselect locationsinIndia.○ MaximumCODlimit: INR10,000.6. EMIOptions○ No-cost EMI andstandardEMI optionsareavailableformajorcredit cards.○ EMI optionsdependonthebankandcardtype.7. BankTransfers○ Direct banktransfersaresupportedforbulkorbusinesspurchases.
            PaymentSecurity
            ● All onlinetransactionsareprocessedthroughsecureandencryptedgatewaystoensurethesafetyof yourinformation.● Wecomplywiththelatest PCI DSS(Payment CardIndustryDataSecurityStandard)guidelines.
            ImportantNotes

            ● Software, CDs, orDVDsthat havebeenopened.● Itemsmarkedas"Final Sale"or"Non-Returnable"at thetimeof purchase.● Gift cardsorpromotional items.
            HowtoInitiateaReturn
            1. Contact Us: Reachout toourcustomerserviceteamviaemail atsupport@futuresmarttechstore.comorcall usat +91-XXXXXXXXXXwithyourorderdetails.2. Approval: Onceyourreturnrequest isapproved, youwill rLxGNHXaoMujveJRY4ajftSQYhZbWuZe6g(RMA)numberanddetailedinstructions.3. ShiptheItem: Packtheitemsecurely, includetheRMAnumber, andsendit toourreturnaddress: FutureSmart TechStoreReturnsDepartment Mumbai, IndiaPIN: 400001
            RefundProcess
            ● Refundswill beprocessedoncethereturneditemisinspectedandapproved.● Refundswill beissuedtotheoriginal payment methodwithin7-10businessdaysafterapproval.● Shippingfeesarenon-refundableunlessthereturnisduetoamistakeonourpart (e.g.,wrongordefectiveitem).
            ExchangePolicy

            Foranypayment-relatedqueries, feel freetocontact usat support@futuresmarttechstore.comorcall usat +91-XXXXXXXXXX.

            At FutureSmart TechStore, wearecommittedtodeliveringyourordersswiftlyandsecurely.Belowarethedetailsof ourshippingpoliciestoensureasmoothandtransparent shoppingexperience.
            ShippingLocations
            ● WecurrentlyshipacrossIndia.● International shippingisnot availableat thistime.
            ShippingCharges
            ● StandardShipping: FreeforordersaboveINR1,000.○ OrdersbelowINR1,000will incurashippingfeeof INR50.● ExpressShipping: Availableat INR150foreligiblelocations.
            DeliveryTimeframes
            ● StandardShipping: 3-7businessdays.● ExpressShipping: 1-3businessdays.● Deliverytimesmayvarybasedonlocationandavailability.
            OrderProcessing
            ● Ordersareprocessedwithin24-48hoursof payment confirmation(excludingweekendsandholidays).● Youwill receiveaconfirmationemail withtrackingdetailsonceyourorderisshipped.
            TrackingYourOrder
            ● All ordersareshippedwithatrackingnumber. Youcanmonitortheprogressof yourdeliveryviathelinkprovidedinyourconfirmationemail.
            UndeliverablePackages

            At FutureSmart TechStore, westrivetoensurecustomersatisfactionwitheverypurchase. Ifyouarenot completelysatisfiedwithyourorder, weofferahassle-freereturnpolicytomaketheprocessassmoothaspossible.
            EligibilityforReturns
            ● Productscanbereturnedwithin30daysof receipt.● Itemsmust beintheiroriginal condition, unused, andintheoriginal packagingwithalltags, manuals, andaccessoriesincluded.● Returnsareapplicableforitemsthat aredamaged, defective, ornot asdescribed.
            Non-ReturnableItems
            Thefollowingitemsarenot eligibleforreturn:
            ● Software, CDs, orDVDsthat havebeenopened.● Itemsmarkedas"Final Sale"or"Non-Returnable"at thetimeof purchase.● Gift cardsorpromotional items.
            HowtoInitiateaReturn)
Assistant: Yes, FutureSmart TechStore accepts Visa credit cards as a payment method. They also support Mastercard, American Express, and RuPay.

Let's break down how our AI agent handled this support request – it's a perfect example of multiple tools working together:

Summary Tool in Action: From Ticket #100, it quickly identified the critical details:

• Order #8 for a USB-C Charger

• Expected delivery: Dec 23

• Customer pain points: Delivery delay and cancellation concerns

NL2SQL Tool at Work:

• Translated the customer's query into precise SQL

• Retrieved real-time status: "Processing"

• Zero manual database handling is needed

Reply Tool Magic: Created a professional response that:

• Acknowledged the delay with empathy

• Provided current order status

• Outlined next steps

• Added support contact details

RAG Tool's Smart Retrieval:

• Instantly pulled payment policy details

• Confirmed Visa acceptance

• Added value by mentioning other payment options (Mastercard, Amex)

See how each tool contributes to creating a complete, professional support experience. This is exactly how your AI agent will handle real customer queries – with precision and context awareness.

Visualizing the LangGraph

Visualization helps you understand the workflow of your AI agent and how it transitions between nodes and tools.

from IPython.display import Image, display

try:
    display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
    print("Error generating graph visualization.")

The generated diagram will showcase nodes (chatbot, tools) and the transitions between them, providing a clear overview of your AI agent’s workflow.

Improvements

1. At Futuresmart AI, we have observed in many projects that if you implement methods like Contextual Retrieval, reranking, etc then the performance of the Agentic RAG system will improve.

2. Set up Workflow Automation to handle routine tasks and use AI-driven analytics for actionable insights. Having deployed these solutions for clients, we know which workflows deliver the best ROI.

3. Empower agents to resolve tickets with AI-suggested replies from knowledge sources ensuring accuracy with a human-in-the-loop.

Conclusion

Congratulations! You've just built a powerful Freshdesk AI agent that combines LangGraph, RAG, and NL2SQL capabilities. Your agent can now handle everything from ticket summaries to product queries, making customer support smoother and more efficient.

But this is just the beginning. At FutureSmart AI, we've helped companies transform their customer support from reactive to proactive using these exact techniques. Our solutions have cut response times and boosted customer satisfaction. Want to see similar results for your business? Check out how we've helped companies like yours in our case studies.

We have another AI agent for HubSpot that revolutionizes CRM management by interpreting user intent, automating tasks, and integrating tools for seamless operations. Whether you need support automation or CRM enhancement, we're here to guide you through your AI journey. Reach out at contact@futuresmart.ai to learn more.

Stay tuned for more tutorials and insights as we continue to explore cutting-edge AI advancements, helping you stay ahead in the rapidly evolving tech landscape!

In the last three blogs in our Ultimate Langraph Tutorial Series, we highlighted different components of LangGraph for beginners, Long-term Memory Support, and building an AI agent with custom tools support. After implementing these systems for various enterprise clients, we at Futuresmart AI, have observed that as systems grow, they can become complex and hard to manage.

To solve this, we break down the application into smaller, independent agents - a pattern we've successfully deployed in production environments for multiple clients. This approach offers modularity, specialization, and control. We can work on individual agents separately, create expert agents, and control how they communicate. Having built numerous such systems in FutureSmart AI, we've seen firsthand how this architecture significantly improves maintainability and scalability.

In this blog post, we will learn to create a complete Multi-Agent System from scratch using LangGraph. Here, there will be one Supervisor Agent which can communicate with other specialized agents. Each agent has their own set of tools, mirroring how we at Futuresmart AI structure enterprise-grade AI solutions.

Setting up the Environment

To get started, you need to install the below dependencies

%%capture --no-stderr
%pip install -U langgraph langchain_community langchain_openai langchain_experimental langchain-chroma pypdf sentence-transformers

Setting up API Keys

Before diving into building your LangGraph AI agents, it's crucial to set up your API keys. These keys allow your agent to interact with external tools like Tavily Search and OpenAI GPT models securely. Without them, the tools cannot function effectively.

Here we are using the OpenAI model, but you can use any LLM of your choice.

import getpass
import os

def _set_env(var: str):
    if not os.environ.get(var):
        os.environ[var] = getpass.getpass(f"{var}: ")

_set_env("TAVILY_API_KEY")
_set_env("OPENAI_API_KEY")

Creating the LLM Object

Here’s how to initialize the LLM using LangChain ChatOpenAI

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model_name="gpt-4o")

Create Tools

In this blog, we will make three agents. One agent can do web research with a search engine tool. One agent can retrieve documents and answers based on that. And one agent can query the SQL database.

Now, in our previous LangGraph blog, we have already shown how to create a WebSearch Tool, a Retriever Tool, and an NL2SQL Tool. You can follow those steps to create tools for this tutorial.

• Create Web Search Tool

• Create Retriever Tool (For RAG)

• Create NL2SQL Tool

Now, once you have the tools ready, we are ready to proceed with creating the AI Agents.

Creating the Supervisor Agent

The supervisor agent is responsible for managing the conversation flow between different specialized agents. It decides which agent should handle the current request and when the task is complete. Let's see how to implement this:

from typing import Literal
from typing_extensions import TypedDict
from langgraph.graph import MessagesState, START, END
from langgraph.types import Command

# Define available agents
members = ["web_researcher", "rag", "nl2sql"]
# Add FINISH as an option for task completion
options = members + ["FINISH"]

# Create system prompt for supervisor
system_prompt = (
    "You are a supervisor tasked with managing a conversation between the"
    f" following workers: {members}. Given the following user request,"
    " respond with the worker to act next. Each worker will perform a"
    " task and respond with their results and status. When finished,"
    " respond with FINISH."
)

# Define router type for structured output
class Router(TypedDict):
    """Worker to route to next. If no workers needed, route to FINISH."""
    next: Literal["web_researcher", "rag", "nl2sql", "FINISH"]

# Create supervisor node function
def supervisor_node(state: MessagesState) -> Command[Literal["web_researcher", "rag", "nl2sql", "__end__"]]:
    messages = [
        {"role": "system", "content": system_prompt},
    ] + state["messages"]
    response = llm.with_structured_output(Router).invoke(messages)
    goto = response["next"]
    print(f"Next Worker: {goto}")
    if goto == "FINISH":
        goto = END
    return Command(goto=goto)

The supervisor agent works by:

1. Taking the current conversation state as input

2. Using the system prompt to understand its role

3. Making a decision about which agent should act next

4. Returning a command that directs the flow to the chosen agent

Implementing Individual Agents

Now, let's create our specialized agents using a custom function that creates individual LangGraph graphs for each agent. This approach allows each agent to have its own workflow while maintaining consistency in how they process requests and use tools.

First, let's define our custom create_agent function:

class AgentState(TypedDict):
    """The state of the agent."""
    messages: Annotated[Sequence[BaseMessage], add_messages]

def create_agent(llm, tools):
    llm_with_tools = llm.bind_tools(tools)
    def chatbot(state: AgentState):
        return {"messages": [llm_with_tools.invoke(state["messages"])]}

    graph_builder = StateGraph(AgentState)
    graph_builder.add_node("agent", chatbot)

    tool_node = ToolNode(tools=tools)
    graph_builder.add_node("tools", tool_node)

    graph_builder.add_conditional_edges(
        "agent",
        tools_condition,
    )
    graph_builder.add_edge("tools", "agent")
    graph_builder.set_entry_point("agent")
    return graph_builder.compile()

This custom function:

1. Takes an LLM and a list of tools as input

2. Creates a new StateGraph for the agent

3. Sets up the necessary nodes for the agent and tool execution

4. Configures the flow between nodes

5. Returns a compiled graph that represents the agent's complete workflow

Now, let's create our specialized agents using this function:

Web Research Agent

websearch_agent = create_agent(llm, [web_search_tool])

def web_research_node(state: MessagesState) -> Command[Literal["supervisor"]]:
    result = websearch_agent.invoke(state)
    return Command(
        update={
            "messages": [
                HumanMessage(content=result["messages"][-1].content, name="web_researcher")
            ]
        },
        goto="supervisor",
    )

RAG Agent

rag_agent = create_agent(llm, [retriever_tool])

def rag_node(state: MessagesState) -> Command[Literal["supervisor"]]:
    result = rag_agent.invoke(state)
    return Command(
        update={
            "messages": [
                HumanMessage(content=result["messages"][-1].content, name="rag")
            ]
        },
        goto="supervisor",
    )

NL2SQL Agent

nl2sql_agent = create_agent(llm, [nl2sql_tool])

def nl2sql_node(state: MessagesState) -> Command[Literal["supervisor"]]:
    result = nl2sql_agent.invoke(state)
    return Command(
        update={
            "messages": [
                HumanMessage(content=result["messages"][-1].content, name="nl2sql")
            ]
        },
        goto="supervisor",
    )

Putting It All Together

Finally, let's create the main graph that connects all our agents:

builder = StateGraph(MessagesState)
builder.add_edge(START, "supervisor")
builder.add_node("supervisor", supervisor_node)
builder.add_node("web_researcher", web_research_node)
builder.add_node("rag", rag_node)
builder.add_node("nl2sql", nl2sql_node)
graph = builder.compile()

This creates a complete multi-agent system where:

1. The supervisor receives the initial request

2. It routes the request to the appropriate specialized agent

3. The specialized agent processes the request using its tools

4. Control returns to the supervisor to decide the next step

5. The process continues until the supervisor determines the task is complete

Visualizing the LangGraph

from IPython.display import Image, display

try:
    display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
    # You can put your exception handling code here
    pass

Testing the System

Let's test our multi-agent system with a couple of examples:

# Example: Complex Query Using Multiple Agents
input_question = "Find the founder of FutureSmart AI and then do a web research on him"
for s in graph.stream(
    {"messages": [("user", input_question)]}, 
    subgraphs=True
):
    print(s)
    print("----")

Output:

Next Worker: rag
((), {'supervisor': None})
----
INSIDE RETRIEVER NODE
(('rag:7c5458df-0abd-944a-27f7-b0bad49ccf3d',), {'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_fK9lMHGrtubenQ697xpd2ZZ2', 'function': {'arguments': '{"question":"Who is the founder of FutureSmart AI?"}', 'name': 'retriever_tool'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 25, 'prompt_tokens': 70, 'total_tokens': 95, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_d28bcae782', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-e538251e-24e9-45ac-a5b7-b4ce111615ad-0', tool_calls=[{'name': 'retriever_tool', 'args': {'question': 'Who is the founder of FutureSmart AI?'}, 'id': 'call_fK9lMHGrtubenQ697xpd2ZZ2', 'type': 'tool_call'}], usage_metadata={'input_tokens': 70, 'output_tokens': 25, 'total_tokens': 95, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})]}})
----
(('rag:7c5458df-0abd-944a-27f7-b0bad49ccf3d',), {'tools': {'messages': [ToolMessage(content='FutureSmart AI provides customized speech to text services, employing cutting-\nedge speech recognition technologies to cater to specific client needs. Ideal for \ncreating efficient documentation and enabling voice-driven commands, this \nsolution boosts productivity and accessibility.\n\nFutureSmart AI provides custom Natural Language Processing (NLP) \nsolutions for companies looking to get ahead of the future. Our \ndedicated team of Data Scientists and ML Engineers provides an end-\nto-end solution from data labeling to modeling and deploying an ML \nmodel tailored to your specific use case. \nFounder: Pradip Nichite \n \nServices: \nText Classification \nAt FutureSmart AI, we develop custom text classification solutions using \nadvanced NLP techniques tailored to your specific business requirements. \nLeveraging Python, Pytorch, and Hugging Face transformers, we enable precise \ndata categorization across applications such as intent detection, document \ncategorization, and sentiment analysis, enhancing your decision-making \nprocesses and operational efficiency. \n \nChatbots \nWe specialize in creating custom chatbots that integrate seamlessly with your \nbusiness environment. Using semantic search and large language models, our', name='retriever_tool', id='fe12dcaa-a380-437f-8c24-5a7cbf6ab031', tool_call_id='call_fK9lMHGrtubenQ697xpd2ZZ2')]}})
----
(('rag:7c5458df-0abd-944a-27f7-b0bad49ccf3d',), {'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_nvmRMsfWcg0YVC9xeTqxZO7z', 'function': {'arguments': '{"question": "Who is Pradip Nichite?"}', 'name': 'retriever_tool'}, 'type': 'function'}, {'id': 'call_IGzCvWkpkzlpwlhFR1MR80U4', 'function': {'arguments': '{"question": "What is the professional background of Pradip Nichite?"}', 'name': 'retriever_tool'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 67, 'prompt_tokens': 322, 'total_tokens': 389, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_d28bcae782', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-a4c6ec42-3aec-449c-ab39-ac029109eaad-0', tool_calls=[{'name': 'retriever_tool', 'args': {'question': 'Who is Pradip Nichite?'}, 'id': 'call_nvmRMsfWcg0YVC9xeTqxZO7z', 'type': 'tool_call'}, {'name': 'retriever_tool', 'args': {'question': 'What is the professional background of Pradip Nichite?'}, 'id': 'call_IGzCvWkpkzlpwlhFR1MR80U4', 'type': 'tool_call'}], usage_metadata={'input_tokens': 322, 'output_tokens': 67, 'total_tokens': 389, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})]}})
----
INSIDE RETRIEVER NODE
INSIDE RETRIEVER NODE
(('rag:7c5458df-0abd-944a-27f7-b0bad49ccf3d',), {'tools': {'messages': [ToolMessage(content='FutureSmart AI provides customized speech to text services, employing cutting-\nedge speech recognition technologies to cater to specific client needs. Ideal for \ncreating efficient documentation and enabling voice-driven commands, this \nsolution boosts productivity and accessibility.\n\nFutureSmart AI provides custom Natural Language Processing (NLP) \nsolutions for companies looking to get ahead of the future. Our \ndedicated team of Data Scientists and ML Engineers provides an end-\nto-end solution from data labeling to modeling and deploying an ML \nmodel tailored to your specific use case. \nFounder: Pradip Nichite \n \nServices: \nText Classification \nAt FutureSmart AI, we develop custom text classification solutions using \nadvanced NLP techniques tailored to your specific business requirements. \nLeveraging Python, Pytorch, and Hugging Face transformers, we enable precise \ndata categorization across applications such as intent detection, document \ncategorization, and sentiment analysis, enhancing your decision-making \nprocesses and operational efficiency. \n \nChatbots \nWe specialize in creating custom chatbots that integrate seamlessly with your \nbusiness environment. Using semantic search and large language models, our', name='retriever_tool', id='57d1c6f3-b789-4ae1-84c4-c156ca34d3c1', tool_call_id='call_nvmRMsfWcg0YVC9xeTqxZO7z'), ToolMessage(content='FutureSmart AI provides customized speech to text services, employing cutting-\nedge speech recognition technologies to cater to specific client needs. Ideal for \ncreating efficient documentation and enabling voice-driven commands, this \nsolution boosts productivity and accessibility.\n\nFutureSmart AI provides custom Natural Language Processing (NLP) \nsolutions for companies looking to get ahead of the future. Our \ndedicated team of Data Scientists and ML Engineers provides an end-\nto-end solution from data labeling to modeling and deploying an ML \nmodel tailored to your specific use case. \nFounder: Pradip Nichite \n \nServices: \nText Classification \nAt FutureSmart AI, we develop custom text classification solutions using \nadvanced NLP techniques tailored to your specific business requirements. \nLeveraging Python, Pytorch, and Hugging Face transformers, we enable precise \ndata categorization across applications such as intent detection, document \ncategorization, and sentiment analysis, enhancing your decision-making \nprocesses and operational efficiency. \n \nChatbots \nWe specialize in creating custom chatbots that integrate seamlessly with your \nbusiness environment. Using semantic search and large language models, our', name='retriever_tool', id='a498df53-f77a-4bfb-abbf-9153790295e5', tool_call_id='call_IGzCvWkpkzlpwlhFR1MR80U4')]}})
----
(('rag:7c5458df-0abd-944a-27f7-b0bad49ccf3d',), {'agent': {'messages': [AIMessage(content="The founder of FutureSmart AI is Pradip Nichite. Unfortunately, the current retrieval did not provide additional information specifically about Pradip Nichite's professional background or further personal details. For more comprehensive insights, you might consider conducting a more extensive web search or accessing professional networking sites like LinkedIn.", additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 63, 'prompt_tokens': 888, 'total_tokens': 951, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_d28bcae782', 'finish_reason': 'stop', 'logprobs': None}, id='run-bc47c29c-0693-41c5-88be-b322e1fbb096-0', usage_metadata={'input_tokens': 888, 'output_tokens': 63, 'total_tokens': 951, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})]}})
----
((), {'rag': {'messages': [HumanMessage(content="The founder of FutureSmart AI is Pradip Nichite. Unfortunately, the current retrieval did not provide additional information specifically about Pradip Nichite's professional background or further personal details. For more comprehensive insights, you might consider conducting a more extensive web search or accessing professional networking sites like LinkedIn.", additional_kwargs={}, response_metadata={}, name='rag')]}})
----
Next Worker: web_researcher
((), {'supervisor': None})
----
(('web_researcher:509bb5e2-bf9e-2c1d-5c65-978a73d5e94c',), {'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_L5d4KhCSPsT5HmHTpHmnyryx', 'function': {'arguments': '{"query":"Pradip Nichite"}', 'name': 'tavily_search_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 23, 'prompt_tokens': 161, 'total_tokens': 184, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_e161c81bbd', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-35518542-a3b6-424b-b4c7-f8fbb56cffd6-0', tool_calls=[{'name': 'tavily_search_results_json', 'args': {'query': 'Pradip Nichite'}, 'id': 'call_L5d4KhCSPsT5HmHTpHmnyryx', 'type': 'tool_call'}], usage_metadata={'input_tokens': 161, 'output_tokens': 23, 'total_tokens': 184, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})]}})
----
(('web_researcher:509bb5e2-bf9e-2c1d-5c65-978a73d5e94c',), {'tools': {'messages': [ToolMessage(content='[{"url": "https://www.youtube.com/c/PradipNichiteAI", "content": "Hello, my name is Pradip Nichite. I am a 🚀 Top Rated Plus Data Science Freelancer with 8+ years of experience, specializing in NLP and Back-End Development. Founder of FutureSmart AI, helping"}, {"url": "https://www.youtube.com/channel/UCwpCmuWq_NPVLNyr8z1IGGQ", "content": "I\'m Pradip Nichite, a Top Rated Plus freelance Data Scientist on Upwork 💼, a successful digital nomad 🌍, and an entrepreneur. My journey in freelancing has led me to earn over $200K 💰"}]', name='tavily_search_results_json', id='5daeafe8-e673-425e-9d7e-35f49ccae710', tool_call_id='call_L5d4KhCSPsT5HmHTpHmnyryx', artifact={'query': 'Pradip Nichite', 'follow_up_questions': None, 'answer': None, 'images': [], 'results': [{'title': 'Pradip Nichite - YouTube', 'url': 'https://www.youtube.com/c/PradipNichiteAI', 'content': 'Hello, my name is Pradip Nichite. I am a 🚀 Top Rated Plus Data Science Freelancer with 8+ years of experience, specializing in NLP and Back-End Development. Founder of FutureSmart AI, helping', 'score': 0.8080827, 'raw_content': None}, {'title': 'Pradip Nichite - YouTube', 'url': 'https://www.youtube.com/channel/UCwpCmuWq_NPVLNyr8z1IGGQ', 'content': "I'm Pradip Nichite, a Top Rated Plus freelance Data Scientist on Upwork 💼, a successful digital nomad 🌍, and an entrepreneur. My journey in freelancing has led me to earn over $200K 💰", 'score': 0.7636429, 'raw_content': None}], 'response_time': 1.73})]}})
----
(('web_researcher:509bb5e2-bf9e-2c1d-5c65-978a73d5e94c',), {'agent': {'messages': [AIMessage(content='Pradip Nichite is a Top Rated Plus Data Science Freelancer with over 8 years of experience, specializing in Natural Language Processing (NLP) and Back-End Development. He is the founder of FutureSmart AI. Additionally, Pradip is recognized as a successful digital nomad and entrepreneur, having earned over $200K through freelancing, primarily on platforms like Upwork. For more insights, you can explore his [YouTube channel](https://www.youtube.com/c/PradipNichiteAI), where he shares more about his experiences and expertise.', additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 116, 'prompt_tokens': 346, 'total_tokens': 462, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_e161c81bbd', 'finish_reason': 'stop', 'logprobs': None}, id='run-32d923b8-a3be-420a-a06f-35a7e27c68bb-0', usage_metadata={'input_tokens': 346, 'output_tokens': 116, 'total_tokens': 462, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})]}})
----
((), {'web_researcher': {'messages': [HumanMessage(content='Pradip Nichite is a Top Rated Plus Data Science Freelancer with over 8 years of experience, specializing in Natural Language Processing (NLP) and Back-End Development. He is the founder of FutureSmart AI. Additionally, Pradip is recognized as a successful digital nomad and entrepreneur, having earned over $200K through freelancing, primarily on platforms like Upwork. For more insights, you can explore his [YouTube channel](https://www.youtube.com/c/PradipNichiteAI), where he shares more about his experiences and expertise.', additional_kwargs={}, response_metadata={}, name='web_researcher')]}})
----
Next Worker: FINISH
((), {'supervisor': None})
----

Do not confuse by seeing {'supervisor': None} . This issue has been faced by others also. The multi-agent system will still work as intended.

Things to Improve

• Add memory support to allow agents to remember previous interactions and maintain context

• Improved the system prompt of the supervisor agent for better decision-making

• Add more relevant tools to each individual agent

Conclusion

In this tutorial, we've built a sophisticated multi-agent system using LangGraph that demonstrates how to create specialized agents working together under centralized supervision. The system showcases advanced concepts in agent orchestration, tool integration, and state management - areas where we at Futuresmart AI, have successfully delivered numerous enterprise solutions.

This architecture provides several advantages:

1. Modularity: Each agent has a specific role and can be modified independently

2. Scalability: New agents can be added without changing existing ones

3. Flexibility: The supervisor can dynamically choose the best agent for each task

4. Control: The workflow is clearly defined and manageable

If you found this guide useful and want to explore more, then you should definitely check our YouTube video on the new multi-agent framework from OpenAI (Swarm)

📚 More Learning Resources

If you prefer visual learning, check out these step-by-step tutorials:

• Build RAG Applications from Scratch Without LangChain or LlamaIndex – Learn to build RAG applications without relying on frameworks for better customization and debugging.

• LangChain RAG Course: From Basics to a Production-Ready RAG Chatbot – A comprehensive guide on implementing RAG using LangChain, covering everything from basics to production deployment.

• Mastering Natural Language to SQL with LangChain and LangSmith | NL2SQL – A hands-on tutorial on converting natural language queries into SQL using LangChain and LangSmith.

At FutureSmart AI, we help businesses develop state-of-the-art AI solutions tailored to their needs. We've implemented similar multi-agent architectures for various industries, from customer service automation to complex AI Interview systems.

If you have inquiries, feel free to reach out to us at contact@futuresmart.ai. For real-world examples of our work, take a look at our case studies, where we showcase how our expertise in LangGraph and other AI technologies has delivered measurable business value.

Stay tuned for the next tutorial in this series, where we'll explore more advanced patterns and optimizations in multi-agent systems.

Resources

Managing the core objects of HubSpot CRM - Contacts, Companies, Tickets, and Deals - is crucial for business operations. However, manually tracking and updating these objects can be a difficult task, prone to errors and inefficiencies. The static workflows and manual processes can limit your team's productivity.

This is where the HubSpot AI Agent excels. But, building a custom HubSpot agent that can intelligently interact with your CRM data and perform tasks autonomously is a challenging task, requiring extensive coding and technical expertise.

Based on our work developing custom HubSpot AI Agents for clients, we’ll guide you through the process of creating your own HubSpot agent using powerful tools integrated within the LangGraph framework. This hubspot developers tutorial offers a practical, step-by-step approach to building a fully functional AI agent capable of performing complex tasks.

Our work has shown that the HubSpot Agent, when integrated with LLMs, can intelligently interact with CRM data. It processes natural language queries, identifies relevant CRM objects, and autonomously performs operations like creating deals, updating contacts, resolving tickets, tracking deals, or generating insights. This ability to interpret user intent, call APIs, and combine multiple tools makes it much more than a CRM assistant—it’s a game-changer.

How the Hubspot Agent Works

Let’s understand the high-level overview of the workflow. This LangGraph Agent utilizes a structured workflow and Tool-calling capabilities.

1. Query Understanding: Our Main Agent uses LLM (Large Language Model) to analyze user queries and identify the intent.

2. Tool Calling: The agent intelligently determines the appropriate workflow and APIs or set of tools required to execute the task.

3. Dynamic Execution: It seamlessly performs operations like creating a contact, updating a deal, or fetching CRM data.

   

If you are not aware of LangGraph Agents, Nodes, Tools, and Edges, then we highly recommend reading our beginner-friendly LangGraph Tutorial

Setting Up Your LangGraph Agent for HubSpot

Before diving into the main coding part, we need to follow some key steps to set up our HubSpot Dashboard. If you have already configured it, then feel free to skip this section.

• Creating a Private App in HubSpot: Create a Private app to generate a Private Key.

  

• 

  Selecting Scopes: Define permissions to control access to CRM objects like Contacts or Deals.

• 

• Generating Private Key: Generate an API key to securely connect your agent with HubSpot.

• 

  Installing the Python HubSpot API SDK: Install and configure the hubspot-api-client Python library for interactions with HubSpot’s CRM objects.

Setting up the Environment

To get started, you need to install the below dependencies

!pip install -U hubspot-api-client langchain langgraph langchain-openai pydantic python-dotenv

Setting up API Keys

Before diving into building your Hubspot AI agent, it's crucial to set up your API keys. These keys allow your agent to interact securely with external environments and tools like Hubspot and OpenAI GPT models. Without them, the tools cannot function effectively.

Here we are using the OpenAI model, but you can use any LLM of your choice.

import getpass
import os

def _set_env(var: str):
    if not os.environ.get(var):
        os.environ[var] = getpass.getpass(f"{var}: ")

_set_env("HUBSPOT_PRIVATE_KEY")
_set_env("OPENAI_API_KEY")

Creating the LLM Object

Here’s how to initialize the LLM using LangChain ChatOpenAI

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model_name="gpt-4o")

Creating Tools for AI Agent

We will create a few custom tools that our langgraph agent can use to interact with Hubspot CRM.

Dynamic Search Tool Integration

The Dynamic Search tool enables to fetch relevant records from CRM.

class SearchRequestSchema(BaseModel):
    object_type: str
    filterGroups: List[FilterGroup]
    properties: Optional[List[str]] = None
    limit: Optional[int] = 10
    sorts: Optional[List[str]] = None
    after: Optional[str] = None
    propertiesWithHistory: Optional[List[str]] = None
    archived: Optional[bool] = False

@tool(args_schema=SearchRequestSchema)
def dynamic_search(
    object_type: str,
    filterGroups: List[FilterGroup],
    properties: Optional[List[str]] = None,
    limit: Optional[int] = 10
):
    """Generic tool for searching objects on HubSpot Ex: contacts,deals,tickets,companies."""
    object_to_search_method = {
        'contacts': 'crm.contacts.search_api.do_search',
        'deals': 'crm.deals.search_api.do_search',
        'tickets': 'crm.tickets.search_api.do_search',
        'companies': 'crm.companies.search_api.do_search',
        'line_items': 'crm.line_items.search_api.do_search',
        'quotes': 'crm.quotes.search_api.do_search'
    }

    if object_type not in object_to_search_method:
        return {"Error": f"Unsupported object type: {object_type}"}

    try:
        api_client = get_client()
        search_method_path = object_to_search_method[object_type]
        search_method = getattr(
            api_client.crm, search_method_path.split('.')[1]
        ).search_api.do_search

        # Prepare filter groups for the request
        formatted_filter_groups = [
            {"filters": [filter.dict() for filter in group.filters]} for group in filterGroups
        ]

        # Construct the search request
        public_object_search_request = PublicObjectSearchRequest(
            filter_groups=formatted_filter_groups,
            properties=properties or [],
            limit=limit
        )

        # Send the request
        api_response = search_method(public_object_search_request=public_object_search_request)
        return {"status": "success", "response_details": api_response}
    except ApiException as e:
        return {"Exception when retrieving object information": str(e)}
    except ValueError as e:
        return {"Error": str(e)}

Output:

Create Object Tool Integration

The Create Object tool will create different crm objects and helps in associations between them as per user's questions.

class CreateObjectSchema(BaseModel):
    object_type: str
    properties: Dict[str, str]

@tool(args_schema=CreateObjectSchema)
def create_object(object_type: str, properties: Dict[str, str]):
    """ Tool for creating a CRM object (e.g., contact, deal, ticket) on HubSpot. """
    try:
        from hubspot.crm.objects import SimplePublicObjectInputForCreate

        # Mapping of object types to their respective API creation methods
        object_to_create_method = {
            'contacts': 'crm.contacts.basic_api.create',
            'deals': 'crm.deals.basic_api.create',
            'tickets': 'crm.tickets.basic_api.create',
            'companies': 'crm.companies.basic_api.create',
            'line_items': 'crm.line_items.basic_api.create',
            'quotes': 'crm.quotes.basic_api.create',
        }

        # Check if the object type is supported
        if object_type not in object_to_create_method:
            raise ValueError(f"Unsupported object type: {object_type}")

        # Initialize the API client
        api_client = get_client()

        # Get the appropriate API creation method
        create_method_path = object_to_create_method[object_type]
        create_method = getattr(api_client.crm, create_method_path.split('.')[1]).basic_api.create

        # Prepare the object input for creation
        simple_public_object_input_for_create = SimplePublicObjectInputForCreate(properties=properties)

        # Call the create method
        api_response = create_method(simple_public_object_input_for_create=simple_public_object_input_for_create)

        # Return the API response
        return {
            "status": "success",
            "response_details": api_response,
            "object_id": api_response.id
        }

    except Exception as e:
        # Handle errors and return the error message
        return {"error": str(e)}

Output:

Update Object Tool Integration

The Update Object tool will be used to update different crm objects.

class UpdateObjectSchema(BaseModel):
    object_type: str
    object_id: str
    properties: Dict[str, str]

@tool(args_schema=UpdateObjectSchema)
def update_object(object_type: str, object_id: str, properties: Dict[str, str]):
    """ Tool for updating a CRM object (e.g., contact, deal, ticket) on HubSpot using its ID. """
    try:
        from hubspot.crm.objects import SimplePublicObjectInput

        # Mapping of object types to their respective API update methods
        object_to_update_method = {
            'contacts': 'crm.contacts.basic_api.update',
            'deals': 'crm.deals.basic_api.update',
            'tickets': 'crm.tickets.basic_api.update',
            'companies': 'crm.companies.basic_api.update',
            'line_items': 'crm.line_items.basic_api.update',
            'quotes': 'crm.quotes.basic_api.update',
        }

        # Check if the object type is supported
        if object_type not in object_to_update_method:
            raise ValueError(f"Unsupported object type: {object_type}")

        # Initialize the API client
        api_client = get_client()

        # Get the appropriate API update method
        update_method_path = object_to_update_method[object_type]
        update_method = getattr(api_client.crm, update_method_path.split('.')[1]).basic_api.update

        # Prepare the object input for updating
        simple_public_object_input = SimplePublicObjectInput(properties=properties)

        # Call the update method
        api_response = update_method(object_id, simple_public_object_input)

        # Return the API response
        return {
            "status": "success",
            "response_details": api_response,
            "object_id": api_response.id
        }

    except Exception as e:
        # Handle errors and return the error message
        return {"error": str(e)}

Output:

Combining the Tools

To create a fully functional Hubspot AI agent, you need to integrate the Dynamic Search, Create Object, and Update Object tools into the LLM.

Here’s how to combine the tools:

tools = [dynamic_search, create_object, update_object]
llm_with_tools = llm.bind_tools(tools)

With this configuration, your LLM can invoke the appropriate tool based on the user’s query.

Building the LangGraph

LangGraph enables you to define a stateful workflow for your AI agent. By structuring nodes and edges, you can define how the agent processes user inputs and transitions between tools.

Steps to Build the LangGraph

1. Define the State: Create a State dictionary to manage the agent’s inputs and outputs.

    from typing import Annotated
    from langgraph.graph import StateGraph
    from langgraph.graph.message import add_messages
    from langgraph.checkpoint.memory import MemorySaver
   
    memory = MemorySaver()
   
    class State(TypedDict):
        messages: Annotated[list, add_messages]
   

2. Add Nodes: Add nodes for the chatbot and tools to handle user queries and invoke the tools.

    def chatbot(state: State):
         return {"messages": [llm_with_tools.invoke(state["messages"])]}
   
     graph_builder = StateGraph(State)
     graph_builder.add_node("chatbot", chatbot)
   
     tool_node = ToolNode(tools=[dynamic_search, create_object, update_object])
     graph_builder.add_node("tools", tool_node)
   

3. Define Edges: Use conditional edges to determine when the agent should switch between nodes.

     from langgraph.prebuilt import tools_condition
   
     graph_builder.add_conditional_edges("chatbot", tools_condition)
     graph_builder.add_edge("tools", "chatbot")
     graph_builder.set_entry_point("chatbot")
   

4. Compile the Graph: Finalize the graph for execution.

    graph = graph_builder.compile(checkpointer=memory)
   

Testing the AI Agent

Once the LangGraph is set up, you can test the agent by simulating user inputs. This ensures the tools and workflows are functioning as expected.

Interactive Testing

Run the following code to test your AI agent interactively:

config = {"configurable": {"thread_id": "1"}}

while True:
    user_input = input("User: ")
    if user_input.lower() in ["quit", "exit", "q"]:
        print("Goodbye!")
        break

    for event in graph.stream({"messages": [("user", user_input)]}, config):
        for value in event.values():
            print("Assistant:", value["messages"][-1].content)

You can provide queries like:

• You can ask questions based on your Hubspot records.

• "What is the current status of ‘ABC’ Deal?"

• "Tell me the name of any two contacts with association with company ‘XYZ’. " (Trigger Search tool)

The AI agent will invoke the appropriate tool to generate responses.

Visualizing the Graph

We can easily plot a flowchart of our LangGraph Workflow.

from IPython.display import Image, display

try:
    display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
    print("Error generating graph visualization.")

Steps in the Workflow:

1. Receive Query: A user inputs a query, such as "Update the status of Deal X to Closed Won."

2. Interpretation: The agent uses LLMs to determine the query's intent and required action.

3. Tool Selection: Based on the query, the agent selects the appropriate custom HubSpot AI tools.

4. Execution: The tool is being executed and you can expect real-time updates in the Hubspot CRM.

5. Feedback: Results are shared with the user in an easily understandable format

Testing and Refinement

Leveraging the insights we've gained from hubspot api integration services for our clients, we understand that thorough testing and refinement are essential:

• Simulate Real-World Scenarios: Use varied queries to validate the agent’s ability to handle different tasks.

• Validate API Operations: Ensure operations are performed correctly and efficiently.

• Iterate and Improve: Gather user feedback and refine the agent’s workflows and capabilities over time.

While building and improving this, you may face challenges like we faced. For example, how to handle complex queries or ambiguous questions. Robust error handling can help while building this type of AI application.

Conclusion

Drawing on the experiences we’ve gained delivering tailored AI solutions to businesses, we can confidently say that the Hubspot Agent is more than just a tool—it’s an intelligent assistant that simplifies CRM management, boosts productivity, and saves valuable time. Automating repetitive tasks and enabling intelligent decision-making allows businesses to focus on what matters most—building strong customer relationships and driving growth. Looking ahead, the principles and technologies used in building the HubSpot AI Agent can also be extended to other areas such as Marketing and CMS, enabling even broader automation and intelligent data interaction across your business operations.

If you’re ready to take your HubSpot experience to the next level, building a Hubspot Agent is the way forward. It’s not just a CRM upgrade—it’s a smarter, faster way to achieve your business goals.

If you found this guide useful and want to explore more advanced techniques, don’t forget to check out our other tutorials. At FutureSmart AI, we help businesses develop state-of-the-art AI solutions tailored to their needs. To see a demo or discuss your requirements, contact us at contact@futuresmart.ai.

For real-world examples of how we’ve helped businesses, take a look at our case studies, where we showcase the practical value of our expertise.

In this blog post, we will walk you through the process of creating a custom AI agent with three powerful tools: Web Search, Retrieval-Augmented Generation (RAG), and Natural Language to SQL (NL2SQL), all integrated within the LangGraph framework. This guide is designed to provide you with a practical, step-by-step approach to building a fully functional AI agent capable of performing complex tasks such as retrieving real-time data from the web, generating responses based on retrieved information from the knowledge base, and translating natural language queries into SQL database queries. By the end of this tutorial, you will have a working AI agent equipped to handle these diverse functionalities seamlessly.

In the previous LangGraph Tutorial blog, we explored the foundational concepts behind LangGraph, focusing on creating simple AI agents by defining nodes and edges. This time, we’re taking things a step further. We will incorporate additional tools that extend the agent’s capabilities—allowing it to interact with the web, retrieve specific information from knowledge-base, and query SQL databases through natural language commands. This comprehensive guide is aimed at developers looking to build customizable, task-oriented AI agents that can be deployed in various real-world applications.

We have already built a similar AI Agent using OpenAI Swarm. This time, we are going to do this using LangGraph.

Let’s dive into the setup process and start by building the environment for our AI agent!

Setting up the Environment

To get started, you need to install the below dependencies

!pip install -U langchain langchain-chroma pypdf sentence-transformers langgraph langchain_openai langchain_community

Setting up API Keys

Before diving into building your LangGraph AI agent, it's crucial to set up your API keys. These keys allow your agent to interact with external tools like Tavily Search and OpenAI GPT models securely. Without them, the tools cannot function effectively.

Here we are using OpenAI model, but you can use any LLM of your choice.

import getpass
import os

def _set_env(var: str):
    if not os.environ.get(var):
        os.environ[var] = getpass.getpass(f"{var}: ")

_set_env("TAVILY_API_KEY")
_set_env("OPENAI_API_KEY")

Creating the LLM Object

Here’s how to initialize the LLM using LangChain ChatOpenAI

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model_name="gpt-4o")

WebSearch Tool Integration

Integrating a WebSearch tool allows your AI agent to fetch real-time information from the web. At FutureSmart AI, we generally use TavilySearch tool for quick and accurate web results. But, you can use any web search tool of your choice.

from langchain_community.tools.tavily_search import TavilySearchResults

web_search_tool = TavilySearchResults(max_results=2)

Agentic RAG Tool Integration

The Retrieval-Augmented Generation (RAG) tool enhances your AI agent by enabling it to fetch relevant documents to the user's questions. As a result, users will get more accurate and context-rich responses.

Steps to Integrate the RAG Tool

1. Load Documents: Use the PyPDFLoader and Docx2txtLoader to load documents from a folder.

    from langchain_community.document_loaders import PyPDFLoader, Docx2txtLoader
   
    def load_documents(folder_path: str) -> List[Document]:
        documents = []
        for filename in os.listdir(folder_path):
            file_path = os.path.join(folder_path, filename)
            if filename.endswith('.pdf'):
                loader = PyPDFLoader(file_path)
            elif filename.endswith('.docx'):
                loader = Docx2txtLoader(file_path)
            else:
                print(f"Unsupported file type: {filename}")
                continue
            documents.extend(loader.load())
        return documents
   
    folder_path = "/content/docs"
    documents = load_documents(folder_path)
    print(f"Loaded {len(documents)} documents from the folder.")
   

    # Output
    Loaded 2 documents from the folder.
   

2. Split Text into Chunks: Prepare documents for vectorization by splitting them into manageable chunks.

    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len
    )
   
    splits = text_splitter.split_documents(documents)
    print(f"Split the documents into {len(splits)} chunks.")
   

    # Output
    Split the documents into 4 chunks.
   

3. Generate Embeddings: Use SentenceTransformers to create embeddings for efficient similarity searches.

    from langchain_community.embeddings.sentence_transformer import SentenceTransformerEmbeddings
   
    embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
   

4. Create and Persist a Vector Store:

    from langchain_chroma import Chroma
   
    collection_name = "my_collection"
    vectorstore = Chroma.from_documents(
        collection_name=collection_name,
        documents=splits,
        embedding=embedding_function,
        persist_directory="./chroma_db"
    )
   

5. Build the Retriever Tool:

    from langchain.tools import tool
    from pydantic import BaseModel
   
    class RagToolSchema(BaseModel):
        question: str
   
    @tool(args_schema=RagToolSchema)
    def retriever_tool(question):
      """Tool to Retrieve Semantically Similar documents to answer User Questions related to FutureSmart AI"""
      print("INSIDE RETRIEVER NODE")
      retriever = vectorstore.as_retriever(search_kwargs={"k": 2})
      retriever_result = retriever.invoke(question)
      return "\n\n".join(doc.page_content for doc in retriever_results)
   

This tool allows your AI agent to retrieve relevant chunks of information from your document database, making it highly effective for knowledge-based tasks.

You can test the retriever right now to see how it is performing

retriever = vectorstore.as_retriever(search_kwargs={"k": 2})
# pass question
retriever_results = retriever.invoke("Who is the founder of Futuresmart AI?")
print(retriever_results)

# Output
[Document(metadata={'page': 1, 'source': '/content/docs/FutureSmart AI .pdf'}, page_content='FutureSmart AI provides customized speech to text services, employing cutting-\nedge speech recognition technologies to cater to specific client needs. Ideal for \ncreating efficient documentation and enabling voice-driven commands, this \nsolution boosts productivity and accessibility.'), 
Document(metadata={'page': 0, 'source': '/content/docs/FutureSmart AI .pdf'}, page_content='FutureSmart AI provides custom Natural Language Processing (NLP) \nsolutions for companies looking to get ahead of the future. Our \ndedicated team of Data Scientists and ML Engineers provides an end-\nto-end solution from data labeling to modeling and deploying an ML \nmodel tailored to your specific use case. \nFounder: Pradip Nichite \n \nServices: \nText Classification \nAt FutureSmart AI, we develop custom text classification solutions using \nadvanced NLP techniques tailored to your specific business requirements. \nLeveraging Python, Pytorch, and Hugging Face transformers, we enable precise \ndata categorization across applications such as intent detection, document \ncategorization, and sentiment analysis, enhancing your decision-making \nprocesses and operational efficiency. \n \nChatbots \nWe specialize in creating custom chatbots that integrate seamlessly with your \nbusiness environment. Using semantic search and large language models, our')]

If you're interested in building RAG from scratch, this video is for you.

NL2SQL Tool Integration

Now we have Web Search and RAG tool ready for the AI Agent. We only need to build the NL2SQL tool.

The SQL Agent bridges the gap between human language and SQL databases by generating and executing SQL queries from natural language questions. It enables your AI agent to answer database-related queries efficiently.

Steps to Integrate the NL2SQL Tool

1. Set Up the Database
   We are using the Chinook SQLite database as a sample dataset for testing SQL queries. Check this blog to learn how to set up your own database for testing.

    !wget https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite
    !mv Chinook_Sqlite.sqlite Chinook.db
   

2. Initialize the Database Connection
   Connect to the SQLite database using the LangChain SQLDatabase utility:

    from langchain_community.utilities import SQLDatabase
   
    db = SQLDatabase.from_uri("sqlite:///Chinook.db")
   

3. Clean SQL Queries
   This function is very important. We often see this problem in many of our client’s projects that the SQL query generated from the LLM consists of some unnecessary symbols, texts, backticks, etc. As a result, you will get an error while executing this query. Thus, we require a function to clean up these additional texts.

import re

def clean_sql_query(text: str) -> str:
    """
    Clean SQL query by removing code block syntax, various SQL tags, backticks,
    prefixes, and unnecessary whitespace while preserving the core SQL query.

    Args:
        text (str): Raw SQL query text that may contain code blocks, tags, and backticks

    Returns:
        str: Cleaned SQL query
    """
    # Step 1: Remove code block syntax and any SQL-related tags
    # This handles variations like ```sql, ```SQL, ```SQLQuery, etc.
    block_pattern = r"```(?:sql|SQL|SQLQuery|mysql|postgresql)?\s*(.*?)\s*```"
    text = re.sub(block_pattern, r"\1", text, flags=re.DOTALL)

    # Step 2: Handle "SQLQuery:" prefix and similar variations
    # This will match patterns like "SQLQuery:", "SQL Query:", "MySQL:", etc.
    prefix_pattern = r"^(?:SQL\s*Query|SQLQuery|MySQL|PostgreSQL|SQL)\s*:\s*"
    text = re.sub(prefix_pattern, "", text, flags=re.IGNORECASE)

    # Step 3: Extract the first SQL statement if there's random text after it
    # Look for a complete SQL statement ending with semicolon
    sql_statement_pattern = r"(SELECT.*?;)"
    sql_match = re.search(sql_statement_pattern, text, flags=re.IGNORECASE | re.DOTALL)
    if sql_match:
        text = sql_match.group(1)

    # Step 4: Remove backticks around identifiers
    text = re.sub(r'`([^`]*)`', r'\1', text)

    # Step 5: Normalize whitespace
    # Replace multiple spaces with single space
    text = re.sub(r'\s+', ' ', text)

    # Step 6: Preserve newlines for main SQL keywords to maintain readability
    keywords = ['SELECT', 'FROM', 'WHERE', 'GROUP BY', 'HAVING', 'ORDER BY',
               'LIMIT', 'JOIN', 'LEFT JOIN', 'RIGHT JOIN', 'INNER JOIN',
               'OUTER JOIN', 'UNION', 'VALUES', 'INSERT', 'UPDATE', 'DELETE']

    # Case-insensitive replacement for keywords
    pattern = '|'.join(r'\b{}\b'.format(k) for k in keywords)
    text = re.sub(f'({pattern})', r'\n\1', text, flags=re.IGNORECASE)

    # Step 7: Final cleanup
    # Remove leading/trailing whitespace and extra newlines
    text = text.strip()
    text = re.sub(r'\n\s*\n', '\n', text)

    return text

We create this function to process the LLM-generated SQL Query Output. If you have a better one, feel free to share with everyone in the comment 🙂

4. Create the NL2SQL Tool
   Define a tool that generates and executes SQL queries:

    from langchain.chains import create_sql_query_chain
    from langchain_community.tools.sql_database.tool import QuerySQLDataBaseTool
    from operator import itemgetter
    import re
    from langchain_core.output_parsers import StrOutputParser
    from langchain_core.prompts import PromptTemplate
    from langchain_core.runnables import RunnablePassthrough, RunnableLambda
   
    class SQLToolSchema(BaseModel):
        question: str
   
    @tool(args_schema=SQLToolSchema)
    def nl2sql_tool(question):
      """Tool to Generate and Execute SQL Query to answer User Questions related to chinook DB"""
      print("INSIDE NL2SQL TOOL")
      execute_query = QuerySQLDataBaseTool(db=db)
      write_query = create_sql_query_chain(llm, db)
   
      chain = (
          RunnablePassthrough.assign(query=write_query | RunnableLambda(clean_sql_query)).assign(
              result=itemgetter("query") | execute_query
          )
      )
   
      response = chain.invoke({"question": question})
      return response['result']
   

5. Test the Tool
   Use a sample query to verify functionality:

    question = "How many employees are there?"
    result = nl2sql_tool.invoke({"question": question})
    print(f"Answer: {result}")
   

    # output
    INSIDE NL2SQL TOOL
    Question: How many employees are there?
    Answer: [(8,)]
   

Combining the Tools

To create a fully functional LangGraph AI agent, you need to integrate the WebSearch, RAG, and NL2SQL tools into the LLM. And then you will get a search agent, a rag agent, and an SQL agent - all in one.

Here’s how to combine the tools:

tools = [web_search_tool, retriever_tool, nl2sql_tool]
llm_with_tools = llm.bind_tools(tools)

With this configuration, your LLM can invoke the appropriate tool based on the user’s query, ensuring dynamic and context-aware responses.

Building the LangGraph

LangGraph enables you to define a stateful workflow for your AI agent. By structuring nodes and edges, you can define how the agent processes user inputs and transitions between tools.

Steps to Build the LangGraph

1. Define the State: Create a State dictionary to manage the agent’s inputs and outputs.

    from typing import Annotated
    from langgraph.graph import StateGraph
    from langgraph.graph.message import add_messages
   
    class State(TypedDict):
        messages: Annotated[list, add_messages]
   

2. Add Nodes: Add nodes for the chatbot and tools to handle user queries and invoke the tools.

    def chatbot(state: State):
        return {"messages": [llm_with_tools.invoke(state["messages"])]}
   
    graph_builder = StateGraph(State)
    graph_builder.add_node("chatbot", chatbot)
   
    tool_node = ToolNode(tools=[web_search_tool, retriever_tool, nl2sql_tool])
    graph_builder.add_node("tools", tool_node)
   

3. Define Edges: Use conditional edges to determine when the agent should switch between nodes.

    from langgraph.prebuilt import tools_condition
   
    graph_builder.add_conditional_edges("chatbot", tools_condition)
    graph_builder.add_edge("tools", "chatbot")
    graph_builder.set_entry_point("chatbot")
   

4. Compile the Graph: Finalize the graph for execution.

    graph = graph_builder.compile()
   

Testing the AI Agent

Once the LangGraph is set up, you can test the agent by simulating user inputs. This ensures the tools and workflows are functioning as expected.

Interactive Testing

Run the following code to test your AI agent interactively:

config = {"configurable": {"thread_id": "1"}}

while True:
    user_input = input("User: ")
    if user_input.lower() in ["quit", "exit", "q"]:
        print("Goodbye!")
        break

    for event in graph.stream({"messages": [("user", user_input)]}, config):
        for value in event.values():
            print("Assistant:", value["messages"][-1].content)

You can provide queries like:

• You can ask questions based on your knowledge-base (Trigger Retriever or RAG tool)

• "What is the current weather in Delhi?" (Trigger Websearch tool)

• "Tell me the name of any two employees from the database" (Trigger NL2SQL tool)

The AI agent will invoke the appropriate tool to generate responses.

Visualizing the LangGraph

Visualization helps you understand the workflow of your AI agent and how it transitions between nodes and tools.

from IPython.display import Image, display

try:
    display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
    print("Error generating graph visualization.")

The generated diagram will showcase nodes (chatbot, tools) and the transitions between them, providing a clear overview of your AI agent’s workflow.

Improvements

1. You can implement methods like Contextual Retrieval, reranking, etc to improve the performance of the Agentic RAG system.

2. You can try implementing few-shot examples, dynamic table selection, custom SQL query generation chain, etc to improve the SQL Agent tool. We would recommend watching this Complete Natural Language to SQL video for a better understanding.

3. Moreover, you can always improve the agent by modifying graph creation, writing better prompts, adding human-in-the-loop, etc.

Conclusion

This concludes our step-by-step guide to creating a custom AI agent using LangGraph with Web Search, RAG, and NL2SQL tools. By following this tutorial, you've built an AI agent capable of performing diverse tasks such as retrieving real-time data, answering questions based on document-based knowledge, and executing SQL queries directly from natural language commands. This hands-on approach has equipped you with practical knowledge of integrating multiple tools into a stateful AI workflow using LangGraph.

If you found this guide useful and want to explore more advanced techniques, don’t forget to check out our other tutorials. At FutureSmart AI, we help businesses develop state-of-the-art AI solutions tailored to their needs. For inquiries, feel free to reach out to us at contact@futuresmart.ai.

For real-world examples of our work, take a look at our case studies, where we showcase the practical value of our expertise.

In recent months, Retrieval Augmented Generation (RAG) has emerged as a powerful pattern for enhancing Large Language Models (LLMs) with private or domain-specific knowledge. While frameworks like LangChain and LlamaIndex have made RAG implementation more accessible, they can sometimes feel like black boxes, making debugging and customization challenging.

Why Build RAG From Scratch?

When I created a tutorial on building RAG applications with LangChain, many developers reached out with a common challenge: debugging LangChain applications was becoming increasingly difficult. The layers of abstraction, while convenient, were obscuring the underlying mechanics of how RAG actually works.

Consider these common pain points with framework-based RAG implementations:

• Debugging Complexity: When something goes wrong, tracing the issue through multiple layers of framework abstraction can be time-consuming

• Documentation Overhead: Understanding framework-specific concepts often requires navigating extensive documentation, taking focus away from core RAG principles

• Limited Control: Framework abstractions can make it harder to customize specific components or optimize for your use case

• Version Dependencies: Framework updates can introduce breaking changes or compatibility issues

A Framework-Free Approach

In this guide, we'll build a complete RAG application from scratch using only fundamental libraries:

# Core dependencies
chromadb        # Vector database for document storage
openai          # LLM API access
pypdf2          # PDF document processing
python-docx     # Word document processing
sentence-transformers  # Text embeddings

Our implementation will include all essential RAG components:

1. Document Processing: Handle multiple document formats (PDF, DOCX, TXT)

2. Vector Storage: Implement semantic search using ChromaDB

3. LLM Integration: Direct interaction with OpenAI's API

4. Conversational Memory: Support for follow-up questions and context

What Makes This Approach Different?

By building without frameworks, you'll gain:

• Deep Understanding: See how each component of RAG works and interacts

• Complete Control: Customize any part of the pipeline to suit your needs

• Simplified Debugging: Trace issues directly to their source

• Easy Maintenance: No framework-specific knowledge required

Here's a high-level overview of our architecture:

This architecture shows how we'll:

• Process and index documents into our knowledge base

• Handle user queries through semantic search

• Maintain conversation history

• Generate contextual responses

Document Processing and Indexing

The foundation of any RAG system is its ability to process and index documents effectively. In this section, we'll build a robust document processing pipeline that can handle multiple file formats and prepare documents for semantic search.

Document Loading Utilities

First, let's create utilities to handle different document formats. Our implementation supports PDF, DOCX, and plain text files:

import docx
import PyPDF2
import os

def read_text_file(file_path: str):
    """Read content from a text file"""
    with open(file_path, 'r', encoding='utf-8') as file:
        return file.read()

def read_pdf_file(file_path: str):
    """Read content from a PDF file"""
    text = ""
    with open(file_path, 'rb') as file:
        pdf_reader = PyPDF2.PdfReader(file)
        for page in pdf_reader.pages:
            text += page.extract_text() + "\n"
    return text

def read_docx_file(file_path: str):
    """Read content from a Word document"""
    doc = docx.Document(file_path)
    return "\n".join([paragraph.text for paragraph in doc.paragraphs])