Introduction
In today's world, chatbots have become an essential tool for businesses to interact with their customers. With the advancements in Natural Language Processing (NLP) technology, chatbots are becoming smarter and more efficient. OpenAI, a leading AI research organization, has developed a powerful suite of tools for NLP tasks, including text embeddings and transcription services through their Whisper API.
In this blog, we will explore OpenAi Whisper and its use cases and create a YouTube question-answering chatbot that is integrated with OpenAI's Whisper API for accurate and efficient transcription of audio and video content. We will also leverage OpenAI's text embedding model, ChatGPT, to understand user input and generate relevant responses. Finally, we will use a vector database to find semantically matching content for the given user request.
By the end of this blog, you will have a deep understanding of how to leverage OpenAI's tools to create a powerful chatbot that can answer your queries in a natural and seamless way. So, let's dive in and see how we can create an intelligent chatbot from scratch.
Unboxing OpenAi Whisper
OpenAI has recently unveiled its latest breakthrough in automatic speech recognition (ASR) technology - Whisper. This system is trained on a massive 680,000 hours of multilingual and multitask supervised data sourced from the web, making it a highly accurate and robust transcription tool.
One of the key benefits of Whisper is its ability to transcribe audio with 50% fewer errors than previous models. It is designed to be highly robust to accents, background noise, and technical language, making it an ideal tool for a variety of applications.
Whisper also offers impressive multilingual capabilities, enabling transcription in 99 different languages, as well as translation from those languages into English. It also supports full punctuation, making it even easier to use for various applications.
OpenAi Whisper Use Cases
OpenAI’s Whisper API allows transcription service providers to transcribe audio and video content in multiple languages accurately and efficiently. The API's advanced machine-learning algorithms enable it to transcribe the audio in near real-time, ensuring faster turnaround times.
Another advantage of using the Whisper API is its support for multiple file formats, including MP3, WAV, and FLAC. This feature provides greater flexibility, allowing transcription service providers to work with a wide range of audio and video files.
So now the time has come to do some hands-on coding by exploring a few use-cases of Whisper.
Use Case 1: Subtitle Generation
Step 1
pip install openai-whisper
Install OpenAi Whisper library
Step 2
import whisper
Install the libraries
Step 3
model = whisper.load_model("base")
result = model.transcribe("provide path of your audio file")
print(result["text"])
Load the base model from the whisper library and store it in a variable. Then pass your audio file to the whisper model and it will generate subtitles for the audio using its transcribe method and finally print the transcribed text.
Use Case 2: Audio Language Translation
Whisper is able to perform voice translation from a vast set of languages to English.
Follow Steps 1 and 2 of Whisper Use Case 1
Step 3
model = whisper.load_model("base")
result = model.transcribe("provide path of your audio file", task = 'translate')
print(result["text"])
Load the 'base' model from the whisper library and store it in a variable. Then pass the audio path and task = 'translate' as parameters to the whisper transcribe method and store the returned object in a variable and finally print the translated audio text.
Let's Create A Youtube Question Answering Chatbot
With this app, we can interact with youtube videos in the form of a Q/A chatbot. First, we will generate subtitles of our desired youtube videos and then we will create the embeddings and we will store the embeddings in a vector database(In this case Pinecone). Finally, we will be using Streamlit to create the interface of our chatbot app.
Install The Requirements
openai
pinecone-client
streamlit
streamlit-chat
pandas
numpy
pytube
python-dotenv
Fork the repository from GitHub - https://github.com/PradipNichite/FutureSmart-AI-Blog/tree/main/Youtube%20Q-A%20Chatbot
Open the folder in your local machine and execute the command in your terminal
pip install -r requirements
This will install all the libraries required to execute the project.
Create a .env file To Initialize API Keys
openai_key = "Enter your openai key"
pinecone_key = "Provide your pinecone key"
Go to openai.com and get your API key.
Retrieving Content From Youtube Videos
Import The Libraries
import openai
import tempfile
import numpy as np
import pandas as pd
from pytube import YouTube, Search
import os
Initialize a dictionary to store the data extracted from youtube videos using pytube and whisper libraries
openai.api_key = os.getenv("openai_key")
video_dict = {
"url": [],
"title": [],
"content": []
}
Extracting Audio Content And Transcribing It Using Whisper
def video_to_audio(video_URL):
video = YouTube(video_URL)
video_dict["url"].append(video_URL)
try:
video_dict["title"].append(video.title)
except:
video_dict["title"].append("Title not found")
audio = video.streams.filter(only_audio=True).first()
temp_dir = tempfile.mkdtemp()
variable = np.random.randint(1111, 1111111)
file_name = f'recording{variable}.mp3'
temp_path = os.path.join(temp_dir, file_name)
output = audio.download(output_path=temp_path)
audio_file = open(output, "rb")
textt = openai.Audio.translate("whisper-1", audio_file)["text"]
return textt
video_to_audio: The video_to_audio method takes a youtube URL as an argument. Then we create an object for the Youtube class and name it a video and then we extract the attributes of the video like title, etc.
Next, we convert the video to audio using the below code
audio = video.streams.filter(only_audio=True).first()
and create a temporary directory to store the audio file.
Finally, we extract the subtitles from the audio using openai 'whisper-1" model and save it in a text variable and return the variable.
Saving The Data To A Dataframe
def create_dataframe(data):
df = pd.DataFrame(data)
df.to_csv("history.csv")
s = Search("Youtube video title")
print(len(s.results))
for ele in s.results[0:5:1]:
transcription = video_to_audio(ele.watch_url)
print(transcription)
print("\n\n\n")
video_dict["content"].append(transcription)
create_dataframe(video_dict)
print("Created Dataframe")
create_dataframe: This method takes a dictionary as an argument and converts that dictionary into a pandas data frame and then stores it in a CSV file.
We search for youtube videos using the Search method. Then we loop through each of the videos and generate its subtitles and store them in a dictionary that we had initialized earlier and store all the data in a CSV file.
Transferring Youtube Content To Pinecone
Import The Libraries
import pinecone
import pandas as pd
import openai
import os
Pinecone is a cloud-native vector database that allows us to build high-performance vector search applications. If you are interested to learn more about Pinecone and its use cases then refer to this link - https://docs.pinecone.io/docs/overview
Instantiate Pinecone Index
pinecone.init(api_key=os.getenv("pinecone_key"), environment="us-east-1-aws")
pinecone.create_index(
"demo-youtube-app",
dimension=1536,
metric="cosine",
pod_type="p1"
)
index = pinecone.Index("demo-youtube-app")
print(index.describe_index_stats())
Get Embeddings From The Context
def get_embedding(text):
response = openai.Embedding.create(
input=text,
model="text-embedding-ada-002"
)
return response['data'][0]['embedding']
An embedding is a numerical vector representation of words or phrases in a high-dimensional space. It captures the context and meaning of words by analyzing large amounts of text data using machine learning algorithms.
If you want to know more about embeddings and their use cases then refer to our blog
Save The Embeddings To Pinecone Database
def addData(index,url, title,context):
my_id = index.describe_index_stats()['total_vector_count']
chunkInfo = (str(my_id),
get_embedding(context),
{'video_url': url, 'title':title,'context':context})
index.upsert(vectors=[chunkInfo])
The arguments passed to the method are -
index: pinecone index where we will upsert our data.
url : Youtube video url
title: Youtube video title, we will pass this as metadata so that when we query our database, we can also look into the title and URL of the video to confirm the accuracy of the generated answer
context: Subtitles generated from the video
Finally, with the help of the upsert method, we append data to the vector database.
Answering User Queries From Stored Youtube Content
Import The Libraries
import streamlit as st
import openai
from streamlit_chat import message
import pinecone
import os
import pinecone_utils
Find Context Semantically Similar To User Query
def find_top_match(query, k):
query_em = pinecone_utils.get_embedding(query)
result = index.query(query_em, top_k=k, includeMetadata=True)
return [result['matches'][i]['metadata']['video_url'] for i in range(k)], [result['matches'][i]['metadata']['title']
for i in range(k)], [
result['matches'][i]['metadata']['context']
for i in range(k)]
With this method, we are able to find the most related video based on the input query.
The method takes two parameters - query which is the user query and k which specifies the number of top results to return.
The variable query_em stores the vector representation of the query generated by the openai embedding model.
The index.query() is called to query the vector database. top_k parameter is used to specify the number of most relevant results to return and the includeMetadata parameter specifies whether to include any metadata associated with the search results.
Integrate ChatGPT To Provide Answers
def get_message_history(contexts):
message_hist = [
{"role": "system",
"content": """As a Bot, it's important to show empathy and understanding when answering questions.You are a smart AI who have to answer the question only from the provided context If you
are unable to understand the question and need more clarity then your response should be 'Could you please be
more specific?'. If you are unable to find the answer from the given context then your response should be 'Answer is not present in the provided video' \n"""},
{"role": "system", "content": contexts},
]
return message_hist
def chat(user_query, message, role="user"):
message_history.append({"role": role, "content": f"{var}"})
completion = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=message
)
reply = completion.choices[0].message.content
message_history.append({"role": "assistant", "content": f"{reply}"})
return reply
get_message_history sets up the initial conversation between the user and the AI tutor.
We pass the most similar context passed by the vector database to this method to ensure that answers are only generated from the provided context.
chat takes the message, user_query, and the role as input, makes an API call to chatGPT, and returns the generated response.
Process User Query And Generate Bot Response
response_container = st.container()
textcontainer = st.container()
with textcontainer:
user_input = get_text()
if st.session_state.past or user_input:
urls, title, context = find_top_match(user_input, 1)
message_history = get_message_history(context[0])
with st.spinner("Generating the answer..."):
response = chat(user_input, message_history)
st.session_state.past.append(user_input)
st.session_state.generated.append(response)
st.subheader("References")
link_expander = st.expander("Context obtained from url")
link_expander.write(urls)
First, the user asks a query and that query gets stored in the use_input variable.
Next, we pass the user query to the find_top_match() method which queries the vector database and provides the document with the highest semantic score.
Then we pass this document and query to chatGPT and its answers to our queries based on the context provided to it.
ChatBot In Action
Congratulations, you have made your AI chatbot with semantic search functionalities.
Now it is time to execute your chatbot
Execute the following command in your terminal
streamlit run app.py
Conclusion
In conclusion, building a YouTube Q/A chatbot with OpenAI Whisper, Pinecone vector database, ChatGPT, and OpenAI embeddings was an exciting project that showcases the potential of cutting-edge technologies in natural language processing. By integrating these tools and technologies, we were able to create a chatbot that accurately understands and responds to user queries in real time. Furthermore, with the addition of Streamlit as a frontend, we were able to create an engaging and interactive user interface that enhances the overall user experience. This project is just the beginning of what is possible with advanced NLP techniques and AI technologies, and we look forward to seeing how these innovations will continue to transform the world of conversational AI.
If you are more interested to learn about embeddings and their use cases, then be sure to check out the below YouTube tutorial.
To know more about Vector Database and its applications, refer to the following video
Also, check the video demonstrations of a chatbot integrated with ChatGPT
To learn about more interesting and cool applications of LLMs look into our other Blogs and YouTube channel.
Also, want to learn about the state-of-the-art stuff in AI? Don't forget to subscribe to AI Demos. A place to learn about the latest and cutting-edge tools in AI!
