Data Science Query Assistant - Part II
A code walkthrough of the Data Science Query Assistant, showing how Cohere, EdenAI, LangChain, and spaCy power topic explanations, quizzes, flash cards, CSV analysis, and website chat.
Building a tool that can explain a concept, quiz you on it, read a spreadsheet, and chat about a website all in one app sounds like four separate projects stitched together. In this follow-up video, Sahil Padyal opens up the actual code behind the Data Science Query Assistant, a Streamlit application built with Cohere, EdenAI, and spaCy, and walks through how each of those pieces fits into one coherent system.
Setting up models and prompt templates
The project begins with environment setup: importing the necessary libraries and loading API keys for the two language models the app relies on, Cohere and EdenAI. Both offer free API keys after signing in, Cohere through GitHub login and EdenAI through its own signup, and Padyal notes that EdenAI is also useful for image and video generation in other projects. Once the keys are in place, the model is initialized with a temperature of 0.75 and a max token count of 800, values chosen because the app needs to generate a fair amount of theoretical text for each query.
From there, the code defines prompt templates using LangChain. A prompt template lets the app plug a variable, such as a topic typed in by the user, into a pre-written instruction, so the model always receives a consistent, well-formed request like "as a data science teacher, could you explain the concept of [topic]." A second template follows the same pattern to generate quiz questions and answers in a fixed format, which matters later when the app needs to parse the model's response back into structured data.
Generating flash cards from key points
One of the trickier parts of the build was flash card generation. The app first gets a full explanation from the model, then needs to identify the key points within that response worth turning into flash cards, and finally sends those key points back to the model to get a short, two-line explanation for each one. To pull out those key points, the app uses spaCy, a natural language processing library with models for tasks like embedding, parsing, and named entity recognition. Here, spaCy's en_core_web_sm model parses the response text and matches it against a list of predefined data science keywords, extracting whichever of those terms appear so they can be turned into flash cards automatically.
Parsing quiz responses and building the interface
Because the model's quiz output follows the same fixed format defined in the prompt template, the app can reliably parse out individual questions and answers from the raw response text. The interface itself is built with Streamlit, including its newer st.chat module for chat-style interactions, along with Streamlit's container components to lay out the quiz and flash cards. To avoid Streamlit's default behavior of re-running the entire script on every button click, the app uses st.session_state to preserve values across interactions, keeping the flow from resetting unexpectedly.
Adding CSV analysis with EdenAI
The next feature is CSV file analysis, built on EdenAI's model, which is provided by OpenAI and uses GPT-3.5 Instruct under the hood. This part of the app uses a LangChain agent, specifically create_csv_agent from LangChain's experimental agents module, which is purpose-built for interacting with CSV files. The interface lets a user upload a file, store it in a variable, and type a natural language question; the agent then runs against the file using agent.run() to produce an answer, an implementation Padyal describes as straightforward once the right LangChain agent was identified.
Chatting with a website through retrieval
The final feature lets users converse with content pulled live from a website, built as a retrieval-augmented generation pipeline. It starts with embeddings, using LangChain's free, open-source embedding option rather than a paid alternative, to convert website content into a form the app can search. A vector store is created from the given URL: the app loads the document, splits it into smaller chunks so the model isn't forced to process the entire page at once, and then retrieves the most relevant chunk based on the user's question.
On top of that vector store, the app builds a retrieval chain using Cohere, combining the language model, the retriever, and a prompt that reformulates the user's input into a better search query given the conversation so far. A second function builds the conversational RAG chain itself, answering the user's question based on the context retrieved from the website and the ongoing chat history, so the model doesn't need to be re-told the website's context on every new question.
Key takeaways
- The app combines Streamlit, Cohere, EdenAI, and spaCy into one system with four distinct features.
- Prompt templates in LangChain let variables like topic or key point get inserted into consistent, reusable instructions.
- spaCy's parsing model extracts key terms from a generated explanation to drive automatic flash card creation.
- CSV analysis is handled by a dedicated LangChain agent designed specifically for querying CSV files.
- Website chat is built as a retrieval-augmented generation pipeline: embed the site, chunk it, retrieve relevant chunks, and answer using both the retrieved context and chat history.
st.session_stateis used to prevent Streamlit's default script rerun behavior from resetting the app's state on every interaction.
Try it yourself
The full code for the Data Science Query Assistant is available on GitHub, and Padyal encourages viewers to experiment with it directly, including trying different LLM models to compare results. This kind of applied, end-to-end AI build is representative of the project work coming out of the Humanitarians AI community.
Full transcript(auto-generated, with timestamps)
[0:01]Hello and welcome to our video where we will dive into the data science quare assistant uh which is a powerful tool built using streamlet coher EDI and spacy this application is designed to answer your data science question generate quizzes analyze csvs data sets and facilitate Dynamic conversation using information from specified websites so let's get started by exploring the code functionality and the code behind them so first of all we'll talk about setting up the environment so our project basically begins with importing necessary libraries and setting up environment variables for our llm models which is goare and Eden API so there you can see I have uh established uh the API Keys here
[0:59]So if you want to get API keys for yourself you can easily get it uh just if you go to their website so yeah so you can see you can build conversational apps with r using this and you can just click on get your API key and log in with your GitHub ID and and you'll be able to get the AP Pi key for yourself and similarly you can get Eden AI which is also a really good AI model so this is also what you can just sign in and get apis for your models it's a really good model it's also available for generating videos and
[1:55]Image generation so I'm thinking of using it forther projects so coming back to the code so just make sure you uh safely import the API Keys here otherwise it will not work so the first thing is like defining to get your model to initialize so you can see here I have initialized my model so I've set the temperature to 75 and Max tokens to 800 because I have to generate a lot of theoretical text uh for the queries so I've increased it otherwise you can keep it low so here uh what I have done is I have created a prompt template using Lang chain so here you can see uh there
[2:59]I have imported a prompt template from Lang chain so what prompt template does it that it's already give a basic template to your model uh where you don't have to enter a full sentence every time you're going to ask something so here you can see as as a data science teacher could you explain the concept of topic so here topic is basically a variable which only will be you'll be typing here uh so here if you type in anything that will be uh sent as in variable topic to the prom template see so you can see how it does and you can also Play Along With The Code by
[3:56]Telling LM to provides more information and also give code example right now I've commented it out because it was getting out of tokens for because of I've used a trial API for that so this is basically another Pro prompt template that I've used to generate quizzes uh for the query model so here you can see I've given it a basic uh prompt generate three questions answers on the topic and this topic will be given as the variable which you'll type in and here I given an example format and how the question and answer should be given so here I have uh defined uh the basic
[4:55]Functions to get response from the model so here I initialized the get goir model uh which is this function called this function and then I've given it the prompt and I've have set the prompt into the chain so then it will give out your response so this function is used for getting the response and similarly I have defined a response function for quizzes now uh after the quizzes uh what you can get is you can see get detail explanation for key points so basically uh for building the flash cards uh it was a tricky part uh to be uh what I I can say to build a flash cards because the flash
[5:53]Cards were going to be built uh from the response what the model will give out as a response and from that response uh we'll have to select uh like the key points to be buil as a flash cards and then these key points uh will be sent back to the model to give out response so this is uh what I have done to get response so here uh I have developed this function to get detail so here I have set a prompt explain in two lines for building a flash card and then I'll give the key point which is been extracted from the First Response of the
[6:40]Model so on based on this uh you can see uh the flash cards are being generated okay so then I'll send here I'm sending uh this PR back to the model with the key point so it will return the response so this is uh a basic function for create generating flash cards based on the key points so I have declared flash cards as an array and for every key points in the key points that have been extracted I have sent it uh to the get detailed explanation function uh which is here and it will get the response automatically so uh one thing that I've used uh to par
[7:43]Quiz questions from the response is like I have given uh basically uh so here I'll show you I have given the llm uh a format in which uh it will send out a quiz questions and answers questions and answers and there in passing the quiz questions from the response I have set the same uh format in which uh the quiz questions will appear and then after that it will just pass the questions and answers apart from the response and set them in a flash cards so this was a bit tricky and for this I have used uh a library called spacy here you can see Spacey so it's a really useful
[8:48]Library for NLP paring and everything so it has very different models so this is how you can use py in any of your projects so okay so here are different CS of spy from what you can do this embeding transformation llm training models layers and architecture so I've used the spaces um parsing and M reading model here uh so that it can just take out the given key points uh from the response every time so which is what I've used and also for generating uh flash cards I've used the spaces model you can see and spy. load uh en code VB SM so it basically
[9:55]Pars the text and then dig out the keywords that you have mentioned so here I have mentioned few of the data science keywords here you can see so wherever uh uh this model sees this word in the response it will uh take out these words and send it back to the model and then generate flash cards based on that so this is how I have generated flash card so this is exactly the same function which I've used for flash cards and then displaying the flash cards and streamlit with using containers you can see so this is a very basic thing which is used in building uh streamlit UI that
[10:51]I've used and similarly the same for quiz quiz flash cards now this is what uh streamlit UI that I've used here so I have used streamlet uh st. chat module uh which is a very new module in streamlet which is used to build chat based apps uh which wasn't previously in streamlet so that's a one code Improvement in I would say so this is the basic uh UI out outlet that I have built these are the tabs and this is the basic display message that will be displayed and here I have used a session State uh which is a function in streamlet that keeps your state active uh because in
[12:03]Streamlet every time you click a button uh streamlit refreshes and rereads your script every time so that's a downside of streamit so for that you can just use session. State and then uh that script won't run beyond that function okay now coming to the next thing that is building uh the CSV file reader for analysis so here I used the edeni model here you can see uh here I've initialized the EDI model so it's basically provid Ed by open Ai and it use g3.5 instruct so it's a really powerful model I would say and here I've defined the function to get basically output and Define the
[13:16]Chains uh from Lang chain to get the output uh for uh the CSV files I have given veros equals to True here and this is the uploaded file that will go into the llm and here the model that I have used uh so basically this create CSV uh that's a inut of Lang chain here you can see uh this is where I have imported L chain from Lang chain experimental agents so this is basically a lang chain agent which is used to interact with the c CSV files so it was a pretty easy implementation so here you can see uh in the tab to this is the UI that I've
[14:12]Created you basically upload a file store it in a variable and then give that file uh to the model and here I've made a query variable which you can type in any question and that will basically go to the llm for analyzing so yeah here I've called the get answer function in which basically you'll get answer from the agent and you have to use agent. run uh to run any query to the llm from the file you use this uh agent. run every day so this is how uh the second part was built defining function from rag website
[15:24]Check so this is uh where I have implemented uh the retrieval argumented generation based uh functionality in my uh project you can see where it retrieves the data from the website and split it into text and give you the output so I'll show you how it was built so for these I have used uh embedding function here you can see embedding is used you can also use open embedding if you have that but I was using the free open source embedding which Lang chain provides so you can basically import these embeddings here and this is the model name that is used for the embedding and here you can see I have
[16:28]Imported imported this from Lang chain and after that uh I have created a vector store uh which switch basically uh create Vector stores of on the given URL and loads the document and then it splits the document into junks uh so it doesn't read take the whole document as in one because it will be uh very high in computational resource if the model has to go through document each time so it splits that document into chunks of documents and then based on the keywords it finds the right chunk of file and then provides the answer from that okay so again for that I'm using cair
[17:38]Llm you can see I've initialized the C here and so you can see uh here I have uh given a prompt uh to this retrieval chain uh which you can just see uh user input so input will be taken as what you write in the text box and then given above the conversation generate or switch query to look up in order to get the relevant information to the conversation so this is what basically a prompt I've have given it uh to the llm so that it can get basic details of your question searched and then I have initialize the retriever chain so it takes three arguments here
[18:34]Llm the retriever which is the vector store and the prompt so after that it will just return the retriever chain now coming to the next next function is that get conversational drag chain built so again you'll just use uh llm model here Define your model and provide us prompt as before uh from chat prompt template so so here you can see answer the users question based on the context below so the context and then it will place it in the chat history so context is basically what uh what the context is basically what it will be given uh to The Prompt based on
[19:48]The link so it will automatically know like what is the context of the question based on the website and you don't have to give context to your model before asking any question related to that website uh this is the basic function to just get response from the LM and there uh I have built this UI for uh for the website URL tab it's a very basic UI image which I have used the uh chat functionality of streamlit to build that yeah so this is how the code works and I would like to know your feedback on this and like yeah there you have it our
[20:59]Data science qu assistance with equipped with AI driven explanation quizzes CSV data analysis and website chat functionality and all made possible with streamlet goare Ed and Spacey this tool is basically a test statement to the power of combining different Ai and NLP Technologies to create an educational and interactive experience and thank you for watching my video uh you can dive into the code and experiment it with or play along with the code make try making changes try using uh different llm models and see how different results you get yeah so yeah that will be all yeah thank you
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53