Coffee Shop Chatbot with RAG

Problem/Opportunities:

When building a bot to answer queries from private data, this creates difficulties for frontier models, as models are trained on their own data and lack access to private proprietary data. Although there are workarounds like coding proprietary data into the program, it is not an effective solution.

Solution:  

Allow AI to reference the proprietary data by reading it in as a knowledge database, converting it to a vector database & using the RAG (Retrieval Augmented Generation) algorithm to retrieve the relevant information without hallucination. This algorithm pulls in facts to handle complex queries and better inform users with the correct information.

Approach:

Demonstrate the solution by using an imaginary coffee shop chain as a walk-through sample.

• First, we have defined the company structure laydown all the information that needs to be communicated to their customer. Structure this data accordingly under file/folders in Markdown format. (Markdown is preferable as the format has structure-aware chunking and improves token efficiency, hence bridging the gap between human-readable text and structured data)

• Read in the directory with folders/files

• Split them into smaller chunks of text by specifying parameters in the codes

• Convert each of these chunks into a vector store with vector embeddings (converting text into numbers & representing “a point in space”

• Review 2D and 3D format in Chroma to see how individual “point in space” look and how the spreading looks

• Use LangChain to bring in all components

  • Users submit a query, then convert it into a vector

  • Create a prompt template with an underlying few-shot template

  • Take in queries

  • Set up a conversation chain including a retriever, prompt, conversation memory and data from the vector store

  • Find the closest response and return it to the user

• Snap on a Gradio frontend to bring them together to make this look user-friendly

​

Challenges:

Tuning the program is required, and accurate answers will be achieved by creating and setting up a detailed few-shot template, giving detailed instructions on where and when to look in the Markdown files, and providing examples.

With identical parameters (e.g. chunking, embeddings, retriever settings) but different LLMs, the result and response may vary, and ChatGPT and frontier models seem to handle them better. The difference boils down to the underlying model’s capabilities rather than a “bug” in the chosen LLM, and some models may be less polished out-of-the-box model.

Sometimes tweaking the parameters and tuning the model might help without choosing another LLM, and hence experimenting is a crucial step before choosing another LLM.

​

Conclusion:

A super useful program to answer complex queries based on a knowledge database using RAG.  This solution allows AI to reference the proprietary data and can tailor accurate responses. The most important step is to experiment with parameters, e.g. use the best chunk strategy in the parameters and ensure the chunk will not be cut off in the middle, i.e. using TextSplitter that intelligently prioritises paragraphs based on \n\n and \n. Set the overlap chunk size to be around 15%-20%; do not set retriever kwargs too high and ensure the temperature parameter is a low number to preserve consistency.  Further accuracy can be obtained by creating a detailed few-shot template and supplying lots of examples to guide and set boundaries for the model. This exercise needs a lot of tweaking and testing effort.

​

Future considerations:

The current algorithm is to use it as an AI chatbot to answer queries. This logic can be expanded and, instead of running as a standalone, it can be incorporated into a company website to answer customer queries. Similar solution can be tailor-made for inhouse company use, instead of searching for information throughout the company, create knowledge base for different departments accordingly, snap on a user access logic and this will be able to deal with internal queries to improve workflow.

Application of use:

Nowadays, it is common to enhance a website or mobile application with AI adoption. Instead of choosing a commercial-ready AI chatbot, a program like this can be created, as it will have more control over data privacy if both the knowledge and vector database are stored locally. We can even extend this further to run and train LLMs locally too. The same methodology and use can be applied to other industries; to name a few, this can be used in legal to answer queries, do analysis and perform document search, customer queries & support for basically any industry.