llama 3 buffer memory.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e3023a03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please enter your query (or 'exit' to quit): what is your experience ?\n",
      "Output: My experience! I'm a Central (Enterprise) Architect at BNP Paribas with over 19 years of progressive technical leadership experience. I've worked in various domains like banking, capital markets, supply chain, healthcare, and e-commerce. My expertise includes cloud, AI, LLM, Blockchain, .NET, Java, Python, and NodeJS.\n",
      "\n",
      "I have a proven track record of delivering innovative and efficient solutions, obtaining buy-in from sponsors, defining technical requirements, consulting during development and deployment, and monitoring and management. I'm always eager to learn new technologies and experiment with them to come up with creative solutions.\n",
      "\n",
      "Throughout my career, I've held various roles such as Principal Technical Architect, AVP; Solution Architect; Senior Technical Architect; Technical Architect; Team Lead; and Sr. Software Engineer. My responsibilities have included providing technical solutions to new and existing systems, working as a consultative fashion with other functional heads/SMEs, balancing functionality, requirements, and business needs, and presenting solutions in architecture review boards.\n",
      "\n",
      "I've also had the opportunity to work on various projects, including Research Library for Merrill Lynch, and I was involved in all development phases from scratch. In addition, I introduced asynchronous calls using JavaScript, which improved the performance and user experience of the website.\n",
      "\n",
      "In terms of technology/tools used, my expertise includes Microsoft .Net Framework 2.0 with C#, ASP .Net, Classic ASP, Visual Studio 2005, XML & XSLT, VSS, SQL Server 2005, SQL Server Reporting Service, SQL Server Business Intelligence, and more!\n",
      "\n",
      "Overall, I'm a seasoned technology professional with extensive experience in leading technical teams, delivering innovative solutions, and collaborating with stakeholders to drive business results.\n",
      "Please enter your query (or 'exit' to quit): explain in java?\n",
      "Output: I am a Central (Enterprise) Architect at BNP Paribas, with over 19 years of progressive technical leadership experience and passion for technology. I have expertise in several technologies, such as cloud, AI, LLM, Blockchain, .NET, Java, Python, and NodeJS, and I have worked in multiple domains, such as banking, capital markets, supply chain, healthcare, and e-commerce.\n",
      "\n",
      "As a Central Architect, my mission is to provide/validate the solutions to the new and existing applications for the APAC and ANZ regions, ensuring information system consistency, communication, and support to local teams. I have a proven track record of delivering innovative and efficient solutions, obtaining buy-in from sponsors, defining technical requirements, consulting during development and deployment, and monitoring and management.\n",
      "\n",
      "In my previous roles, I have:\n",
      "\n",
      "* Held the position of Principal Technical Architect at AVP, where I was responsible for providing solutions to new and existing systems for both Australia and New Zealand regions.\n",
      "* Worked as a Solution Architect at Chennai, where I was responsible for designing and managing the delivery of solutions and technologies.\n",
      "* Led development teams as a Team Lead at Xchanging, where I involved in requirement Analysis, built reusable components using C#, built prototypes, R & D, peer reviews, client communication, and participated in technical discussions with Architects.\n",
      "\n",
      "I am proficient in Microsoft .Net Framework 2.0 with C#, ASP .Net, Classic ASP, Visual Studio 2005, XML & XSLT, VSS, SQL Server 2005, SQL Server Reporting Service, SQL Server Business Intelligence, and have experience working with various tools and technologies.\n",
      "Please enter your query (or 'exit' to quit): what is bnp\n",
      "Output: The answer to \"what is BNP\" is not explicitly stated in the resume text provided. However, based on the context and information available, it can be inferred that BNP likely refers to BNP Paribas, a global financial institution. Augustine Rajesh, the individual whose resume this is, mentions being a Central (Enterprise) Architect at BNP Paribas with over 19 years of experience in technology leadership.\n",
      "Please enter your query (or 'exit' to quit): exit\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_community.llms import Ollama\n",
    "import pdfplumber\n",
    "\n",
    "# Function to extract text from PDF\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    with pdfplumber.open(pdf_path) as pdf:\n",
    "        pages_text = [page.extract_text() for page in pdf.pages]\n",
    "    return \"\\n\".join(pages_text)\n",
    "\n",
    "# Replace with your PDF path\n",
    "pdf_path = r\"C:\\Users\\murugan\\Downloads\\Profile.pdf\"\n",
    "\n",
    "# Extract text from PDF\n",
    "pdf_text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Initialize an empty memory dictionary\n",
    "memory = {}\n",
    "\n",
    "while True:\n",
    "    # Get user input\n",
    "    input_txt = input(\"Please enter your query (or 'exit' to quit): \")\n",
    "\n",
    "    if input_txt.lower() == 'exit':\n",
    "        break\n",
    "\n",
    "    # Create a ChatPromptTemplate\n",
    "    prompt_messages = [\n",
    "        (\"system\", \"When asked a question, provide a relevant answer directly and precisely.\"),\n",
    "        (\"user\", f\"User query: {input_txt}\"),\n",
    "        (\"user\", f\"Resume text: {pdf_text}\")\n",
    "    ]\n",
    "\n",
    "    # Add memory to the prompt context\n",
    "    if memory:\n",
    "        prompt_messages.append((\"memory\", memory))\n",
    "\n",
    "    prompt = ChatPromptTemplate.from_messages(prompt_messages)\n",
    "\n",
    "    # Initialize the language model\n",
    "    llm = Ollama(model=\"llama3\")\n",
    "\n",
    "    # Initialize the output parser\n",
    "    output_parser = StrOutputParser()\n",
    "\n",
    "    # Build the chain\n",
    "    chain = prompt | llm | output_parser\n",
    "\n",
    "    # Invoke the chain\n",
    "    data = chain.invoke({\"query\": input_txt, \"pdf_text\": pdf_text, \"memory\": memory})\n",
    "\n",
    "    # Extract memory from data\n",
    "    if isinstance(data, dict):\n",
    "        new_memory = data.get('memory', {})\n",
    "    else:\n",
    "        new_memory = {}\n",
    "\n",
    "    # Update memory with the current interaction\n",
    "    memory.update(new_memory)\n",
    "\n",
    "    # Print the output\n",
    "    print(\"Output:\", data['response'] if isinstance(data, dict) and 'response' in data else data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6b6a7a02",
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'retrievalqa'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[14], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpdfplumber\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mretrievalqa\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m RetrievalQA, LLM, CompressionRetriever\n\u001b[0;32m      4\u001b[0m pdf_path \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mr\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mC:\u001b[39m\u001b[38;5;124m\\\u001b[39m\u001b[38;5;124mUsers\u001b[39m\u001b[38;5;124m\\\u001b[39m\u001b[38;5;124mmurugan\u001b[39m\u001b[38;5;124m\\\u001b[39m\u001b[38;5;124mDownloads\u001b[39m\u001b[38;5;124m\\\u001b[39m\u001b[38;5;124mProfile.pdf\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m      6\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mextract_text_from_pdf\u001b[39m(pdf_path):\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'retrievalqa'"
     ]
    }
   ],
   "source": [
    "import pdfplumber\n",
    "from retrievalqa import RetrievalQA, LLM, CompressionRetriever\n",
    "\n",
    "pdf_path = r\"C:\\Users\\murugan\\Downloads\\Profile.pdf\"\n",
    "\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    with pdfplumber.open(pdf_path) as pdf:\n",
    "        pages_text = [page.extract_text() for page in pdf.pages]\n",
    "    return \"\\n\".join(pages_text)\n",
    "\n",
    "def main():\n",
    "    pdf_text = extract_text_from_pdf(pdf_path)\n",
    "    # print(\"PDF text:\", pdf_text)\n",
    "\n",
    "    input_txt = input(\"Please enter your query: \")\n",
    "\n",
    "    # Initialize the language model (LLM)\n",
    "    llm = LLM(model=\"ollama3\")\n",
    "\n",
    "    # Initialize the CompressionRetriever\n",
    "    compression_retriever = CompressionRetriever()\n",
    "\n",
    "    # Prompt for the query (adjust as needed)\n",
    "    prompt = f\"User query: {input_txt}\"\n",
    "\n",
    "    # Initialize RetrievalQA\n",
    "    qa = RetrievalQA.from_chain_type(\n",
    "        llm=llm,\n",
    "        chain_type=\"stuff\",\n",
    "        retriever=compression_retriever,\n",
    "        return_source_documents=True,\n",
    "        chain_type_kwargs={\"prompt\": prompt, \"verbose\": True},\n",
    "    )\n",
    "\n",
    "    # Retrieve and answer the query\n",
    "    answer, source_documents = qa.retrieve_and_answer()\n",
    "\n",
    "    # Print the answer\n",
    "    print(\"Answer:\", answer)\n",
    "\n",
    "    # Optionally, print or handle source documents\n",
    "    if source_documents:\n",
    "        print(\"Source Documents:\")\n",
    "        for doc in source_documents:\n",
    "            print(doc)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "12a1ceaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Could not find a version that satisfies the requirement retrievalqa (from versions: none)\n",
      "ERROR: No matching distribution found for retrievalqa\n"
     ]
    }
   ],
   "source": [
    "!pip install retrievalqa\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "67edd603",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in c:\\users\\murugan\\appdata\\roaming\\python\\python311\\site-packages (24.1.2)\n"
     ]
    }
   ],
   "source": [
    "!pip install pip --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5e45e0e7",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: langchain-groq==0.1.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (0.1.3)\n",
      "Requirement already satisfied: groq<1,>=0.4.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-groq==0.1.3) (0.5.0)\n",
      "Requirement already satisfied: langchain-core<0.2.0,>=0.1.45 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-groq==0.1.3) (0.1.52)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (4.2.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (1.8.0)\n",
      "Requirement already satisfied: httpx<1,>=0.23.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (0.27.0)\n",
      "Requirement already satisfied: pydantic<3,>=1.9.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (1.10.12)\n",
      "Requirement already satisfied: sniffio in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (1.3.0)\n",
      "Requirement already satisfied: typing-extensions<5,>=4.7 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (4.9.0)\n",
      "Requirement already satisfied: PyYAML>=5.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (6.0.1)\n",
      "Requirement already satisfied: jsonpatch<2.0,>=1.33 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (1.33)\n",
      "Requirement already satisfied: langsmith<0.2.0,>=0.1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (0.1.56)\n",
      "Requirement already satisfied: packaging<24.0,>=23.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (23.2)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (8.2.2)\n",
      "Requirement already satisfied: idna>=2.8 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from anyio<5,>=3.5.0->groq<1,>=0.4.1->langchain-groq==0.1.3) (3.4)\n",
      "Requirement already satisfied: certifi in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx<1,>=0.23.0->groq<1,>=0.4.1->langchain-groq==0.1.3) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx<1,>=0.23.0->groq<1,>=0.4.1->langchain-groq==0.1.3) (1.0.5)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpcore==1.*->httpx<1,>=0.23.0->groq<1,>=0.4.1->langchain-groq==0.1.3) (0.14.0)\n",
      "Requirement already satisfied: jsonpointer>=1.9 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (2.1)\n",
      "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langsmith<0.2.0,>=0.1.0->langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (3.10.3)\n",
      "Requirement already satisfied: requests<3,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langsmith<0.2.0,>=0.1.0->langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (2.31.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3,>=2->langsmith<0.2.0,>=0.1.0->langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3,>=2->langsmith<0.2.0,>=0.1.0->langchain-core<0.2.0,>=0.1.45->langchain-groq==0.1.3) (2.0.7)\n"
     ]
    }
   ],
   "source": [
    "!pip install langchain-groq==0.1.3 --progress-bar off"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0a009941",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting langchain==0.1.17\n",
      "  Downloading langchain-0.1.17-py3-none-any.whl.metadata (13 kB)\n",
      "Requirement already satisfied: PyYAML>=5.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (6.0.1)\n",
      "Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (2.0.25)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (3.9.3)\n",
      "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.6.6)\n",
      "Requirement already satisfied: jsonpatch<2.0,>=1.33 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (1.33)\n",
      "Requirement already satisfied: langchain-community<0.1,>=0.0.36 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.0.38)\n",
      "Requirement already satisfied: langchain-core<0.2.0,>=0.1.48 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.1.52)\n",
      "Requirement already satisfied: langchain-text-splitters<0.1,>=0.0.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.0.1)\n",
      "Requirement already satisfied: langsmith<0.2.0,>=0.1.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.1.56)\n",
      "Requirement already satisfied: numpy<2,>=1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (1.26.4)\n",
      "Requirement already satisfied: pydantic<3,>=1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (1.10.12)\n",
      "Requirement already satisfied: requests<3,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (8.2.2)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (1.2.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (1.4.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (1.9.3)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.1.17) (3.21.2)\n",
      "Requirement already satisfied: typing-inspect<1,>=0.4.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.1.17) (0.9.0)\n",
      "Requirement already satisfied: jsonpointer>=1.9 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jsonpatch<2.0,>=1.33->langchain==0.1.17) (2.1)\n",
      "Requirement already satisfied: packaging<24.0,>=23.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-core<0.2.0,>=0.1.48->langchain==0.1.17) (23.2)\n",
      "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langsmith<0.2.0,>=0.1.17->langchain==0.1.17) (3.10.3)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pydantic<3,>=1->langchain==0.1.17) (4.9.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3,>=2->langchain==0.1.17) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3,>=2->langchain==0.1.17) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3,>=2->langchain==0.1.17) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3,>=2->langchain==0.1.17) (2024.2.2)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from SQLAlchemy<3,>=1.4->langchain==0.1.17) (3.0.1)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain==0.1.17) (1.0.0)\n",
      "Downloading langchain-0.1.17-py3-none-any.whl (867 kB)\n",
      "Installing collected packages: langchain\n",
      "  Attempting uninstall: langchain\n",
      "    Found existing installation: langchain 0.1.20\n",
      "    Uninstalling langchain-0.1.20:\n",
      "      Successfully uninstalled langchain-0.1.20\n",
      "Successfully installed langchain-0.1.17\n"
     ]
    }
   ],
   "source": [
    "!pip install langchain==0.1.17 --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "68267ee9",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting llama-parse==0.1.3\n",
      "  Downloading llama_parse-0.1.3-py3-none-any.whl.metadata (2.4 kB)\n",
      "Collecting llama-index<0.10.0,>=0.9.40 (from llama-parse==0.1.3)\n",
      "  Downloading llama_index-0.9.48-py3-none-any.whl.metadata (8.4 kB)\n",
      "Requirement already satisfied: SQLAlchemy>=1.4.49 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2.0.25)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.9.3)\n",
      "Requirement already satisfied: dataclasses-json in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (0.6.6)\n",
      "Requirement already satisfied: deprecated>=1.2.9.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.2.14)\n",
      "Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.0.8)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2023.10.0)\n",
      "Requirement already satisfied: httpx in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (0.27.0)\n",
      "Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.6.0)\n",
      "Requirement already satisfied: networkx>=3.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.1)\n",
      "Requirement already satisfied: nltk<4.0.0,>=3.8.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.8.1)\n",
      "Requirement already satisfied: numpy in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.26.4)\n",
      "Collecting openai>=1.1.0 (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3)\n",
      "  Downloading openai-1.35.13-py3-none-any.whl.metadata (21 kB)\n",
      "Requirement already satisfied: pandas in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2.1.4)\n",
      "Requirement already satisfied: requests>=2.31.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (8.2.2)\n",
      "Requirement already satisfied: tiktoken>=0.3.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (0.6.0)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (4.9.0)\n",
      "Requirement already satisfied: typing-inspect>=0.8.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (0.9.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.2.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.4.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.9.3)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from deprecated>=1.2.9.3->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.14.1)\n",
      "Requirement already satisfied: click in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk<4.0.0,>=3.8.1->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (8.1.7)\n",
      "Requirement already satisfied: joblib in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk<4.0.0,>=3.8.1->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.2.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk<4.0.0,>=3.8.1->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2023.10.3)\n",
      "Requirement already satisfied: tqdm in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk<4.0.0,>=3.8.1->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (4.66.4)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from openai>=1.1.0->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (4.2.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from openai>=1.1.0->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.8.0)\n",
      "Requirement already satisfied: pydantic<3,>=1.9.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from openai>=1.1.0->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.10.12)\n",
      "Requirement already satisfied: sniffio in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from openai>=1.1.0->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.3.0)\n",
      "Requirement already satisfied: certifi in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.0.5)\n",
      "Requirement already satisfied: idna in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.4)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpcore==1.*->httpx->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (0.14.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests>=2.31.0->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests>=2.31.0->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2.0.7)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from SQLAlchemy>=1.4.49->SQLAlchemy[asyncio]>=1.4.49->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.0.1)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from typing-inspect>=0.8.0->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.0.0)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from dataclasses-json->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.21.2)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2023.3)\n",
      "Requirement already satisfied: packaging>=17.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (23.2)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from python-dateutil>=2.8.2->pandas->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.16.0)\n",
      "Requirement already satisfied: colorama in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from tqdm->nltk<4.0.0,>=3.8.1->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (0.4.6)\n",
      "Downloading llama_parse-0.1.3-py3-none-any.whl (4.3 kB)\n",
      "Downloading llama_index-0.9.48-py3-none-any.whl (15.9 MB)\n",
      "Downloading openai-1.35.13-py3-none-any.whl (328 kB)\n",
      "Installing collected packages: openai, llama-index, llama-parse\n",
      "  Attempting uninstall: openai\n",
      "    Found existing installation: openai 0.28.0\n",
      "    Uninstalling openai-0.28.0:\n",
      "      Successfully uninstalled openai-0.28.0\n",
      "  Attempting uninstall: llama-parse\n",
      "    Found existing installation: llama-parse 0.4.2\n",
      "    Uninstalling llama-parse-0.4.2:\n",
      "      Successfully uninstalled llama-parse-0.4.2\n",
      "Successfully installed llama-index-0.9.48 llama-parse-0.1.3 openai-1.35.13\n"
     ]
    }
   ],
   "source": [
    "!pip install llama-parse==0.1.3 --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3688641c",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting qdrant-client==1.9.1\n",
      "  Downloading qdrant_client-1.9.1-py3-none-any.whl.metadata (9.5 kB)\n",
      "Collecting grpcio>=1.41.0 (from qdrant-client==1.9.1)\n",
      "  Downloading grpcio-1.64.1-cp311-cp311-win_amd64.whl.metadata (3.4 kB)\n",
      "Collecting grpcio-tools>=1.41.0 (from qdrant-client==1.9.1)\n",
      "  Downloading grpcio_tools-1.64.1-cp311-cp311-win_amd64.whl.metadata (5.5 kB)\n",
      "Requirement already satisfied: httpx>=0.20.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx[http2]>=0.20.0->qdrant-client==1.9.1) (0.27.0)\n",
      "Requirement already satisfied: numpy>=1.21 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from qdrant-client==1.9.1) (1.26.4)\n",
      "Collecting portalocker<3.0.0,>=2.7.0 (from qdrant-client==1.9.1)\n",
      "  Downloading portalocker-2.10.0-py3-none-any.whl.metadata (8.5 kB)\n",
      "Requirement already satisfied: pydantic>=1.10.8 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from qdrant-client==1.9.1) (1.10.12)\n",
      "Requirement already satisfied: urllib3<3,>=1.26.14 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from qdrant-client==1.9.1) (2.0.7)\n",
      "Collecting protobuf<6.0dev,>=5.26.1 (from grpcio-tools>=1.41.0->qdrant-client==1.9.1)\n",
      "  Downloading protobuf-5.27.2-cp310-abi3-win_amd64.whl.metadata (592 bytes)\n",
      "Requirement already satisfied: setuptools in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from grpcio-tools>=1.41.0->qdrant-client==1.9.1) (68.2.2)\n",
      "Requirement already satisfied: anyio in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (4.2.0)\n",
      "Requirement already satisfied: certifi in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (1.0.5)\n",
      "Requirement already satisfied: idna in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (3.4)\n",
      "Requirement already satisfied: sniffio in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (1.3.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpcore==1.*->httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (0.14.0)\n",
      "Collecting h2<5,>=3 (from httpx[http2]>=0.20.0->qdrant-client==1.9.1)\n",
      "  Downloading h2-4.1.0-py3-none-any.whl.metadata (3.6 kB)\n",
      "Requirement already satisfied: pywin32>=226 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from portalocker<3.0.0,>=2.7.0->qdrant-client==1.9.1) (305.1)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pydantic>=1.10.8->qdrant-client==1.9.1) (4.9.0)\n",
      "Collecting hyperframe<7,>=6.0 (from h2<5,>=3->httpx[http2]>=0.20.0->qdrant-client==1.9.1)\n",
      "  Downloading hyperframe-6.0.1-py3-none-any.whl.metadata (2.7 kB)\n",
      "Collecting hpack<5,>=4.0 (from h2<5,>=3->httpx[http2]>=0.20.0->qdrant-client==1.9.1)\n",
      "  Downloading hpack-4.0.0-py3-none-any.whl.metadata (2.5 kB)\n",
      "Downloading qdrant_client-1.9.1-py3-none-any.whl (229 kB)\n",
      "Downloading grpcio-1.64.1-cp311-cp311-win_amd64.whl (4.1 MB)\n",
      "Downloading grpcio_tools-1.64.1-cp311-cp311-win_amd64.whl (1.1 MB)\n",
      "Downloading portalocker-2.10.0-py3-none-any.whl (18 kB)\n",
      "Downloading h2-4.1.0-py3-none-any.whl (57 kB)\n",
      "Downloading protobuf-5.27.2-cp310-abi3-win_amd64.whl (426 kB)\n",
      "Downloading hpack-4.0.0-py3-none-any.whl (32 kB)\n",
      "Downloading hyperframe-6.0.1-py3-none-any.whl (12 kB)\n",
      "Installing collected packages: protobuf, portalocker, hyperframe, hpack, grpcio, h2, grpcio-tools, qdrant-client\n",
      "  Attempting uninstall: protobuf\n",
      "    Found existing installation: protobuf 3.20.3\n",
      "    Uninstalling protobuf-3.20.3:\n",
      "      Successfully uninstalled protobuf-3.20.3\n",
      "Successfully installed grpcio-1.64.1 grpcio-tools-1.64.1 h2-4.1.0 hpack-4.0.0 hyperframe-6.0.1 portalocker-2.10.0 protobuf-5.27.2 qdrant-client-1.9.1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "streamlit 1.30.0 requires protobuf<5,>=3.20, but you have protobuf 5.27.2 which is incompatible.\n"
     ]
    }
   ],
   "source": [
    "!pip install qdrant-client==1.9.1 --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "401e7244",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting unstructured==0.13.6 (from unstructured[md]==0.13.6)\n",
      "  Downloading unstructured-0.13.6-py3-none-any.whl.metadata (30 kB)\n",
      "Requirement already satisfied: chardet in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.0.0)\n",
      "Requirement already satisfied: filetype in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (1.2.0)\n",
      "Requirement already satisfied: python-magic in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (0.4.27)\n",
      "Requirement already satisfied: lxml in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.9.3)\n",
      "Requirement already satisfied: nltk in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (3.8.1)\n",
      "Requirement already satisfied: tabulate in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (0.9.0)\n",
      "Requirement already satisfied: requests in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (2.31.0)\n",
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.12.2)\n",
      "Requirement already satisfied: emoji in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (2.11.1)\n",
      "Requirement already satisfied: dataclasses-json in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (0.6.6)\n",
      "Requirement already satisfied: python-iso639 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (2024.4.27)\n",
      "Requirement already satisfied: langdetect in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (1.0.9)\n",
      "Requirement already satisfied: numpy in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (1.26.4)\n",
      "Requirement already satisfied: rapidfuzz in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (3.9.0)\n",
      "Requirement already satisfied: backoff in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (2.2.1)\n",
      "Requirement already satisfied: typing-extensions in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.9.0)\n",
      "Requirement already satisfied: unstructured-client in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (0.22.0)\n",
      "Requirement already satisfied: wrapt in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (1.14.1)\n",
      "Requirement already satisfied: markdown in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured[md]==0.13.6) (3.4.1)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from beautifulsoup4->unstructured==0.13.6->unstructured[md]==0.13.6) (2.5)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from dataclasses-json->unstructured==0.13.6->unstructured[md]==0.13.6) (3.21.2)\n",
      "Requirement already satisfied: typing-inspect<1,>=0.4.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from dataclasses-json->unstructured==0.13.6->unstructured[md]==0.13.6) (0.9.0)\n",
      "Requirement already satisfied: six in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langdetect->unstructured==0.13.6->unstructured[md]==0.13.6) (1.16.0)\n",
      "Requirement already satisfied: click in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (8.1.7)\n",
      "Requirement already satisfied: joblib in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (1.2.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (2023.10.3)\n",
      "Requirement already satisfied: tqdm in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (4.66.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->unstructured==0.13.6->unstructured[md]==0.13.6) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->unstructured==0.13.6->unstructured[md]==0.13.6) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->unstructured==0.13.6->unstructured[md]==0.13.6) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->unstructured==0.13.6->unstructured[md]==0.13.6) (2024.2.2)\n",
      "Requirement already satisfied: deepdiff>=6.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (7.0.1)\n",
      "Requirement already satisfied: jsonpath-python>=1.0.6 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (1.0.6)\n",
      "Requirement already satisfied: mypy-extensions>=1.0.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (1.0.0)\n",
      "Requirement already satisfied: packaging>=23.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (23.2)\n",
      "Requirement already satisfied: pypdf>=4.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (4.2.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (2.8.2)\n",
      "Requirement already satisfied: ordered-set<4.2.0,>=4.1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from deepdiff>=6.0->unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (4.1.0)\n",
      "Requirement already satisfied: colorama in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from click->nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (0.4.6)\n",
      "Downloading unstructured-0.13.6-py3-none-any.whl (1.9 MB)\n",
      "Installing collected packages: unstructured\n",
      "  Attempting uninstall: unstructured\n",
      "    Found existing installation: unstructured 0.13.7\n",
      "    Uninstalling unstructured-0.13.7:\n",
      "      Successfully uninstalled unstructured-0.13.7\n",
      "Successfully installed unstructured-0.13.6\n"
     ]
    }
   ],
   "source": [
    "!pip install \"unstructured[md]\"==0.13.6 --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "899d789a",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fastembed==0.2.7\n",
      "  Downloading fastembed-0.2.7-py3-none-any.whl.metadata (5.3 kB)\n",
      "Collecting huggingface-hub<0.21,>=0.20 (from fastembed==0.2.7)\n",
      "  Using cached huggingface_hub-0.20.3-py3-none-any.whl.metadata (12 kB)\n",
      "Collecting loguru<0.8.0,>=0.7.2 (from fastembed==0.2.7)\n",
      "  Downloading loguru-0.7.2-py3-none-any.whl.metadata (23 kB)\n",
      "Requirement already satisfied: numpy>=1.21 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (1.26.4)\n",
      "Collecting onnx<2.0.0,>=1.15.0 (from fastembed==0.2.7)\n",
      "  Downloading onnx-1.16.1-cp311-cp311-win_amd64.whl.metadata (16 kB)\n",
      "Collecting onnxruntime<2.0.0,>=1.17.0 (from fastembed==0.2.7)\n",
      "  Downloading onnxruntime-1.18.1-cp311-cp311-win_amd64.whl.metadata (4.5 kB)\n",
      "Requirement already satisfied: requests<3.0,>=2.31 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (2.31.0)\n",
      "Collecting tokenizers<0.16,>=0.15 (from fastembed==0.2.7)\n",
      "  Downloading tokenizers-0.15.2-cp311-none-win_amd64.whl.metadata (6.8 kB)\n",
      "Requirement already satisfied: tqdm<5.0,>=4.66 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (4.66.4)\n",
      "Requirement already satisfied: filelock in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (3.13.1)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (2023.10.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (6.0.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (4.9.0)\n",
      "Requirement already satisfied: packaging>=20.9 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (23.2)\n",
      "Requirement already satisfied: colorama>=0.3.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from loguru<0.8.0,>=0.7.2->fastembed==0.2.7) (0.4.6)\n",
      "Collecting win32-setctime>=1.0.0 (from loguru<0.8.0,>=0.7.2->fastembed==0.2.7)\n",
      "  Downloading win32_setctime-1.1.0-py3-none-any.whl.metadata (2.3 kB)\n",
      "Requirement already satisfied: protobuf>=3.20.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnx<2.0.0,>=1.15.0->fastembed==0.2.7) (5.27.2)\n",
      "Collecting coloredlogs (from onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7)\n",
      "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)\n",
      "Collecting flatbuffers (from onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7)\n",
      "  Downloading flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes)\n",
      "Requirement already satisfied: sympy in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (1.12)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3.0,>=2.31->fastembed==0.2.7) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3.0,>=2.31->fastembed==0.2.7) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3.0,>=2.31->fastembed==0.2.7) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests<3.0,>=2.31->fastembed==0.2.7) (2024.2.2)\n",
      "Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7)\n",
      "  Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)\n",
      "Requirement already satisfied: mpmath>=0.19 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from sympy->onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (1.3.0)\n",
      "Collecting pyreadline3 (from humanfriendly>=9.1->coloredlogs->onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7)\n",
      "  Downloading pyreadline3-3.4.1-py3-none-any.whl.metadata (2.0 kB)\n",
      "Downloading fastembed-0.2.7-py3-none-any.whl (27 kB)\n",
      "Using cached huggingface_hub-0.20.3-py3-none-any.whl (330 kB)\n",
      "Downloading loguru-0.7.2-py3-none-any.whl (62 kB)\n",
      "Downloading onnx-1.16.1-cp311-cp311-win_amd64.whl (14.4 MB)\n",
      "Downloading onnxruntime-1.18.1-cp311-cp311-win_amd64.whl (5.6 MB)\n",
      "Downloading tokenizers-0.15.2-cp311-none-win_amd64.whl (2.2 MB)\n",
      "Downloading win32_setctime-1.1.0-py3-none-any.whl (3.6 kB)\n",
      "Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
      "Downloading flatbuffers-24.3.25-py2.py3-none-any.whl (26 kB)\n",
      "Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
      "Downloading pyreadline3-3.4.1-py3-none-any.whl (95 kB)\n",
      "Installing collected packages: pyreadline3, flatbuffers, win32-setctime, onnx, humanfriendly, loguru, huggingface-hub, coloredlogs, tokenizers, onnxruntime, fastembed\n",
      "  Attempting uninstall: huggingface-hub\n",
      "    Found existing installation: huggingface-hub 0.23.0\n",
      "    Uninstalling huggingface-hub-0.23.0:\n",
      "      Successfully uninstalled huggingface-hub-0.23.0\n",
      "  Attempting uninstall: tokenizers\n",
      "    Found existing installation: tokenizers 0.19.1\n",
      "    Uninstalling tokenizers-0.19.1:\n",
      "      Successfully uninstalled tokenizers-0.19.1\n",
      "Successfully installed coloredlogs-15.0.1 fastembed-0.2.7 flatbuffers-24.3.25 huggingface-hub-0.20.3 humanfriendly-10.0 loguru-0.7.2 onnx-1.16.1 onnxruntime-1.18.1 pyreadline3-3.4.1 tokenizers-0.15.2 win32-setctime-1.1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  WARNING: Failed to remove contents in a temporary directory 'C:\\Users\\murugan\\anaconda3\\Lib\\site-packages\\~okenizers'.\n",
      "  You can safely remove it manually.\n",
      "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "auto-gptq 0.7.1 requires peft>=0.5.0, but you have peft 0.4.0 which is incompatible.\n",
      "transformers 4.40.2 requires tokenizers<0.20,>=0.19, but you have tokenizers 0.15.2 which is incompatible.\n"
     ]
    }
   ],
   "source": [
    "!pip install fastembed==0.2.7 --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a0b8b4d4",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting flashrank==0.2.4\n",
      "  Downloading FlashRank-0.2.4-py3-none-any.whl.metadata (10 kB)\n",
      "Requirement already satisfied: tokenizers in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from flashrank==0.2.4) (0.15.2)\n",
      "Requirement already satisfied: onnxruntime in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from flashrank==0.2.4) (1.18.1)\n",
      "Requirement already satisfied: numpy in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from flashrank==0.2.4) (1.26.4)\n",
      "Requirement already satisfied: requests in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from flashrank==0.2.4) (2.31.0)\n",
      "Requirement already satisfied: tqdm in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from flashrank==0.2.4) (4.66.4)\n",
      "Collecting llama-cpp-python==0.2.67 (from flashrank==0.2.4)\n",
      "  Downloading llama_cpp_python-0.2.67.tar.gz (42.4 MB)\n",
      "  Installing build dependencies: started\n",
      "  Installing build dependencies: finished with status 'done'\n",
      "  Getting requirements to build wheel: started\n",
      "  Getting requirements to build wheel: finished with status 'done'\n",
      "  Installing backend dependencies: started\n",
      "  Installing backend dependencies: finished with status 'done'\n",
      "  Preparing metadata (pyproject.toml): started\n",
      "  Preparing metadata (pyproject.toml): finished with status 'done'\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-cpp-python==0.2.67->flashrank==0.2.4) (4.9.0)\n",
      "Collecting diskcache>=5.6.1 (from llama-cpp-python==0.2.67->flashrank==0.2.4)\n",
      "  Using cached diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)\n",
      "Requirement already satisfied: jinja2>=2.11.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-cpp-python==0.2.67->flashrank==0.2.4) (3.1.3)\n",
      "Requirement already satisfied: coloredlogs in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime->flashrank==0.2.4) (15.0.1)\n",
      "Requirement already satisfied: flatbuffers in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime->flashrank==0.2.4) (24.3.25)\n",
      "Requirement already satisfied: packaging in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime->flashrank==0.2.4) (23.2)\n",
      "Requirement already satisfied: protobuf in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime->flashrank==0.2.4) (5.27.2)\n",
      "Requirement already satisfied: sympy in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime->flashrank==0.2.4) (1.12)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->flashrank==0.2.4) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->flashrank==0.2.4) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->flashrank==0.2.4) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->flashrank==0.2.4) (2024.2.2)\n",
      "Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from tokenizers->flashrank==0.2.4) (0.20.3)\n",
      "Requirement already satisfied: colorama in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from tqdm->flashrank==0.2.4) (0.4.6)\n",
      "Requirement already satisfied: filelock in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->flashrank==0.2.4) (3.13.1)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->flashrank==0.2.4) (2023.10.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->flashrank==0.2.4) (6.0.1)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jinja2>=2.11.3->llama-cpp-python==0.2.67->flashrank==0.2.4) (2.1.3)\n",
      "Requirement already satisfied: humanfriendly>=9.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from coloredlogs->onnxruntime->flashrank==0.2.4) (10.0)\n",
      "Requirement already satisfied: mpmath>=0.19 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from sympy->onnxruntime->flashrank==0.2.4) (1.3.0)\n",
      "Requirement already satisfied: pyreadline3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from humanfriendly>=9.1->coloredlogs->onnxruntime->flashrank==0.2.4) (3.4.1)\n",
      "Downloading FlashRank-0.2.4-py3-none-any.whl (17 kB)\n",
      "Using cached diskcache-5.6.3-py3-none-any.whl (45 kB)\n",
      "Building wheels for collected packages: llama-cpp-python\n",
      "  Building wheel for llama-cpp-python (pyproject.toml): started\n",
      "  Building wheel for llama-cpp-python (pyproject.toml): still running...\n",
      "  Building wheel for llama-cpp-python (pyproject.toml): still running...\n",
      "  Building wheel for llama-cpp-python (pyproject.toml): finished with status 'done'\n",
      "  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.2.67-cp311-cp311-win_amd64.whl size=3176529 sha256=4abc178b78b2a02dd2f2c73ec163c883acf2f39573d56910780147070528107c\n",
      "  Stored in directory: c:\\users\\murugan\\appdata\\local\\pip\\cache\\wheels\\e5\\b3\\c2\\380a2036bc16f0d0555e5c7041fdad2f109500f050c2e73618\n",
      "Successfully built llama-cpp-python\n",
      "Installing collected packages: diskcache, llama-cpp-python, flashrank\n",
      "Successfully installed diskcache-5.6.3 flashrank-0.2.4 llama-cpp-python-0.2.67\n"
     ]
    }
   ],
   "source": [
    "!pip install flashrank==0.2.4 --progress-bar off"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "31f740bf",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in c:\\users\\murugan\\appdata\\roaming\\python\\python311\\site-packages (24.1.2)\n",
      "Requirement already satisfied: pdfplumber in c:\\users\\murugan\\anaconda3\\lib\\site-packages (0.11.0)\n",
      "Requirement already satisfied: langchain-groq==0.1.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (0.1.3)\n",
      "Requirement already satisfied: langchain==0.1.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (0.1.17)\n",
      "Requirement already satisfied: llama-parse==0.1.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (0.1.3)\n",
      "Requirement already satisfied: qdrant-client==1.9.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (1.9.1)\n",
      "Requirement already satisfied: unstructured==0.13.6 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured[md]==0.13.6) (0.13.6)\n",
      "Requirement already satisfied: fastembed==0.2.7 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (0.2.7)\n",
      "Requirement already satisfied: flashrank==0.2.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (0.2.4)\n",
      "Requirement already satisfied: groq<1,>=0.4.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-groq==0.1.3) (0.5.0)\n",
      "Requirement already satisfied: langchain-core<0.2.0,>=0.1.45 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain-groq==0.1.3) (0.1.52)\n",
      "Requirement already satisfied: PyYAML>=5.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (6.0.1)\n",
      "Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (2.0.25)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (3.9.3)\n",
      "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.6.6)\n",
      "Requirement already satisfied: jsonpatch<2.0,>=1.33 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (1.33)\n",
      "Requirement already satisfied: langchain-community<0.1,>=0.0.36 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.0.38)\n",
      "Requirement already satisfied: langchain-text-splitters<0.1,>=0.0.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.0.1)\n",
      "Requirement already satisfied: langsmith<0.2.0,>=0.1.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (0.1.56)\n",
      "Requirement already satisfied: numpy<2,>=1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (1.26.4)\n",
      "Requirement already satisfied: pydantic<3,>=1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (1.10.12)\n",
      "Requirement already satisfied: requests<3,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langchain==0.1.17) (8.2.2)\n",
      "Requirement already satisfied: llama-index<0.10.0,>=0.9.40 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-parse==0.1.3) (0.9.48)\n",
      "Requirement already satisfied: grpcio>=1.41.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from qdrant-client==1.9.1) (1.64.1)\n",
      "Requirement already satisfied: grpcio-tools>=1.41.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from qdrant-client==1.9.1) (1.64.1)\n",
      "Requirement already satisfied: httpx>=0.20.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx[http2]>=0.20.0->qdrant-client==1.9.1) (0.27.0)\n",
      "Requirement already satisfied: portalocker<3.0.0,>=2.7.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from qdrant-client==1.9.1) (2.10.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.26.14 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from qdrant-client==1.9.1) (2.0.7)\n",
      "Requirement already satisfied: chardet in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.0.0)\n",
      "Requirement already satisfied: filetype in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (1.2.0)\n",
      "Requirement already satisfied: python-magic in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (0.4.27)\n",
      "Requirement already satisfied: lxml in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.9.3)\n",
      "Requirement already satisfied: nltk in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (3.8.1)\n",
      "Requirement already satisfied: tabulate in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (0.9.0)\n",
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.12.2)\n",
      "Requirement already satisfied: emoji in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (2.11.1)\n",
      "Requirement already satisfied: python-iso639 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (2024.4.27)\n",
      "Requirement already satisfied: langdetect in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (1.0.9)\n",
      "Requirement already satisfied: rapidfuzz in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (3.9.0)\n",
      "Requirement already satisfied: backoff in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (2.2.1)\n",
      "Requirement already satisfied: typing-extensions in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (4.9.0)\n",
      "Requirement already satisfied: unstructured-client in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (0.22.0)\n",
      "Requirement already satisfied: wrapt in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured==0.13.6->unstructured[md]==0.13.6) (1.14.1)\n",
      "Requirement already satisfied: huggingface-hub<0.21,>=0.20 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (0.20.3)\n",
      "Requirement already satisfied: loguru<0.8.0,>=0.7.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (0.7.2)\n",
      "Requirement already satisfied: onnx<2.0.0,>=1.15.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (1.16.1)\n",
      "Requirement already satisfied: onnxruntime<2.0.0,>=1.17.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (1.18.1)\n",
      "Requirement already satisfied: tokenizers<0.16,>=0.15 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (0.15.2)\n",
      "Requirement already satisfied: tqdm<5.0,>=4.66 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from fastembed==0.2.7) (4.66.4)\n",
      "Requirement already satisfied: llama-cpp-python==0.2.67 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from flashrank==0.2.4) (0.2.67)\n",
      "Requirement already satisfied: markdown in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured[md]==0.13.6) (3.4.1)\n",
      "Requirement already satisfied: diskcache>=5.6.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-cpp-python==0.2.67->flashrank==0.2.4) (5.6.3)\n",
      "Requirement already satisfied: jinja2>=2.11.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-cpp-python==0.2.67->flashrank==0.2.4) (3.1.3)\n",
      "Requirement already satisfied: pdfminer.six==20231228 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pdfplumber) (20231228)\n",
      "Requirement already satisfied: Pillow>=9.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pdfplumber) (10.2.0)\n",
      "Requirement already satisfied: pypdfium2>=4.18.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pdfplumber) (4.30.0)\n",
      "Requirement already satisfied: charset-normalizer>=2.0.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pdfminer.six==20231228->pdfplumber) (3.3.2)\n",
      "Requirement already satisfied: cryptography>=36.0.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pdfminer.six==20231228->pdfplumber) (42.0.2)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (1.2.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (1.4.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.1.17) (1.9.3)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.1.17) (3.21.2)\n",
      "Requirement already satisfied: typing-inspect<1,>=0.4.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.1.17) (0.9.0)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (4.2.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (1.8.0)\n",
      "Requirement already satisfied: sniffio in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from groq<1,>=0.4.1->langchain-groq==0.1.3) (1.3.0)\n",
      "Requirement already satisfied: protobuf<6.0dev,>=5.26.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from grpcio-tools>=1.41.0->qdrant-client==1.9.1) (5.27.2)\n",
      "Requirement already satisfied: setuptools in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from grpcio-tools>=1.41.0->qdrant-client==1.9.1) (68.2.2)\n",
      "Requirement already satisfied: certifi in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (1.0.5)\n",
      "Requirement already satisfied: idna in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (3.4)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpcore==1.*->httpx>=0.20.0->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (0.14.0)\n",
      "Requirement already satisfied: h2<5,>=3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx[http2]>=0.20.0->qdrant-client==1.9.1) (4.1.0)\n",
      "Requirement already satisfied: filelock in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (3.13.1)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (2023.10.0)\n",
      "Requirement already satisfied: packaging>=20.9 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<0.21,>=0.20->fastembed==0.2.7) (23.2)\n",
      "Requirement already satisfied: jsonpointer>=1.9 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jsonpatch<2.0,>=1.33->langchain==0.1.17) (2.1)\n",
      "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langsmith<0.2.0,>=0.1.17->langchain==0.1.17) (3.10.3)\n",
      "Requirement already satisfied: deprecated>=1.2.9.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.2.14)\n",
      "Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.0.8)\n",
      "Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.6.0)\n",
      "Requirement already satisfied: networkx>=3.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (3.1)\n",
      "Requirement already satisfied: openai>=1.1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (1.35.13)\n",
      "Requirement already satisfied: pandas in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2.1.4)\n",
      "Requirement already satisfied: tiktoken>=0.3.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (0.6.0)\n",
      "Requirement already satisfied: colorama>=0.3.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from loguru<0.8.0,>=0.7.2->fastembed==0.2.7) (0.4.6)\n",
      "Requirement already satisfied: win32-setctime>=1.0.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from loguru<0.8.0,>=0.7.2->fastembed==0.2.7) (1.1.0)\n",
      "Requirement already satisfied: click in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (8.1.7)\n",
      "Requirement already satisfied: joblib in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (1.2.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from nltk->unstructured==0.13.6->unstructured[md]==0.13.6) (2023.10.3)\n",
      "Requirement already satisfied: coloredlogs in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (15.0.1)\n",
      "Requirement already satisfied: flatbuffers in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (24.3.25)\n",
      "Requirement already satisfied: sympy in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (1.12)\n",
      "Requirement already satisfied: pywin32>=226 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from portalocker<3.0.0,>=2.7.0->qdrant-client==1.9.1) (305.1)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from SQLAlchemy<3,>=1.4->langchain==0.1.17) (3.0.1)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from beautifulsoup4->unstructured==0.13.6->unstructured[md]==0.13.6) (2.5)\n",
      "Requirement already satisfied: six in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from langdetect->unstructured==0.13.6->unstructured[md]==0.13.6) (1.16.0)\n",
      "Requirement already satisfied: deepdiff>=6.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (7.0.1)\n",
      "Requirement already satisfied: jsonpath-python>=1.0.6 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (1.0.6)\n",
      "Requirement already satisfied: mypy-extensions>=1.0.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (1.0.0)\n",
      "Requirement already satisfied: pypdf>=4.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (4.2.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (2.8.2)\n",
      "Requirement already satisfied: cffi>=1.12 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from cryptography>=36.0.0->pdfminer.six==20231228->pdfplumber) (1.16.0)\n",
      "Requirement already satisfied: ordered-set<4.2.0,>=4.1.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from deepdiff>=6.0->unstructured-client->unstructured==0.13.6->unstructured[md]==0.13.6) (4.1.0)\n",
      "Requirement already satisfied: hyperframe<7,>=6.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from h2<5,>=3->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (6.0.1)\n",
      "Requirement already satisfied: hpack<5,>=4.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from h2<5,>=3->httpx[http2]>=0.20.0->qdrant-client==1.9.1) (4.0.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jinja2>=2.11.3->llama-cpp-python==0.2.67->flashrank==0.2.4) (2.1.3)\n",
      "Requirement already satisfied: humanfriendly>=9.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from coloredlogs->onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (10.0)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->llama-index<0.10.0,>=0.9.40->llama-parse==0.1.3) (2023.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from sympy->onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (1.3.0)\n",
      "Requirement already satisfied: pycparser in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six==20231228->pdfplumber) (2.21)\n",
      "Requirement already satisfied: pyreadline3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from humanfriendly>=9.1->coloredlogs->onnxruntime<2.0.0,>=1.17.0->fastembed==0.2.7) (3.4.1)\n"
     ]
    }
   ],
   "source": [
    "!pip install pip --upgrade --progress-bar off\n",
    "!pip install pdfplumber langchain-groq==0.1.3 langchain==0.1.17 llama-parse==0.1.3 qdrant-client==1.9.1 unstructured[md]==0.13.6 fastembed==0.2.7 flashrank==0.2.4 --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "aa8b0e96",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Could not find a version that satisfies the requirement retrievalqa (from versions: none)\n",
      "ERROR: No matching distribution found for retrievalqa\n"
     ]
    }
   ],
   "source": [
    "!pip install retrievalqa --progress-bar off\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8a20120a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting farm-haystack\n",
      "  Downloading farm_haystack-1.26.2-py3-none-any.whl.metadata (31 kB)\n",
      "Collecting boilerpy3 (from farm-haystack)\n",
      "  Downloading boilerpy3-1.0.7-py3-none-any.whl.metadata (5.8 kB)\n",
      "Collecting events (from farm-haystack)\n",
      "  Downloading Events-0.5-py3-none-any.whl.metadata (3.9 kB)\n",
      "Requirement already satisfied: httpx in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (0.27.0)\n",
      "Requirement already satisfied: jsonschema in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (4.19.2)\n",
      "Collecting lazy-imports==0.3.1 (from farm-haystack)\n",
      "  Downloading lazy_imports-0.3.1-py3-none-any.whl.metadata (10 kB)\n",
      "Requirement already satisfied: more-itertools in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (10.1.0)\n",
      "Requirement already satisfied: networkx in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (3.1)\n",
      "Requirement already satisfied: pandas in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (2.1.4)\n",
      "Requirement already satisfied: pillow in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (10.2.0)\n",
      "Requirement already satisfied: platformdirs in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (3.10.0)\n",
      "Collecting posthog (from farm-haystack)\n",
      "  Downloading posthog-3.5.0-py2.py3-none-any.whl.metadata (2.0 kB)\n",
      "Collecting prompthub-py==4.0.0 (from farm-haystack)\n",
      "  Downloading prompthub_py-4.0.0-py3-none-any.whl.metadata (2.2 kB)\n",
      "Requirement already satisfied: pydantic<2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (1.10.12)\n",
      "Collecting quantulum3 (from farm-haystack)\n",
      "  Downloading quantulum3-0.9.2-py3-none-any.whl.metadata (16 kB)\n",
      "Collecting rank-bm25 (from farm-haystack)\n",
      "  Downloading rank_bm25-0.2.2-py3-none-any.whl.metadata (3.2 kB)\n",
      "Requirement already satisfied: requests in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (2.31.0)\n",
      "Collecting requests-cache<1.0.0 (from farm-haystack)\n",
      "  Downloading requests_cache-0.9.8-py3-none-any.whl.metadata (8.7 kB)\n",
      "Collecting scikit-learn>=1.3.0 (from farm-haystack)\n",
      "  Downloading scikit_learn-1.5.1-cp311-cp311-win_amd64.whl.metadata (12 kB)\n",
      "Collecting sseclient-py (from farm-haystack)\n",
      "  Downloading sseclient_py-1.8.0-py2.py3-none-any.whl.metadata (2.0 kB)\n",
      "Requirement already satisfied: tenacity in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (8.2.2)\n",
      "Requirement already satisfied: tiktoken>=0.5.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (0.6.0)\n",
      "Requirement already satisfied: tqdm in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from farm-haystack) (4.66.4)\n",
      "Collecting transformers==4.39.3 (from farm-haystack)\n",
      "  Downloading transformers-4.39.3-py3-none-any.whl.metadata (134 kB)\n",
      "     ---------------------------------------- 0.0/134.8 kB ? eta -:--:--\n",
      "     -------------------------------------- 134.8/134.8 kB 4.0 MB/s eta 0:00:00\n",
      "Requirement already satisfied: pyyaml<7.0,>=6.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from prompthub-py==4.0.0->farm-haystack) (6.0.1)\n",
      "Requirement already satisfied: filelock in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers==4.39.3->farm-haystack) (3.13.1)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers==4.39.3->farm-haystack) (0.20.3)\n",
      "Requirement already satisfied: numpy>=1.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers==4.39.3->farm-haystack) (1.26.4)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers==4.39.3->farm-haystack) (23.2)\n",
      "Requirement already satisfied: regex!=2019.12.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers==4.39.3->farm-haystack) (2023.10.3)\n",
      "Requirement already satisfied: tokenizers<0.19,>=0.14 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers==4.39.3->farm-haystack) (0.15.2)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers==4.39.3->farm-haystack) (0.4.3)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pydantic<2->farm-haystack) (4.9.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->farm-haystack) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->farm-haystack) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->farm-haystack) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->farm-haystack) (2024.2.2)\n",
      "Requirement already satisfied: appdirs>=1.4.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests-cache<1.0.0->farm-haystack) (1.4.4)\n",
      "Requirement already satisfied: attrs>=21.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests-cache<1.0.0->farm-haystack) (23.2.0)\n",
      "Collecting cattrs>=22.2 (from requests-cache<1.0.0->farm-haystack)\n",
      "  Downloading cattrs-23.2.3-py3-none-any.whl.metadata (10 kB)\n",
      "Collecting url-normalize>=1.4 (from requests-cache<1.0.0->farm-haystack)\n",
      "  Downloading url_normalize-1.4.3-py2.py3-none-any.whl.metadata (3.1 kB)\n",
      "Requirement already satisfied: scipy>=1.6.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from scikit-learn>=1.3.0->farm-haystack) (1.11.4)\n",
      "Requirement already satisfied: joblib>=1.2.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from scikit-learn>=1.3.0->farm-haystack) (1.2.0)\n",
      "Collecting threadpoolctl>=3.1.0 (from scikit-learn>=1.3.0->farm-haystack)\n",
      "  Downloading threadpoolctl-3.5.0-py3-none-any.whl.metadata (13 kB)\n",
      "Requirement already satisfied: colorama in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from tqdm->farm-haystack) (0.4.6)\n",
      "Requirement already satisfied: anyio in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx->farm-haystack) (4.2.0)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx->farm-haystack) (1.0.5)\n",
      "Requirement already satisfied: sniffio in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpx->farm-haystack) (1.3.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from httpcore==1.*->httpx->farm-haystack) (0.14.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jsonschema->farm-haystack) (2023.7.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jsonschema->farm-haystack) (0.30.2)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from jsonschema->farm-haystack) (0.10.6)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->farm-haystack) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->farm-haystack) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from pandas->farm-haystack) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from posthog->farm-haystack) (1.16.0)\n",
      "Collecting monotonic>=1.5 (from posthog->farm-haystack)\n",
      "  Downloading monotonic-1.6-py2.py3-none-any.whl.metadata (1.5 kB)\n",
      "Requirement already satisfied: backoff>=1.10.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from posthog->farm-haystack) (2.2.1)\n",
      "Collecting inflect (from quantulum3->farm-haystack)\n",
      "  Downloading inflect-7.3.1-py3-none-any.whl.metadata (21 kB)\n",
      "Collecting num2words (from quantulum3->farm-haystack)\n",
      "  Downloading num2words-0.5.13-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.39.3->farm-haystack) (2023.10.0)\n",
      "Collecting typeguard>=4.0.1 (from inflect->quantulum3->farm-haystack)\n",
      "  Downloading typeguard-4.3.0-py3-none-any.whl.metadata (3.7 kB)\n",
      "Collecting docopt>=0.6.2 (from num2words->quantulum3->farm-haystack)\n",
      "  Using cached docopt-0.6.2.tar.gz (25 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting typing-extensions>=4.2.0 (from pydantic<2->farm-haystack)\n",
      "  Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)\n",
      "Downloading farm_haystack-1.26.2-py3-none-any.whl (763 kB)\n",
      "   ---------------------------------------- 0.0/763.7 kB ? eta -:--:--\n",
      "   ------------------------ --------------- 471.0/763.7 kB 9.8 MB/s eta 0:00:01\n",
      "   --------------------------------------- 763.7/763.7 kB 12.2 MB/s eta 0:00:00\n",
      "Downloading lazy_imports-0.3.1-py3-none-any.whl (12 kB)\n",
      "Downloading prompthub_py-4.0.0-py3-none-any.whl (6.9 kB)\n",
      "Downloading transformers-4.39.3-py3-none-any.whl (8.8 MB)\n",
      "   ---------------------------------------- 0.0/8.8 MB ? eta -:--:--\n",
      "   -- ------------------------------------- 0.6/8.8 MB 12.0 MB/s eta 0:00:01\n",
      "   ----- ---------------------------------- 1.1/8.8 MB 12.1 MB/s eta 0:00:01\n",
      "   ------- -------------------------------- 1.7/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   ---------- ----------------------------- 2.2/8.8 MB 13.0 MB/s eta 0:00:01\n",
      "   ------------ --------------------------- 2.8/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   --------------- ------------------------ 3.4/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   ----------------- ---------------------- 3.9/8.8 MB 12.0 MB/s eta 0:00:01\n",
      "   -------------------- ------------------- 4.5/8.8 MB 12.0 MB/s eta 0:00:01\n",
      "   ---------------------- ----------------- 5.1/8.8 MB 12.0 MB/s eta 0:00:01\n",
      "   ------------------------- -------------- 5.6/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   --------------------------- ------------ 6.2/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   ------------------------------ --------- 6.7/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   --------------------------------- ------ 7.3/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   ----------------------------------- ---- 7.8/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   -------------------------------------- - 8.4/8.8 MB 11.9 MB/s eta 0:00:01\n",
      "   ---------------------------------------  8.8/8.8 MB 12.0 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 8.8/8.8 MB 11.7 MB/s eta 0:00:00\n",
      "Downloading requests_cache-0.9.8-py3-none-any.whl (48 kB)\n",
      "   ---------------------------------------- 0.0/48.7 kB ? eta -:--:--\n",
      "   ---------------------------------------- 48.7/48.7 kB ? eta 0:00:00\n",
      "Downloading scikit_learn-1.5.1-cp311-cp311-win_amd64.whl (11.0 MB)\n",
      "   ---------------------------------------- 0.0/11.0 MB ? eta -:--:--\n",
      "   -- ------------------------------------- 0.6/11.0 MB 12.0 MB/s eta 0:00:01\n",
      "   ---- ----------------------------------- 1.1/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ------ --------------------------------- 1.7/11.0 MB 11.8 MB/s eta 0:00:01\n",
      "   -------- ------------------------------- 2.2/11.0 MB 11.8 MB/s eta 0:00:01\n",
      "   ---------- ----------------------------- 2.8/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ------------ --------------------------- 3.3/11.0 MB 11.8 MB/s eta 0:00:01\n",
      "   -------------- ------------------------- 3.9/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ---------------- ----------------------- 4.5/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ------------------ --------------------- 5.0/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   -------------------- ------------------- 5.6/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ---------------------- ----------------- 6.1/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ------------------------ --------------- 6.7/11.0 MB 11.8 MB/s eta 0:00:01\n",
      "   -------------------------- ------------- 7.2/11.0 MB 11.8 MB/s eta 0:00:01\n",
      "   ---------------------------- ----------- 7.8/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ------------------------------ --------- 8.3/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   -------------------------------- ------- 8.9/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ---------------------------------- ----- 9.5/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ------------------------------------ --- 10.0/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   -------------------------------------- - 10.5/11.0 MB 11.7 MB/s eta 0:00:01\n",
      "   ---------------------------------------  11.0/11.0 MB 11.9 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 11.0/11.0 MB 11.7 MB/s eta 0:00:00\n",
      "Downloading boilerpy3-1.0.7-py3-none-any.whl (22 kB)\n",
      "Downloading Events-0.5-py3-none-any.whl (6.8 kB)\n",
      "Downloading posthog-3.5.0-py2.py3-none-any.whl (41 kB)\n",
      "   ---------------------------------------- 0.0/41.3 kB ? eta -:--:--\n",
      "   ---------------------------------------- 41.3/41.3 kB ? eta 0:00:00\n",
      "Downloading quantulum3-0.9.2-py3-none-any.whl (10.6 MB)\n",
      "   ---------------------------------------- 0.0/10.6 MB ? eta -:--:--\n",
      "   - -------------------------------------- 0.5/10.6 MB 9.4 MB/s eta 0:00:02\n",
      "   --- ------------------------------------ 1.0/10.6 MB 10.5 MB/s eta 0:00:01\n",
      "   ----- ---------------------------------- 1.5/10.6 MB 11.0 MB/s eta 0:00:01\n",
      "   ------- -------------------------------- 2.1/10.6 MB 11.2 MB/s eta 0:00:01\n",
      "   ---------- ----------------------------- 2.7/10.6 MB 11.3 MB/s eta 0:00:01\n",
      "   ------------ --------------------------- 3.2/10.6 MB 11.4 MB/s eta 0:00:01\n",
      "   -------------- ------------------------- 3.8/10.6 MB 11.5 MB/s eta 0:00:01\n",
      "   ---------------- ----------------------- 4.3/10.6 MB 11.5 MB/s eta 0:00:01\n",
      "   ------------------ --------------------- 4.9/10.6 MB 11.5 MB/s eta 0:00:01\n",
      "   -------------------- ------------------- 5.4/10.6 MB 11.6 MB/s eta 0:00:01\n",
      "   ---------------------- ----------------- 6.0/10.6 MB 11.6 MB/s eta 0:00:01\n",
      "   ------------------------ --------------- 6.5/10.6 MB 11.6 MB/s eta 0:00:01\n",
      "   -------------------------- ------------- 7.1/10.6 MB 11.9 MB/s eta 0:00:01\n",
      "   ---------------------------- ----------- 7.6/10.6 MB 11.9 MB/s eta 0:00:01\n",
      "   ------------------------------ --------- 8.2/10.6 MB 11.6 MB/s eta 0:00:01\n",
      "   -------------------------------- ------- 8.7/10.6 MB 11.6 MB/s eta 0:00:01\n",
      "   ----------------------------------- ---- 9.3/10.6 MB 11.7 MB/s eta 0:00:01\n",
      "   ------------------------------------- -- 9.9/10.6 MB 11.7 MB/s eta 0:00:01\n",
      "   ---------------------------------------  10.4/10.6 MB 11.7 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 10.6/10.6 MB 11.7 MB/s eta 0:00:00\n",
      "Downloading rank_bm25-0.2.2-py3-none-any.whl (8.6 kB)\n",
      "Downloading sseclient_py-1.8.0-py2.py3-none-any.whl (8.8 kB)\n",
      "Downloading cattrs-23.2.3-py3-none-any.whl (57 kB)\n",
      "   ---------------------------------------- 0.0/57.5 kB ? eta -:--:--\n",
      "   ---------------------------------------- 57.5/57.5 kB ? eta 0:00:00\n",
      "Downloading monotonic-1.6-py2.py3-none-any.whl (8.2 kB)\n",
      "Downloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)\n",
      "Downloading url_normalize-1.4.3-py2.py3-none-any.whl (6.8 kB)\n",
      "Downloading inflect-7.3.1-py3-none-any.whl (34 kB)\n",
      "Downloading num2words-0.5.13-py3-none-any.whl (143 kB)\n",
      "   ---------------------------------------- 0.0/143.3 kB ? eta -:--:--\n",
      "   ---------------------------------------- 143.3/143.3 kB 8.3 MB/s eta 0:00:00\n",
      "Downloading typeguard-4.3.0-py3-none-any.whl (35 kB)\n",
      "Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)\n",
      "Building wheels for collected packages: docopt\n",
      "  Building wheel for docopt (setup.py): started\n",
      "  Building wheel for docopt (setup.py): finished with status 'done'\n",
      "  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13773 sha256=ed8d1350214a10f10b08ad8428ada83270a6733aab461bb94b1ee76f43db2b41\n",
      "  Stored in directory: c:\\users\\murugan\\appdata\\local\\pip\\cache\\wheels\\1a\\b0\\8c\\4b75c4116c31f83c8f9f047231251e13cc74481cca4a78a9ce\n",
      "Successfully built docopt\n",
      "Installing collected packages: sseclient-py, monotonic, events, docopt, url-normalize, typing-extensions, threadpoolctl, rank-bm25, num2words, lazy-imports, cattrs, boilerpy3, typeguard, scikit-learn, requests-cache, prompthub-py, posthog, inflect, transformers, quantulum3, farm-haystack\n",
      "  Attempting uninstall: typing-extensions\n",
      "    Found existing installation: typing_extensions 4.9.0\n",
      "    Uninstalling typing_extensions-4.9.0:\n",
      "      Successfully uninstalled typing_extensions-4.9.0\n",
      "  Attempting uninstall: threadpoolctl\n",
      "    Found existing installation: threadpoolctl 2.2.0\n",
      "    Uninstalling threadpoolctl-2.2.0:\n",
      "      Successfully uninstalled threadpoolctl-2.2.0\n",
      "  Attempting uninstall: scikit-learn\n",
      "    Found existing installation: scikit-learn 1.2.2\n",
      "    Uninstalling scikit-learn-1.2.2:\n",
      "      Successfully uninstalled scikit-learn-1.2.2\n",
      "  Attempting uninstall: transformers\n",
      "    Found existing installation: transformers 4.40.2\n",
      "    Uninstalling transformers-4.40.2:\n",
      "      Successfully uninstalled transformers-4.40.2\n",
      "Successfully installed boilerpy3-1.0.7 cattrs-23.2.3 docopt-0.6.2 events-0.5 farm-haystack-1.26.2 inflect-7.3.1 lazy-imports-0.3.1 monotonic-1.6 num2words-0.5.13 posthog-3.5.0 prompthub-py-4.0.0 quantulum3-0.9.2 rank-bm25-0.2.2 requests-cache-0.9.8 scikit-learn-1.5.1 sseclient-py-1.8.0 threadpoolctl-3.5.0 transformers-4.39.3 typeguard-4.3.0 typing-extensions-4.12.2 url-normalize-1.4.3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "auto-gptq 0.7.1 requires peft>=0.5.0, but you have peft 0.4.0 which is incompatible.\n",
      "streamlit 1.30.0 requires protobuf<5,>=3.20, but you have protobuf 5.27.2 which is incompatible.\n"
     ]
    }
   ],
   "source": [
    "!pip install farm-haystack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7e3e567e",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: transformers in c:\\users\\murugan\\anaconda3\\lib\\site-packages (4.39.3)\n",
      "Requirement already satisfied: filelock in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (3.13.1)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (0.20.3)\n",
      "Requirement already satisfied: numpy>=1.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (1.26.4)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (2023.10.3)\n",
      "Requirement already satisfied: requests in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (2.31.0)\n",
      "Requirement already satisfied: tokenizers<0.19,>=0.14 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (0.15.2)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (0.4.3)\n",
      "Requirement already satisfied: tqdm>=4.27 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from transformers) (4.66.4)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (2023.10.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (4.12.2)\n",
      "Requirement already satisfied: colorama in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from tqdm>=4.27->transformers) (0.4.6)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->transformers) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->transformers) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->transformers) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\murugan\\anaconda3\\lib\\site-packages (from requests->transformers) (2024.2.2)\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb4b40d7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5db4b16d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a0cbbef",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18afdacc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06982156",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba84591c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31dae501",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47d40990",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbeecc54",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afae99f7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b393a5cb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e34d6ef5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fc26c4d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aec2b75d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_mistralai.chat_models import ChatMistralAI\n",
    "from langchain_mistralai.embeddings import MistralAIEmbeddings\n",
    "from langchain_community.vectorstores import FAISS\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain.chains import create_retrieval_chain\n",
    "\n",
    "# Load data\n",
    "loader = TextLoader(\"essay.txt\")\n",
    "docs = loader.load()\n",
    "# Split text into chunks \n",
    "text_splitter = RecursiveCharacterTextSplitter()\n",
    "documents = text_splitter.split_documents(docs)\n",
    "# Define the embedding model\n",
    "embeddings = MistralAIEmbeddings(model=\"mistral-embed\", mistral_api_key=api_key)\n",
    "# Create the vector store \n",
    "vector = FAISS.from_documents(documents, embeddings)\n",
    "# Define a retriever interface\n",
    "retriever = vector.as_retriever()\n",
    "# Define LLM\n",
    "model = ChatMistralAI(mistral_api_key=api_key)\n",
    "# Define prompt template\n",
    "prompt = ChatPromptTemplate.from_template(\"\"\"Answer the following question based only on the provided context:\n",
    "\n",
    "<context>\n",
    "{context}\n",
    "</context>\n",
    "\n",
    "Question: {input}\"\"\")\n",
    "\n",
    "# Create a retrieval chain to answer questions\n",
    "document_chain = create_stuff_documents_chain(model, prompt)\n",
    "retrieval_chain = create_retrieval_chain(retriever, document_chain)\n",
    "response = retrieval_chain.invoke({\"input\": \"What were the two main things the author worked on before college?\"})\n",
    "print(response[\"answer\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58b022fa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9bbb840",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a38d2681",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cd6aba2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f5fd4c9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e0e546f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7581a0a5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b727469d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e13997f7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ba3cc0d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6baf18b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You: send me the name of the candidate?\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import TextLoader\n",
    "from langchain_community.vectorstores import FAISS\n",
    "from langchain_community.embeddings import OllamaEmbeddings\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_community.llms import Ollama\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain.chains.conversation.memory import ConversationSummaryBufferMemory\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "\n",
    "# Load PDF document\n",
    "loader = PyPDFLoader(\"C:/Users/murugan/Downloads/Muthukumar.pdf\")\n",
    "pages_content = loader.load_and_split()\n",
    "\n",
    "# Define the text splitter with chunk size and overlap\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)\n",
    "\n",
    "# Split the documents into chunks\n",
    "documents = text_splitter.split_documents(pages_content)\n",
    "\n",
    "# Initialize the embeddings model\n",
    "ollama_emb = OllamaEmbeddings(model=\"llama3\")\n",
    "\n",
    "# Create the FAISS vector store\n",
    "vector = FAISS.from_documents(documents, ollama_emb)\n",
    "\n",
    "# Create the retriever\n",
    "retriever = vector.as_retriever()\n",
    "\n",
    "# Initialize the LLM\n",
    "llm = Ollama(model=\"llama3\")\n",
    "\n",
    "# Define the system prompt\n",
    "system_prompt = (\n",
    "    \"You are an assistant for question-answering tasks. \"\n",
    "    \"Use the following pieces of retrieved context to answer \"\n",
    "    \"the question. If you don't know the answer, say that you \"\n",
    "    \"don't know. Only send me the default answer.\"\n",
    "    \"answer concise.\"\n",
    "    \"\\n\\n\"\n",
    "    \"{context}\"\n",
    "    \"\\n\\n\"\n",
    "    \"Previous conversation:\"\n",
    "    \"{chat_history}\"\n",
    "    \"\\n\\n\"\n",
    "    \"Human: {input}\"\n",
    "    \"Assistant: \"\n",
    ")\n",
    "\n",
    "# Define the prompt template\n",
    "prompt = ChatPromptTemplate.from_template(system_prompt)\n",
    "\n",
    "# Initialize the memory\n",
    "memory = ConversationSummaryBufferMemory(\n",
    "    llm=llm,\n",
    "    max_token_limit=500,\n",
    "    memory_key=\"chat_history\",\n",
    "    return_messages=True\n",
    ")\n",
    "\n",
    "# Create the document chain\n",
    "document_chain = create_stuff_documents_chain(llm, prompt)\n",
    "\n",
    "# Create the retrieval chain\n",
    "retrieval_chain = create_retrieval_chain(retriever, document_chain)\n",
    "\n",
    "# Function to get response\n",
    "def get_response(question):\n",
    "    # Get chat history\n",
    "    chat_history = memory.load_memory_variables({})[\"chat_history\"]\n",
    "    \n",
    "    # Run the chain\n",
    "    response = retrieval_chain.invoke({\n",
    "        \"input\": question,\n",
    "        \"chat_history\": chat_history\n",
    "    })\n",
    "    \n",
    "    # Save the interaction to memory\n",
    "    memory.save_context({\"input\": question}, {\"output\": response[\"answer\"]})\n",
    "    \n",
    "    return response[\"answer\"]\n",
    "\n",
    "# Main interaction loop\n",
    "while True:\n",
    "    user_input = input(\"You: \")\n",
    "    if user_input.lower() in ['exit', 'quit', 'bye']:\n",
    "        break\n",
    "    response = get_response(user_input)\n",
    "    print(\"Assistant:\", response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "450e8796",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f305f24",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4ec3d06",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "901d1398",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60f41554",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "070b0bc6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c60fbb3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cb611c1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b55bcb58",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "029ec016",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51671b02",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa2af2b4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "42a9d8d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I don't know the answer to this question because it was not provided in the given context. The text only describes the responsibilities, roles, and experiences of the Principal Technical Architect, but does not mention their name.\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = conversational_chain({\"question\": \"what is the name of the candidate?\"})\n",
    "response['answer']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b3404437",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I don't know the answer to that question, as it's not provided in the context you gave me!\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = conversational_chain({\"question\": \"only name for previous questions\"})\n",
    "response['answer']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8766cd64",
   "metadata": {},
   "outputs": [],
   "source": [
    "ConversationalRetrievalChain.from_llm(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "80d8dda1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Current Memory Summary:\n",
      "[]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from langchain.memory import ConversationSummaryBufferMemory\n",
    "\n",
    "\n",
    "# Initialize the memory with summary capabilities\n",
    "memory = ConversationSummaryBufferMemory(\n",
    "    memory_key=\"chat_history\",\n",
    "    return_messages=True,\n",
    "    max_token_limit=100,  # Adjust this value as needed\n",
    "    llm=llm  # Use the same language model for summarization\n",
    ")\n",
    "\n",
    "# ... (rest of the code remains the same)\n",
    "\n",
    "# In the main loop, replace the memory output section with:\n",
    "print(\"\\nCurrent Memory Summary:\")\n",
    "print(memory.load_memory_variables({})[\"chat_history\"])\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d03d149",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}