gensphere_tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOzT1gQvQtq0Bj1tximUpMC",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/octopus2023-inc/gensphere/blob/main/gensphere_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **GenSphere tutorial**"
      ],
      "metadata": {
        "id": "TlQGkk0vfyoB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This quick tutorial will walk you through the main functionalities of [GenSphere](https://github.com/octopus2023-inc/gensphere).\n",
        "\n",
        "We will follow a guided example, **where we create a workflow that finds what are the latest product releases at [producthunt.com](http//producthunt.com), searches for traction information like revenue, number of users, and analyzes a new startup idea based on that.**\n",
        "\n",
        "By completing this tutorial, you will learn about the main functionalities of GenSphere, such as:\n",
        "\n",
        "\n",
        "1.   Defining workflows with yaml files;\n",
        "2.   Pulling from the platform;\n",
        "3.   Nesting workflows;\n",
        "4.   Using custom functions and schemas, as well as using langchain and composio tools;\n",
        "5.   Visualizing workflows;\n",
        "6.   Pushing to the platform.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "7lyQrTqqf4dR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **0. Install GenSphere and other libs to be used**"
      ],
      "metadata": {
        "id": "mvglv1mRhjhK"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "330uMhRrfwVG",
        "collapsed": true
      },
      "outputs": [],
      "source": [
        "!pip install gensphere"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **1. Import GenSphere**"
      ],
      "metadata": {
        "id": "cLMJ6JWqp4Fn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "import traceback\n",
        "\n",
        "\n",
        "# Set up logging configuration before importing other modules\n",
        "logging.basicConfig(\n",
        "    level=logging.DEBUG,\n",
        "    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',\n",
        "    handlers=[\n",
        "        logging.FileHandler(\"../../app.log\", mode='w'),\n",
        "        logging.StreamHandler()\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "fJ_x1QT_OUgD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from gensphere import genflow, yaml_utils\n",
        "from gensphere.genflow import GenFlow\n",
        "from gensphere.yaml_utils import YamlCompose\n",
        "from gensphere.visualizer import Visualizer\n",
        "from gensphere.hub import Hub\n",
        "import dotenv\n",
        "from dotenv import load_dotenv\n",
        "import os"
      ],
      "metadata": {
        "id": "B3MaB28Rp8IC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **2. Define your enviroment variables**"
      ],
      "metadata": {
        "id": "aY3K4vfihwJI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Replace these env variables with your corresponding API key."
      ],
      "metadata": {
        "id": "j6XRCre6h0-T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ['OPENAI_API_KEY']=\"PLACE_YOUR_OPENAI_API_KEY\"\n",
        "os.environ['COMPOSIO_API_KEY']=\"PLACE_YOUR_COMPOSIO_API_KEY\" #if you don't have one, visit composio.dev\n",
        "os.environ['FIRECRAWL_API_KEY']=\"PLACE-YOUR-FIRECRAWL-API-KEY\" # # if you don't have one, visit firecrawl.dev"
      ],
      "metadata": {
        "id": "JtxHn-sbhzqc",
        "collapsed": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!composio add firecrawl #for this project, we will be using firecrawl. Get an API key and add it by following the steps here."
      ],
      "metadata": {
        "id": "X9nSmz830ysD",
        "collapsed": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **3. Define your workflow with a yaml file.**"
      ],
      "metadata": {
        "id": "1b1-N3bJip6a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Our aim is to create workflow that automatically finds **latest product releases from producthunt, explores their revenue and traction, and analyzes a new startup idea based on that**. We will use pre-built components from the platform to accelerate our development."
      ],
      "metadata": {
        "id": "vFPWTV3c6EtU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **GenSphere project structure**"
      ],
      "metadata": {
        "id": "wScKRAjqddKl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "There are 3 fundamental files in a GenSphere project.\n",
        "\n",
        "\n",
        "1.   **Yaml file** - contains the workflow definition\n",
        "2.   **Functions file** - .py file containing all functions to be used, either as nodes in the graph or as tools during LLM function calling\n",
        "3.   **Schemas file** - .py file containing pydantic schemas. These are used when you want to use structured outputs from openAI.\n",
        "\n"
      ],
      "metadata": {
        "id": "09Z3TLZZdllN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **3.1 Pull a base yaml file from the platform**"
      ],
      "metadata": {
        "id": "AR10QjIzkPl5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this tutorial, we will use a **pre-built workflow from our open platform** that extracts information from producthunt.com. We will nest that into a bigger workflow to achieve our objective of analyzing a new startup idea.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "ZHLs_l1sofcx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "path_to_save_yaml_file='product_hunt_analyzer.yaml'\n",
        "path_to_save_functions_file='gensphere_functions.py'\n",
        "path_to_save_schema_file='structured_output_schema.py'\n",
        "\n",
        "hub=Hub()\n",
        "hub.pull(push_id='de8afbeb-06cb-4f8f-8ead-64d9e6ef5326',\n",
        "         yaml_filename=path_to_save_yaml_file,\n",
        "         functions_filename=path_to_save_functions_file,\n",
        "         schema_filename=path_to_save_schema_file,\n",
        "         save_to_disk=True)"
      ],
      "metadata": {
        "collapsed": true,
        "id": "pvZ4XdDP8Qif"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The yaml file has been saved locally as **\"product_hunt_analyzer.yaml\"**. We also saved the functions and schema files as **gensphere_functions.py** and **structured_output_schema.py**. Here are the full contents of the yaml file:"
      ],
      "metadata": {
        "id": "XP4ZGMVkEGSj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "```\n",
        "# product_hunt_analyzer.yaml\n",
        "\n",
        "nodes:      \n",
        "\n",
        "  - name: get_current_date\n",
        "    type: function_call\n",
        "    function: get_current_date_function\n",
        "    outputs:\n",
        "      - current_date\n",
        "\n",
        "  - name: get_timewindow\n",
        "\n",
        "    type: function_call\n",
        "    function: get_timewindow_function\n",
        "    outputs:\n",
        "      - time_window\n",
        "\n",
        "  - name: product_hunt_scrape\n",
        "    type: llm_service\n",
        "    service: openai\n",
        "    model: \"gpt-4o-2024-08-06\"\n",
        "    tools:\n",
        "      - COMPOSIO.FIRECRAWL_SCRAPE\n",
        "    params:\n",
        "      prompt: |\n",
        "\n",
        "         You should visit producthunt at https://www.producthunt.com/leaderboard/monthly/yyyy/mm\n",
        "         Today is {{ get_current_date.current_date }}\n",
        "         You should subsitute yyyy and mm by year and month you want to search.\n",
        "         The search time window should be {{ get_timewindow.time_window }}.\n",
        "         After that, you should extract raw content from the htmls associated,\n",
        "         which will contain information about new product launches, their companies, number of upvotes, etc.\n",
        "         Scroll the page until the end and wait a few miliseconds for it to launch before scraping.\n",
        "\n",
        "    outputs:\n",
        "      - product_hunt_scrape_results    \n",
        "\n",
        "      \n",
        "  - name: extract_info_from_search\n",
        "    type: llm_service\n",
        "    service: openai\n",
        "    model: \"gpt-4o-2024-08-06\"\n",
        "    structured_output_schema: StartupInformationList\n",
        "    params:\n",
        "      prompt: |\n",
        "\n",
        "         You are given reports from a search to https://www.producthunt.com/leaderboard/monthly/, containing\n",
        "         products featured there last month:\n",
        "         {{ product_hunt_scrape.product_hunt_scrape_results }}.\n",
        "         We want to extract accurate information about these new product launches.\n",
        "         Structure the information there by the following dimensions:  product name, company name, company url, number of upvotes, business model\n",
        "         brief description of it.\n",
        "    outputs:\n",
        "\n",
        "      - structured_search_info\n",
        "\n",
        "  - name: postprocess_search_results\n",
        "    type: function_call\n",
        "    function: postprocess_search_results_functions\n",
        "    params:\n",
        "      info: '{{ extract_info_from_search.structured_search_info }}'\n",
        "    outputs:\n",
        "      - postprocessed_search_results\n",
        "\n",
        "      \n",
        "\n",
        "  - name: find_extra_info\n",
        "    type: llm_service\n",
        "    service: openai\n",
        "    model: \"gpt-4o-2024-08-06\"\n",
        "    tools:\n",
        "\n",
        "      - COMPOSIO.TAVILY_TAVILY_SEARCH\n",
        "\n",
        "    params:\n",
        "      prompt: |\n",
        "\n",
        "         You should conduct a comprehensive search on the web about the following entry from producthunt.com:\n",
        "         {{ postprocess_search_results.postprocessed_search_results[i] }}. You should look to find relevant news\n",
        "         about the company, specially related to its revenue, valuation, traction, acquisition if applicable, number of users, etc.\n",
        "\n",
        "    outputs:\n",
        "      - startup_extra_info\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "c7F7ZjQMETrS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **3.2 Visualize your project**"
      ],
      "metadata": {
        "id": "ZxGi5xVUdMjE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's analyze the project we just pulled by using the visualizer class. **You can zoom in and out, and by clicking on a node, you can see all functions and schemas, inputs and outputs associated with it.**\n",
        "\n",
        "**OBS**: It is slightly cumbersome to visualize the graph from inside google colab. If you run locally, you can determine the address where the visualization will be run and access it through your browser."
      ],
      "metadata": {
        "id": "W5xEoi-FehOR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "viz=Visualizer('product_hunt_analyzer.yaml',\n",
        "               'gensphere_functions.py',\n",
        "               'structured_output_schema.py',\n",
        "               address='127.0.0.1', port=8050)\n",
        "viz.start_visualization()"
      ],
      "metadata": {
        "id": "GOHf80RUexbg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **3.3 Understand the syntax of the yaml file**"
      ],
      "metadata": {
        "id": "hBbISMzNpevC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "Let's understand how to work with YAML files step by step. **There are 3 types of node types: function_call, llm_service and yml_flow.**"
      ],
      "metadata": {
        "id": "j6XA2_NTaW_O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### 3.3.1 **function_call nodes**\n",
        "\n",
        "function_call nodes trigger function execution defined on a .py file (which you will pass when triggering execution). They have a **params field and output fields**.\n",
        "\n",
        "For instance, have a look at the get_current_date node:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "  - name: get_current_date\n",
        "    type: function_call\n",
        "    function: get_current_date_function\n",
        "    outputs:\n",
        "      - current_date\n",
        "```\n",
        "Here we are instructing GenSphere to execute the function get_current_date_function, and in this case there are no 'params'. This functions is defined on gensphere_functions.py which was pulled from the platform together with the yaml file.\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "# gensphere_functions.py\n",
        "\n",
        "import datetime\n",
        "\n",
        "def get_current_date_function():\n",
        "    return {'current_date':datetime.today().strftime('%Y-%m-%d')}\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "**Important notes**:  \n",
        "\n",
        "1.   If you want to use other nodes outputs as inputs, you can reference them with the syntax **{{ node name.output_name }}** in the 'params' field of the node.\n",
        "2.   **Functions output must be a dict**, whose keys must match the outputs defined in the yaml file.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "VpYH2gE4S0x7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### **3.3.2 llm_service nodes**\n",
        "\n",
        "These nodes execute LLM API calls. In the current version, we only support openAI, including [structured outputs](https://https://openai.com/index/introducing-structured-outputs-in-the-api/) and [function calling](https://https://platform.openai.com/docs/guides/function-calling). For instance, have a look at the node product_hunt_scrape:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "#product_hunt_analyzer.yaml\n",
        "\n",
        "   - name: product_hunt_scrape\n",
        "    type: llm_service\n",
        "    service: openai\n",
        "    model: \"gpt-4o-2024-08-06\"\n",
        "    tools:\n",
        "      - COMPOSIO.FIRECRAWL_SCRAPE\n",
        "    params:\n",
        "      prompt: |\n",
        "         You should visit producthunt at https://www.producthunt.com/leaderboard/monthly/yyyy/mm\n",
        "         Today is {{ get_current_date.current_date }}\n",
        "         You should subsitute yyyy and mm by year and month you want to search.\n",
        "         The search time window should be {{ get_timewindow.time_window }}.\n",
        "         After that, you should extract raw content from the htmls associated,\n",
        "         which will contain information about new product launches, their companies, number of upvotes, etc.\n",
        "         Scroll the page until the end and wait a few miliseconds for it to launch before scraping.\n",
        "    outputs:\n",
        "      - product_hunt_scrape_results\n",
        "```\n",
        "The **tools** field can refer to any function on your .py that defines functions.\n",
        "\n",
        "**You can also use [Composio](https://composio.dev/) tools**, with the syntax \"COMPOSIO.composio_tool_name\". Check [Composio's documentation](https://app.composio.dev/sdk_guide) for a detailed view on all available tools.\n",
        "\n",
        "**We also support [Langchain](https://langchain.com) tools**. You can use any tool from [langchain_community.tools](https://python.langchain.com/api_reference/community/tools.html) with the syntax \"LANGCHAIN.langchain_tool_name\"\n",
        "\n",
        "If you want your output from openAI to be a dict with a predetermined schema, **you can use the structured_output_schema field**, as in the node 'extract_info_from_search':\n",
        "\n",
        "```\n",
        "#product_hunt_analyzer.yaml\n",
        "\n",
        "  - name: extract_info_from_search\n",
        "    type: llm_service\n",
        "    service: openai\n",
        "    model: \"gpt-4o-2024-08-06\"\n",
        "    structured_output_schema: StartupInformationList\n",
        "    params:\n",
        "      prompt: |\n",
        "         You are given reports from a search to https://www.producthunt.com/leaderboard/monthly/, containing\n",
        "         products featured there in the following time window: {{ get_timewindow.time_window }}. Here\n",
        "         is the content of the search:\n",
        "         {{ product_hunt_scrape.product_hunt_scrape_results }}.\n",
        "         We want to extract accurate information about these new product launches.\n",
        "         Structure the information there by the following dimensions:  product name, company name, company url, number of upvotes, business model\n",
        "         brief description of it.\n",
        "    outputs:\n",
        "      - structured_search_info\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "XH2Rlaj_UAKj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The output will be an instance of the class **StartupInformationList**, which is defined on **structured_output_schemas.py** as\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "# structured_output_schema.py\n",
        "\n",
        "from pydantic import BaseModel, Field\n",
        "from typing import List\n",
        "\n",
        "class StartupInformation(BaseModel):\n",
        "    product_name: str = Field(..., description=\"The name of the product\")\n",
        "    company_name: str = Field(..., description=\"The name of the company that offers the product. Could be equal to name of the product\")\n",
        "    url: str = Field(..., description=\"URL associated with the product.\")\n",
        "    number_upvotes: int = Field(..., description=\"Number of upvotes associated with the product\")\n",
        "    business_model: str = Field(..., description=\"A brief description about the business model of the product or company\")\n",
        "    brief_description: str = Field(..., description=\"A brief description about the product\")\n",
        "\n",
        "class StartupInformationList(BaseModel):\n",
        "    information_list:List[StartupInformation]\n",
        "```\n",
        "\n",
        "The output of nodes with structured_output_schema are instances of the class defined on the schemas file (structured_output_schema.py in our case). To reference this output on other nodes, it is useful to introduce an additional post-processing node to extract information we want from the class instance. That's the purpose of the postprocess_search_results node:\n",
        "\n",
        "```\n",
        "  - name: postprocess_search_results\n",
        "    type: function_call\n",
        "    function: postprocess_search_results_functions\n",
        "    params:\n",
        "      info: '{{ extract_info_from_search.structured_search_info }}'\n",
        "    outputs:\n",
        "      - postprocessed_search_results\n",
        "```\n",
        "\n",
        "which applies the function **postprocess_search_results_functions**, defined on **gensphere_functions.py**.\n",
        "\n",
        "```\n",
        "def postprocess_search_results_functions(info):\n",
        "    result=info.model_dump().get('information_list')\n",
        "    return {'postprocessed_search_results':result}\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "bwD7tCloqFll"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### **3.3.3 yml_flow nodes**\n",
        "\n",
        "These nodes represent entire yaml files themselves. So, you can easily nest workflows by referencing other yaml files here. We will see an example of yaml file that contains yaml nodes below. For now, have a look at an example of a yml_flow node\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "- name: example_node_name\n",
        "    type: yml_flow\n",
        "    yml_file: path-to-yaml-file\n",
        "    params:\n",
        "      yml_flow_argument_example: 'xyz'\n",
        "    outputs:\n",
        "      - yml_flow_output_example\n",
        "```\n",
        "\n",
        "when referecing yml_flow nodes inside your yaml file, GenSphere will handle dependencies and **compose a combined yaml file** that is ready to run.\n",
        "\n"
      ],
      "metadata": {
        "id": "utah1ws-cZpg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### **3.3.4 Working with lists**\n",
        "\n",
        "Many times, the output of a node will be a python list and we will want to apply the next node to each individual element of the list.\n",
        "\n",
        "You can easily accomplish this with the syntax by **appending [i] after a node reference, as in {{node_name.output_name[i] }}**. If a node references that (either in its 'params' field or in the 'prompt' field for llm_service nodes), **GenSphere will execute the node to each element of the iterable and collect output as a list**. For instance, lets examine the node 'find_extra_info':\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "  - name: find_extra_info\n",
        "    type: llm_service\n",
        "    service: openai\n",
        "    model: \"gpt-4o-2024-08-06\"\n",
        "    tools:\n",
        "      - COMPOSIO.TAVILY_TAVILY_SEARCH\n",
        "    params:\n",
        "      prompt: |\n",
        "         You should conduct a comprehensive search on the web about the following entry from producthunt.com:\n",
        "         {{ postprocess_search_results.postprocessed_search_results[i] }}. You should look to find relevant news\n",
        "         about the company, specially related to its revenue, valuation, traction, acquisition if applicable,\n",
        "         number of users, etc.\n",
        "    outputs:\n",
        "      - startup_extra_info\n",
        "```\n",
        "\n",
        "Notice that the 'prompt' field references the output of the node 'postprocess_search_results' as:\n",
        "\n",
        "```\n",
        "{{ postprocess_search_results.postprocessed_search_results[i] }}\n",
        "```\n",
        "\n",
        "That means this GenSphere will take each element of  \"postprocessed_search_results\" (which is a list, as defined by its structured_output_schema applied in the node extract_info_from_search) and apply \"find_extra_info\" to every element of this node. **The outputs are then collected as a list**\n",
        "\n",
        "The end result is that we will do a different LLM API call for each entry that we found on product hunt separately. By doing so, we will get much better results than if we tried to find information about all entries at once.\n"
      ],
      "metadata": {
        "id": "YC3WuVythx6N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **3.4 Combine workflows to compose final yaml**"
      ],
      "metadata": {
        "id": "i6Er-bYL-_70"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The workflow we saw so far is able to retrieve information from product hunt and perform some extra web research on  companies there to find their revenue, number of users etc. **Now, we will embed this into a larger workflow** that takes a startup idea as input, runs the product hunt search workflow and creates a report explaining if there are potential competitors to the idea on recent product hunt launches, what are some market trends etc.\n",
        "\n",
        "Let's start with a new yaml file. It is already saved locally in the repo, inside the examples folder, at **https://raw.githubusercontent.com/octopus2023-inc/gensphere/refs/heads/main/examples/startup_idea_evaluator.yaml**"
      ],
      "metadata": {
        "id": "uiVGIkGKP6U5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "```\n",
        "#startup_idea_evaluator.yaml\n",
        "\n",
        "nodes:\n",
        "  - name: read_idea\n",
        "    type: function_call\n",
        "    function: read_file_as_string\n",
        "    params:\n",
        "      file_path: \"domains_to_search.txt\"\n",
        "    outputs:\n",
        "      - domains\n",
        "      \n",
        "  - name: product_hunt_analyzer\n",
        "    type: yml_flow\n",
        "    yml_file: product_hunt_analyzer.yaml\n",
        "    outputs:\n",
        "      - postprocessed_search_results\n",
        "      - startup_extra_info\n",
        "      \n",
        "  - name: generate_report\n",
        "    type: llm_service\n",
        "    service: openai\n",
        "    model: \"gpt-4o-2024-08-06\"\n",
        "    params:\n",
        "      prompt: |\n",
        "         You are a world class VC analyst. You are currently analyzing the following startup idea:\n",
        "         {{ read_idea.domains }}\n",
        "         Your task is to help analyze this idea in face of recent launches in product hunt.\n",
        "         Some recents launches in producthunt.com are:\n",
        "         {{ product_hunt_analyzer.postprocessed_search_results }}\n",
        "         Besides that, some extra information about these companies is:\n",
        "         {{ product_hunt_analyzer.startup_extra_info }}.\n",
        "        \n",
        "         Given that, you should create a detailed report containing the following:\n",
        "         1. An overview of recent launches in producthunt.com. What are the main ideas being explored?\n",
        "         2. A list of companies from producthunt launches that may become direct competitors to the startup idea.\n",
        "         Explain your rational\n",
        "         3. Create a list of the most promising startups from the producthunt launches, as defined by their\n",
        "         valuation, revenue, traction or other relevant metrics.\n",
        "         4. A table containing all information you found from producthunt launches.\n",
        "         \n",
        "         Answer in markdown format.\n",
        "         \n",
        "    outputs:\n",
        "      - report\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "tCeksmClSYYW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice that we have a **yml_flow** node being referenced here,\n",
        "\n",
        "```\n",
        "  - name: product_hunt_analyzer\n",
        "    type: yml_flow\n",
        "    yml_file: product_hunt_analyzer.yaml\n",
        "    outputs:\n",
        "      - postprocessed_search_results\n",
        "      - startup_extra_info\n",
        "```\n",
        "\n",
        "In the yml_file field, we have product_hunt_analyzer.yaml which is the path to the yaml file with the product analysis we were looking before. **That means this node will trigger the execution of the entire workflow defined on product_hunt_analyzer.yaml.**\n",
        "\n",
        "Now let's a new yaml, which we name \"combined.yaml\" with YamlCompose. GenSphere's class YamlCompose receives as input a yaml file, looks for yml_flow nodes there and resolves dependencies to create a final yaml file that is ready to run."
      ],
      "metadata": {
        "id": "Ksoro8mrVG4v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#copy the file from repo to local working directory in the notebook\n",
        "!wget -O startup_idea_evaluator.yaml https://raw.githubusercontent.com/octopus2023-inc/gensphere/refs/heads/main/examples/startup_idea_evaluator.yaml"
      ],
      "metadata": {
        "collapsed": true,
        "id": "XFhbtx6P86WY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "composer=YamlCompose('startup_idea_evaluator.yaml',\n",
        "                     'gensphere_functions.py',\n",
        "                     'structured_output_schema.py')\n",
        "combined_yaml_data=composer.compose(save_combined_yaml=True, output_file='combined.yaml')"
      ],
      "metadata": {
        "id": "93YFjt1PUDTU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Note:** When calling YamlCompose, you need to pass also the functions and schema files of the yaml file you want to parse. For simplicity, in our case you defined all functions we would need on the functions and schema files we pulled from the platform gensphere_functions.py, structured_output_schema.py"
      ],
      "metadata": {
        "id": "Xc3wImC_Ui2z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can now visualize the combined yaml file, and check that the workflows have been correctly nested."
      ],
      "metadata": {
        "id": "bq0Xq-JoYLBp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "viz=Visualizer('combined.yaml',\n",
        "               'gensphere_functions.py',\n",
        "               'structured_output_schema.py',\n",
        "               address='127.0.0.1', port=8050)\n",
        "viz.start_visualization()"
      ],
      "metadata": {
        "id": "VJmOoQAbYKcc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4. Run your project**\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "CtgGsPgTjC-j"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Having defined the yaml, functions and schema file, we can now trigger execution using the **GenFlow class**. We simply pass file paths to it and call the **\".run()\"** method.\n",
        "\n"
      ],
      "metadata": {
        "id": "7ngNboN8jJ8M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first node of combined.yaml, read_idea, expects a txt file saved locally as **\"domains_to_search.txt\"**. Let's create this file before executing the flow:"
      ],
      "metadata": {
        "id": "k1j4dRClaM1k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#create a save domains_to_search.txt\n",
        "\n",
        "startup_idea=\"\"\"\n",
        "startup that creates interactive voice agents using generative AI with emphasis on applications like\n",
        "language tutoring, entertainment or mental health. The business model would be B2C.\n",
        "\"\"\"\n",
        "with open(\"domains_to_search.txt\", \"w\") as text_file:\n",
        "    text_file.write(startup_idea)"
      ],
      "metadata": {
        "id": "4gJsMSWKaeCi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "logging.getLogger('composio').setLevel(logging.WARNING)\n",
        "logging.getLogger('gensphere').setLevel(logging.DEBUG)\n",
        "logging.getLogger('GenFlow').setLevel(logging.DEBUG)"
      ],
      "metadata": {
        "id": "AfpB-8fGSJgk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "flow=GenFlow('combined.yaml',\n",
        "             'gensphere_functions.py',\n",
        "             'structured_output_schema.py')\n",
        "flow.parse_yaml()\n",
        "flow.run()"
      ],
      "metadata": {
        "id": "EL228aVFjANp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "After execution is complete, you can access results with the **.outputs** atribute of GenFlow, which returns a dict with every node as key, and their outputs as values."
      ],
      "metadata": {
        "id": "pkze3I8JlrtP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "flow.outputs"
      ],
      "metadata": {
        "id": "IadJ5XRePOqD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Visualize final output node\n",
        "final_node_output=flow.outputs.get(\"generate_report\").get(\"report\")\n",
        "\n",
        "from IPython.display import display, Markdown\n",
        "display(Markdown(final_node_output))"
      ],
      "metadata": {
        "id": "QGgblEp2YsfG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **5. Push to the platform**"
      ],
      "metadata": {
        "id": "d6RQ30OH5Xzo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "After you have finished your project, you can now push your yaml, functions and schema to the platform. This will generate a push_id, that you or anyone else can use to pull your project locally. To do that, we simply call hub.push(), passing the path to the files. You can also add a brief description with \"push_name\"."
      ],
      "metadata": {
        "id": "EtFeWfPk5aaU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "hub=Hub(yaml_file='combined.yaml',\n",
        "        functions_file='gensphere_functions.py',\n",
        "        schema_file='structured_output_schema.py')\n",
        "result=hub.push(push_name='workflow to analyze startup idea based on recent producthunt launches.')"
      ],
      "metadata": {
        "id": "8wJ_wewkkJZY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"push id is {result.get('push_id')}\")\n",
        "print(f\"uploaded files are {result.get('uploaded_files')}\")"
      ],
      "metadata": {
        "id": "eQV13P-9-3NU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **6. Check project popularity**"
      ],
      "metadata": {
        "id": "n9fnIPd46VEv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can check how many times your project was pulled from the platform by using the 'count_pulls' method, and passing your push_id."
      ],
      "metadata": {
        "id": "KRmNLDZA6iz0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the total number of pulls for the push_id\n",
        "total_pulls = hub.count_pulls(push_id='de8afbeb-06cb-4f8f-8ead-64d9e6ef5326')\n",
        "print(f\"Total pulls for push_id: {total_pulls}\")"
      ],
      "metadata": {
        "id": "mIcEFbMx6feu"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}