Skip to content

Web Scraper and Crawler for LLM Apps and AI Workflows with NoCode / LowCode. Plug and play with your own logic and customize it flexibly and scalably on BuildShip.

Notifications You must be signed in to change notification settings

rowyio/LLM-Web-Crawler

Repository files navigation

Web Crawler for LLM Apps

Flexible and Scalable low-code Web Crawler. Give it a try on the LIVE playground: https://llm-web-crawler.vercel.app/

Uses BuildShip - a visual AI workflow builder to extract and gather data from your websites or sources. This data can then be used as a knowledge base to power your own LLM apps 🤖, or paired with BuildShip's AI Assistant to unlock powerful use cases and enhance your business or services.

Video Tutorial

yotube

Features

Node Info Documentation Template
Scrape Easy to get started with, scrape a given web URL and return the text content. Works great for less complex sites that don't rely on JavaScript to load. Read more Remix
Dynamic Scrape Scrape a given web URL and return the text content. This method works well for more complex sites and allows for more interactive scraping by providing a set of steps to execute after loading the page. For example, loading an ecommerce site, searching for an item, and then scraping the search results info. Read more Remix
Web Crawler Extract data from an entire website by crawling through and scraping all its pages. Perfect for aggregating data to create your own custom GPTs or "Chat with Data" apps. Read more Remix
LLM Extraction Extract structured data (just the data you care about) from any website. No need to scrape an entire webpage; simply specify the URL and the fields you want to extract. The LLM will handle the rest, delivering only the relevant data in a structured format. Read more Remix

How to use

Read full documentation to learn more

About

Web Scraper and Crawler for LLM Apps and AI Workflows with NoCode / LowCode. Plug and play with your own logic and customize it flexibly and scalably on BuildShip.

Topics

Resources

Stars

Watchers

Forks