​
Overview

This Runchat uses the Tavily API to crawl web pages starting from a given URL. It can either map out the site structure by returning URLs (map mode) or extract content snippets from crawled pages based on a query (crawl mode). Crawl depth and breadth can be controlled.

​
Inputs

  • url (string): The starting URL for the crawl.
  • maxDepth (number): The maximum depth of links to follow from the starting URL.
  • maxBreadth (number): The maximum number of links to follow on each page.
  • limit (number): The maximum number of results (URLs or snippets) to return.
  • query (string): (Optional) Natural language instructions or keywords to guide the crawler, especially relevant in crawl mode to find specific content.
  • mode (select): Choose between “crawl” (returns content snippets matching the query) or “map” (returns only the URLs found during the crawl).

​
Outputs

  • response (object): The JSON response from the Tavily API. In map mode, it primarily contains a list of discovered URLs. In crawl mode, it contains snippets of content relevant to the query from the crawled pages. Includes the base URL and response time.