# Web
**Version:** 2.0.1

Arcade.dev LLM tools for web scraping related tasks

## Overview

Arcade.dev provides a powerful toolkit designed for web scraping tasks, enabling developers to efficiently manage and extract data from websites. This toolkit leverages the Firecrawl API to provide a range of functionalities for both synchronous and asynchronous crawling.

**Capabilities**:
- Initiate and manage web crawls, with options for both synchronous and asynchronous operations.
- Retrieve crawl data and status updates for ongoing or recently completed tasks.
- Map entire websites starting from a single URL.
- Scrape specific URLs and receive data in various formats.

**Secrets**:
- API key required for accessing Firecrawl, named `FIRECRAWL_API_KEY`.

## Tools

This toolkit provides **6 tools**:

### Quick Reference

- [`Web.CancelCrawl`](#webcancelcrawl) — Cancel an asynchronous crawl job that is in progress using the Firecrawl API.
- [`Web.CrawlWebsite`](#webcrawlwebsite) — Crawl a website using Firecrawl. If the crawl is asynchronous, then returns the crawl ID.
- [`Web.GetCrawlData`](#webgetcrawldata) — Get the data of a Firecrawl 'crawl' that is either in progress or recently completed.
- [`Web.GetCrawlStatus`](#webgetcrawlstatus) — Get the status of a Firecrawl 'crawl' that is either in progress or recently completed.
- [`Web.MapWebsite`](#webmapwebsite) — Map a website from a single URL to a map of the entire website.
- [`Web.ScrapeUrl`](#webscrapeurl) — Scrape a URL using Firecrawl and return the data in specified formats.

---

## Tool Details

### Web.CancelCrawl

**Version:** 2.0.1

Cancel an asynchronous crawl job that is in progress using the Firecrawl API.

**Parameters:**

- `crawl_id`: `string` *(required)*
  - The ID of the asynchronous crawl job to cancel

**Required Secrets:**
- `FIRECRAWL_API_KEY` (api_key)

**Output:** `json`
- Cancellation status information

**Example:**

```json
{
  "tool": "Web.CancelCrawl",
  "parameters": {
    "crawl_id": "crawl_12345"
  },
  "secrets": [
    {
      "name": "FIRECRAWL_API_KEY",
      "type": "api_key"
    }
  ]
}
```

---

### Web.CrawlWebsite

**Version:** 2.0.1

Crawl a website using Firecrawl. If the crawl is asynchronous, then returns the crawl ID.
If the crawl is synchronous, then returns the crawl data.

**Parameters:**

- `url`: `string` *(required)*
  - URL to crawl
- `exclude_paths`: `array[string]` *(optional)*
  - URL patterns to exclude from the crawl
- `include_paths`: `array[string]` *(optional)*
  - URL patterns to include in the crawl
- `max_depth`: `integer` *(optional)*
  - Maximum depth to crawl relative to the entered URL
- `ignore_sitemap`: `boolean` *(optional)*
  - Ignore the website sitemap when crawling
- `limit`: `integer` *(optional)*
  - Limit the number of pages to crawl
- `allow_backward_links`: `boolean` *(optional)*
  - Enable navigation to previously linked pages and enable crawling sublinks that are not children of the 'url' input parameter.
- `allow_external_links`: `boolean` *(optional)*
  - Allow following links to external websites
- `webhook`: `string` *(optional)*
  - The URL to send a POST request to when the crawl is started, updated and completed.
- `async_crawl`: `boolean` *(optional)*
  - Run the crawl asynchronously

**Required Secrets:**
- `FIRECRAWL_API_KEY` (api_key)

**Output:** `json`
- Crawl status and data

**Example:**

```json
{
  "tool": "Web.CrawlWebsite",
  "parameters": {
    "url": "https://example.com",
    "exclude_paths": [
      "/exclude/*",
      "/private/*"
    ],
    "include_paths": [
      "/include/*",
      "/public/*"
    ],
    "max_depth": 3,
    "ignore_sitemap": false,
    "limit": 100,
    "allow_backward_links": true,
    "allow_external_links": false,
    "webhook": "https://webhook.example.com/crawl",
    "async_crawl": true
  },
  "secrets": [
    {
      "name": "FIRECRAWL_API_KEY",
      "type": "api_key"
    }
  ]
}
```

---

### Web.GetCrawlData

**Version:** 2.0.1

Get the data of a Firecrawl 'crawl' that is either in progress or recently completed.

**Parameters:**

- `crawl_id`: `string` *(required)*
  - The ID of the crawl job

**Required Secrets:**
- `FIRECRAWL_API_KEY` (api_key)

**Output:** `json`
- Crawl data information

**Example:**

```json
{
  "tool": "Web.GetCrawlData",
  "parameters": {
    "crawl_id": "12345-abcde-67890-fghij"
  },
  "secrets": [
    {
      "name": "FIRECRAWL_API_KEY",
      "type": "api_key"
    }
  ]
}
```

---

### Web.GetCrawlStatus

**Version:** 2.0.1

Get the status of a Firecrawl 'crawl' that is either in progress or recently completed.

**Parameters:**

- `crawl_id`: `string` *(required)*
  - The ID of the crawl job

**Required Secrets:**
- `FIRECRAWL_API_KEY` (api_key)

**Output:** `json`
- Crawl status information

**Example:**

```json
{
  "tool": "Web.GetCrawlStatus",
  "parameters": {
    "crawl_id": "abc12345-def6-7890-gh12-ijklmnopqrst"
  },
  "secrets": [
    {
      "name": "FIRECRAWL_API_KEY",
      "type": "api_key"
    }
  ]
}
```

---

### Web.MapWebsite

**Version:** 2.0.1

Map a website from a single URL to a map of the entire website.

**Parameters:**

- `url`: `string` *(required)*
  - The base URL to start crawling from
- `search`: `string` *(optional)*
  - Search query to use for mapping
- `ignore_sitemap`: `boolean` *(optional)*
  - Ignore the website sitemap when crawling
- `include_subdomains`: `boolean` *(optional)*
  - Include subdomains of the website
- `limit`: `integer` *(optional)*
  - Maximum number of links to return

**Required Secrets:**
- `FIRECRAWL_API_KEY` (api_key)

**Output:** `json`
- Website map data

**Example:**

```json
{
  "tool": "Web.MapWebsite",
  "parameters": {
    "url": "https://example.com",
    "search": "products",
    "ignore_sitemap": false,
    "include_subdomains": true,
    "limit": 50
  },
  "secrets": [
    {
      "name": "FIRECRAWL_API_KEY",
      "type": "api_key"
    }
  ]
}
```

---

### Web.ScrapeUrl

**Version:** 2.0.1

Scrape a URL using Firecrawl and return the data in specified formats.

**Parameters:**

- `url`: `string` *(required)*
  - URL to scrape
- `formats`: `array[string]` *(optional)*
  - Formats to retrieve. Defaults to ['markdown'].
  - Allowed values: `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `screenshot@fullPage`
- `only_main_content`: `boolean` *(optional)*
  - Only return the main content of the page excluding headers, navs, footers, etc.
- `include_tags`: `array[string]` *(optional)*
  - List of tags to include in the output
- `exclude_tags`: `array[string]` *(optional)*
  - List of tags to exclude from the output
- `wait_for`: `integer` *(optional)*
  - Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load.
- `timeout`: `integer` *(optional)*
  - Timeout in milliseconds for the request

**Required Secrets:**
- `FIRECRAWL_API_KEY` (api_key)

**Output:** `json`
- Scraped data in specified formats

**Example:**

```json
{
  "tool": "Web.ScrapeUrl",
  "parameters": {
    "url": "https://example.com",
    "formats": [
      "markdown",
      "html"
    ],
    "only_main_content": true,
    "include_tags": [
      "p",
      "h1",
      "h2"
    ],
    "exclude_tags": [
      "header",
      "footer",
      "nav"
    ],
    "wait_for": 2000,
    "timeout": 5000
  },
  "secrets": [
    {
      "name": "FIRECRAWL_API_KEY",
      "type": "api_key"
    }
  ]
}
```

---


*Documentation generated: 2026-01-26T17:46:38.751Z*