Scrape Pipeline

Pipeline for web scraping operations with configurable depth and page limits.

Interactive Demo

Scrape a URL and see the results

URL

Max Depth

Max Pages

Code Example

scrape-demo.ts

import { ScrapePipeline, MemoryStore } from "@shivamsharma11/agent-sdk";

const pipeline = new ScrapePipeline({
  storage: new MemoryStore(),
  scrape: async (input) => {
    // Your scraping logic here
    const response = await fetch(input.url);
    return response.text();
  },
});

const result = await pipeline.run({
  url: "https://example.com",
  maxDepth: 2,
  maxPages: 10,
});