WC
Website Content Crawler
antsdata/website-crawler
AntsData 4.6 (205 reviews) 132K runs
Crawl whole sites and output structured text to feed RAG, vector stores and LLM apps.
#AI#RAG
INPUT
Configure your task
| Parameter | Type | Required |
|---|---|---|
startUrls Start URLs | string[] | Required |
maxDepth Crawl depth | number | Optional |
maxItems Max pages | number | Optional |
outputFormat Output format | string | Optional |
EXAMPLE
Input & output
input.json
{
"startUrls": [
"https://example.com"
],
"maxDepth": 2,
"outputFormat": "markdown"
}output.json
{
"url": "https://example.com/post",
"title": "Example Post",
"text": "…",
"markdown": "# Example…",
"wordCount": 860,
"crawledAt": "2026-06-01T10:00:00Z"
}API
Run with one request
POST · cURL
curl -X POST https://api.antsdata.com/v1/actors/antsdata/website-crawler/runs \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{"startUrls":["https://example.com"],"maxDepth":2,"outputFormat":"markdown"}'Full integration guide in the API docs.
Need different fields or sites?
Any public data — we can custom-scrape it and package it as a stable API.