Website content crawler: how to get clean page text as JSON/CSV (API, Python, no code)

Crawls any public website from a start URL, follows internal links up to a set depth, and extracts title, breadcrumbs, clean text body, and word count. This guide shows the output fields, a working input, and Python, JavaScript and cURL code for the Actor v0iddo/website-content-crawler on Apify.

Open Website content crawler on Apify

What data you get

One row per result. Every row can be downloaded as JSON, CSV, Excel or HTML, or read through the API.

Input example

This is the input that is pre-filled on the Actor page. Change the values for your own run.

{
  "startUrls": [
    "https://docs.python.org/3/library/"
  ],
  "maxDepth": 1,
  "maxPagesPerUrl": 15,
  "maxItems": 20
}

Output sample

Real rows from a run of the Actor. Fields that name people are left out of the sample.

[
  {
    "url": "https://flask.palletsprojects.com/en/stable/",
    "title": "Welcome to Flask",
    "h1": "Welcome to Flask",
    "breadcrumbs": [
      "Flask Documentation (3.1.x)",
      "Welcome to Flask"
    ],
    "textContent": "Welcome to Flask  Welcome to Flask’s documentation. Flask is a lightweight WSGI web application framework.\nIt is designed to make getting...",
    "wordCount": 920,
    "depth": 0
  },
  {
    "url": "https://flask.palletsprojects.com/en/stable/genindex/",
    "title": "Index",
    "h1": "Index",
    "breadcrumbs": [
      "Flask Documentation (3.1.x)",
      "Index"
    ],
    "textContent": "Index _ | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y _ _AppCtxGlobals (class in flask....",
    "wordCount": 1644,
    "depth": 1
  }
]

Code: Python, JavaScript and cURL

Get your API token in Apify Console under Settings, then Integrations. The Actor id is v0iddo/website-content-crawler.

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {'startUrls': ['https://docs.python.org/3/library/'],
 'maxDepth': 1,
 'maxPagesPerUrl': 15,
 'maxItems': 20}

run = client.actor("v0iddo/website-content-crawler").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const input = {
  "startUrls": [
    "https://docs.python.org/3/library/"
  ],
  "maxDepth": 1,
  "maxPagesPerUrl": 15,
  "maxItems": 20
};

const run = await client.actor('v0iddo/website-content-crawler').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

cURL

curl -X POST "https://api.apify.com/v2/acts/v0iddo~website-content-crawler/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.python.org/3/library/"], "maxDepth": 1, "maxPagesPerUrl": 15, "maxItems": 20}'

Ready-to-run example scripts: example repository on GitHub.

Price

$2.55 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.78 per 1,000. See the pricing tab for your plan. A new Apify account gets $5 of platform credit every month.

Good to know

• The crawler stays within the same origin and path prefix as each start URL. Use includeUrlPattern or excludeUrlPattern to narrow scope on large sites; combine with maxPagesPerUrl to control cost.
• Set maxDepth=0 to extract only the start page. maxDepth=1 adds all pages linked from the start page. Depths above 3 can produce thousands of pages on large sites.
• Pages behind login, paywalls, or bot walls are not accessible.
• Not affiliated with any documentation platform or website.

FAQ

What data does the Website content crawler return?

Each result is one row in an Apify dataset. Fields include url, title, h1, breadcrumbs, description, textContent, wordCount, depth. The table above shows every field with an example value.

How do I use the Website content crawler from Python?

Install the apify-client package, create a client with your Apify API token, call the Actor v0iddo/website-content-crawler with an input like the one above, and read the rows from the run's dataset. The Python snippet above does exactly that.

Can I use it without writing code?

Yes. Open the Actor page on Apify, fill in the input form (an example is already filled in), press Start, and download the results as JSON, CSV, Excel or HTML. You can also schedule runs there.

How much does it cost?

$2.55 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.78 per 1,000. A new Apify account gets $5 of platform credit every month.

How do I limit the number of results?

Set the maxItems input field to the number of results you need; the run stops there, so it also caps the cost of one run. You can also set a maximum cost for a run in the run options on Apify, and the run stops when it is reached.

Can I send the results to Google Sheets, Make, Zapier or n8n?

Yes. Apify datasets connect to Google Sheets, Slack, Make, Zapier and n8n, and a webhook can call your own service when a run finishes. You can also export the dataset as JSON, CSV or Excel.

Open Website content crawler on Apify

Related guides

• Google Maps Scraper: Scrape Google Maps business listings by search and place: name, category, address, coordinates, rating, rev...
• Bulk file and image downloader: Download files in bulk from direct links, or every image, video, audio file, PDF, document or archive on a...
• Video frame extractor: Extract still frames from video files as JPEG, PNG or WebP images with timecodes: one every N seconds, a nu...