How to turn a website into a research corpus

Saving a web page the normal way keeps the navigation bar, the cookie banner and the ads, and often loses the article underneath them. A page rendered with JavaScript can save as an almost empty file. Neither is something an AI, or a person, can read as a source.

Even when a save does capture the article, it is a single static file with no easy way to combine it with dozens of others into one document.

Web Extractor fetches a page and returns the main content as Markdown, text, HTML or JSON, with headings and links kept and the surrounding chrome dropped. Pages that need JavaScript to render are handled by a browser-rendering mode that scrolls for lazy-loaded content and can click a load-more control.

The choice between a plain fetch and browser rendering does not need to be made manually every time; the tool can be told to render when a page depends on JavaScript.

A crawl starts from one page and follows links within a defined scope, up to 200 pages and 5 levels deep, and downloads the result as one set. That turns a documentation site or a multi-page report into a single corpus instead of dozens of individual saves.

Scope rules keep the crawl inside the intended section of a site, such as a documentation subdirectory, instead of wandering into unrelated pages linked from the footer.

For a page that is really a list, such as search results or a product catalogue, fields can be defined with CSS selectors so the extraction returns JSON objects instead of prose.

This turns a page that would otherwise need manual copying, such as a directory or a results page, into rows that a spreadsheet or a script can use directly.

Every extracted page lands in your history, where it can be summarized, questioned with citations, or turned into a brief alongside documents and transcripts from other sources.