Crawling your website is the quickest way to teach a Clone everything you've already published. You give IcloneU one starting URL; the crawler discovers the site's internal links on its own, extracts the page text, downloads linked files and transcribes embedded media — and the result lands in your Knowledge library as a single searchable source.
You'll need an existing Knowledge library (you can create one on the way), the full URL of the site or page you want to crawl, and the right to ingest that content — the platform doesn't check copyright or terms-of-service compliance for you. New to libraries? Start with What is the Knowledge Library?
What the crawler collects
A website source brings in three kinds of content — and the dialog you'll meet below has a switch for the last two:
- Page content — the text of every page the crawler discovers from your starting URL.
- Linked files — documents the crawled pages link to are downloaded and indexed alongside the pages themselves, when you include them (the estimate and the review leave documents out by default; see below). That includes files on a file address of your own, such as cdn., files. or static. in front of your site's address; a file kept on another domain is skipped without being listed anywhere. Each file can be up to 50 MB; one the crawl tries but can't download or read is listed under Errors in the run's Activity details (see Why are some pages or files missing from the crawl? below).
- Embedded media — audio and video found on the pages is transcribed, with YouTube, Vimeo, Dailymotion, SoundCloud, Spotify, iVoox, podcast feeds and HLS/DASH streams all recognised. Instagram, TikTok, Facebook and X are never read: their terms don't allow downloading their content, so neither a video a page embeds from them nor an address on one of them adds anything to your library.
After a whole YouTube channel or playlist rather than a website? That's the other web-family source: choose YouTube instead of Website when you add the source and give it the channel or playlist URL: before anything is processed, a Here's what I found panel shows how many new videos it found and their total length, and lets you take All the videos, The latest 100 or The latest 25 (the last two only when there are more new videos than that; on a big channel the latest 100 is picked for you), each with its estimated units. It checks up to 20 of the newest videos for captions, which cost nothing to transcribe. Start imports and transcribes the videos you chose; Choose the videos one by one brings back the full list. An address with no public videos shows I couldn't find any videos. Since production 1.122.0 the page stays on Sources while the videos import: each one shows as a line with its title, Queued and then Processing (a single video above the library's list, a channel's or playlist's videos inside its row), until it becomes its own source.
Add a website to your library
Configuration takes under a minute; the crawl itself runs in the background, so you can keep working while it does.
Open your library's sources
Select Knowledge in the left sidebar. On the Sources sub-tab, find the target library and click the chevron at the left of its row to expand it. No library yet? Click + New library — one is created immediately, named "New Library".
Choose Website from the Add Knowledge dropdown
Click the Add Knowledge dropdown and choose Website. (The same dropdown is also home to the other source types — YouTube, Files & Folders and Link Existing Collection.)
Enter the starting URL
Paste the full URL, including https://. One starting point is enough — the crawler discovers the site's internal links from there. If the URL is malformed, the dialog shows validation feedback.
Set the crawl scope, then Discover
Give the collection a name and adjust the scope controls — each one is explained in the table below. Then click Discover to find pages before anything is ingested. If you've added this site before, Collection Name becomes a list with that website already picked, and Discover takes you to it instead of starting over — see Adding a site you've already added.
TipThe defaults suit most sites: Include Files and Include Media on, with Max per channel preset to 100 if you switch Limit Media on. Often the only thing worth typing is the Collection Name.
Check the estimate, then Start
For a website you haven't added before, the dialog first reads the site's sitemap (Looking at the site…) and shows Here's what I found: how many addresses the sitemap lists and the pages in groups (content pages, policy pages, blog posts, product pages, archive listings and documents), each with a short reason, its estimated units (≈ N u) and a see example link, under I'll learn from or I'll skip. By default it learns from content, policy, blog and product pages and skips archive listings and documents; on a Shopify store the product pages are skipped too, because the products come from the store connection. Content and policy pages can't be skipped. Adjust lets you Include or Skip the other groups, and a blog with more than 50 posts offers All, Latest 50 or None. The footer shows the Estimated total; videos and files inside the pages are priced after the crawl, before anything is processed. Click Start to begin the crawl, or Cancel. A site with no sitemap shows I couldn't count this site's pages up front instead, and you choose after discovery.
NoteA group you skip here becomes one of this website's crawl rules, for later syncs too: its folders are added to Excluded Paths when they can be told apart from the pages you keep, and when the sitemap lists documents, leaving documents skipped switches Include Files off. Turn it back on in the website's settings if you want linked files.
Review what was found, then Start
Once discovery is queued, the dialog closes and the website shows up in the library as its own row. Its Status shows Queued, then a live count like Discovering: 22 found, 140 reviewed — found counts unique pages and often plateaus while the crawler works through pagination and duplicates; the reviewed number keeps climbing the whole time, so a rising reviewed count means the crawl is healthy even when found stops moving. When discovery finishes, the row shows Review Items and the bell shows Discovery complete. Click Review Items on the row (or on the notice) and the dialog shows Here's what the crawl found: the same groups over what the crawl actually found, plus the videos (N videos · length), learned by default while Include Media is on and priced ≈ N u, and the linked files it found, left out by default and priced up to ≈ N u. On a website already in your library it adds how many items you already have; if nothing is new it says Everything the crawl found is already in your library, with Close. Choose what to learn and click Start. Cancel removes a new website that holds nothing yet; the dialog's own Close keeps the review waiting on the row. (If the review can't be priced, you get the older list instead: every item with a checkbox, Pages / Files / Media filters and Process (N).) The dialog closes again, and the row shows Queued and a live count while your picks are processed, then Ready; the Activity sub-tab logs the progress of each phase. If discovery finds nothing to add, the row shows No sources found instead and a bell notice says why — see Why did the crawl find 0 pages? below.
When the row shows Ready, everything the crawl extracted is searchable by any Clone attached to the library.
Expand the website's row to see what the estimate became: ≈ N estimated · M used, with Learned and Skipped columns by group. Include anyway on a skipped group (other than a Shopify store's products) adds its remaining items in the background (N items will be added in the background.). To take out single pages, remove them from the library. Websites added before this estimate existed show no summary.
The scope controls, explained
Step 4's controls decide how far the crawl reaches and how much it brings back. These are the controls and their defaults:
| Control | Default | What it's for |
|---|---|---|
| Collection Name | — | The label this source shows in your library |
| Include Files | On | Download the files that crawled pages link to |
| Include Media | On | Transcribe embedded audio and video found on pages |
| Limit Media | Off | Cap how much embedded media the crawl takes on |
| Max per channel | 100 (range 1–2000) | Upper bound on items collected per channel |
| Excluded Subdomains | — | Comma-separated subdomains to skip, e.g. music, audio, store |
| Excluded Paths | — | Comma-separated paths to skip, e.g. /legal, /privacy, /admin |
Legal boilerplate, admin areas and store pages rarely make good knowledge. A minute spent on Excluded Paths and Excluded Subdomains keeps the crawl lean — and the answers cleaner.
What a crawl covers
The starting URL decides how far a crawl reaches. These rules tell you what to expect back:
- Subdomains — a starting URL at the top of your site, such as https://www.example.com, also takes in its subdomains (blog., shop. and the like). Technical ones such as cdn., mail., login., admin. or staging. are never crawled for pages, though a document your pages link to on a technical address of your own, such as cdn., files. or static., is still taken (see Linked files above); list any others you want left out under Excluded Subdomains.
- Folders — a starting URL inside a folder keeps the crawl to that folder on that one address: https://example.com/blog/my-post covers /blog/ and nothing else. A starting URL such as https://example.com/about, or /blog without the final slash, covers the whole site.
- Redirects — if the starting URL redirects to another address (example.com to example.com.mx, say), the crawl follows it and covers the site that answered. For the fullest crawl, start from the address your site ends up on.
- Sitemaps — pages listed in your site's sitemap (the list of pages many sites publish for search engines) are crawled even when no menu links to them, as long as they're inside the crawl's reach. The tag, category and author sitemaps that WordPress and similar systems publish are skipped, unless the site's sitemap lists nothing else: their pages only repeat excerpts of posts the crawl already reads.
- Archive pages — page 2 and later of a paginated listing, such as a blog archive's /page/2, /pagina/3 or ?paged=2, is read so the crawl can reach the older posts it links to, but it isn't added to your library; the listing's first page is.
- Size — the crawl follows links up to five steps from the starting page and takes up to 1,000 pages; that can rise to 5,000 for a site whose sitemap lists more (tag, category and author sitemaps don't count).
- robots.txt — the crawler respects your site's robots.txt (the file where a site tells crawlers what they may read), so pages it rules out aren't read. The run's details in Activity list them under Skipped at the site's request (N).
- Public addresses only — since production 1.119.0 the crawler reads only addresses on the public internet. An address that is, or whose name points to, localhost, a private network address such as 192.168.x.x or 10.x.x.x, or another internal address isn't read, and neither is a link or redirect to one, so a site on an office network or an intranet can't be crawled: export its pages and upload them instead. If you add such an address, the dialog says We couldn't find this website. Check that the URL is spelled correctly; the domain may have a typo., even when it's spelled right.
- Language — pages are requested in the account's language (the one set under Your details, in Account settings; in a team, the account owner's). On a site that shows each visitor a different language, that's the version the crawler reads. When a site publishes a page in several languages and lists its versions (most multilingual sites do, for search engines), the crawler learns each page once: the version in the account's language or, when the site has none, the one in its default or main language. The other versions aren't added to your library. If your library learned a site's other versions before the crawler worked this way, the next New + Changed sync moves the ones it no longer learns to the Recycle Bin, unless that sync would remove more than a fifth of the site's pages in your library: then its syncs remove nothing while that's so, and both languages stay until you remove the other language's pages yourself or delete the website and add it again.
- Shopify stores — if the site is a Shopify store, the crawl skips its product and collection pages and reads the content pages (about, policies, blog, FAQ and so on). Products come from connecting the store instead — see Connecting your Shopify store. A WooCommerce site works the same way once your account has an active WooCommerce connection: the crawl leaves out its product pages and product listings, and the clone gets products from the connection.
Adding a site you've already added
The dialog compares the domain in the address you type (the example.com part) with the websites you've already added, in all your libraries — this library's first. A website counts only if its domain is exactly the same, with or without www.: example.com matches www.example.com, but not example.com.mx or blog.example.com, and websites in the Recycle Bin don't count. When one matches, Collection Name becomes a list with that website already picked, showing its number of sources, and Discover takes you to it instead of starting a second crawl:
- Something is running on it (a crawl or processing) — the dialog says so under the address and stays open. Wait for that job to finish, then try again.
- Its pages were found but never processed — the dialog opens the list of pages found last time, with the date of that search (Pages found on <date>); if the list is more than 7 days old, Scan again searches the site afresh in the background. Choose what to learn and click Start (the review groups it as described above): the dialog closes and the processing runs in the background, as with any website you add.
- It already has sources — if it's in another library, it's added to this one too, so its pages show here as well. The dialog closes and the Sync window opens, the same one as the Synchronize button on the website's row (see When the website changes).
- It has no sources and no pages found yet — it's reused, and the crawl runs as it would for a new site.
To add the site as a separate website instead — just one folder of it, say — choose New Collection in the list once the address is typed, just before you click Discover (editing the address picks the matching website again); the new website gets a numbered name, such as “example.com (2)”.
When the website changes
A website source is a snapshot taken at the moment of the crawl, not a live mirror. Your Clone reads from that indexed snapshot — publish a price change or a new blog post five minutes after the crawl and the Clone won't see it until the source is re-synchronised. Three ways to bring it up to date:
- Scheduled re-sync — on the Knowledge page's Synchronization sub-tab, switch on the site's own Auto-sync and the platform re-fetches it on a schedule; it stays off until you do. See Keeping your library up to date.
- Manual re-sync — click the circular-arrow Synchronize button on the website's row in Sources when you've pushed a change you want reflected straight away. The Sync window it opens starts on New + Changed, which adds new pages, updates changed ones and moves pages the crawl no longer finds to the Recycle Bin; New only just adds new pages. Every option is explained in Keeping your library up to date.
- Replace the source — delete the website source and add it again. A fresh crawl from scratch is sometimes cleaner than a re-sync when the site's structure has changed significantly.
Two things a re-sync won't preserve. First, on a New + Changed sync, pages the sync can't find anywhere — neither through links from your starting URL nor in your site's sitemap — are moved to the Recycle Bin. The run lists them on the Activity sub-tab as Removed (N) — no longer on the site, and you can restore them from the Recycle Bin as a set. A page you've unlinked from your menus stays in the library as long as your sitemap still lists it. If more than a fifth of the site's pages in your library would go at once, IcloneU treats the crawl as unreliable and removes none of them. The same sync also removes pages the crawl reaches but no longer keeps — page 2 and later of archive listings, and tag, category or author archives it found only through the site's tag, category and author sitemaps, which it skips — and if those are more than a fifth of the site's pages in your library, it removes nothing at all; delete the website and add it again to start clean. Second, content that has moved behind a login simply stops appearing, because the crawler doesn't authenticate. If a specific page must survive regardless, ingest it separately as a file source (export it to PDF, say, and upload that).
After every re-sync, the Activity sub-tab logs the run — worth a spot-check that it finished and the new content is in. Cadences, cost guards and the rest of the sync configuration are covered in Keeping your library up to date.
Frequently asked
Setting one up takes under a minute; the crawl and processing then run in the background. How long that takes depends on the site's size and depth and on how much embedded media has to be downloaded and transcribed — small text-heavy sites are typically quick, media-heavy ones take longer. Treat those as rough guides rather than guaranteed times (IcloneU doesn't publish processing SLAs), and watch the Activity sub-tab for live progress. During discovery the source row also shows a live status like Discovering: 22 found, 140 reviewed — a climbing reviewed count means the crawl is still working, even when found has stopped moving.
Check the bell. When a crawl of a site you're adding finds nothing, its row shows No sources found and a Discovery found no items notice explains why (after a sync that finds nothing, the notice is We could not check your site). It gives one of four kinds of reason. The first is that the site is blocking automated requests, with what gave it away in parentheses: the error code the site sent to turn the crawler away (such as HTTP 403), robots.txt when your site's robots.txt rules the crawler out, or the anti-bot page the crawler recognised (such as Cloudflare challenge). The notice then says what that block needs: for robots.txt, allow the user agent IcloneU-KnowledgeCrawler in it; for a login (HTTP 401), start from a public page or upload the content, since pages behind a login can't be crawled; for a rate limit (HTTP 429 or 503), nothing needs to change, just try again in a few minutes; for anything else, ask whoever runs the site to allow the crawler, User-Agent IcloneU-KnowledgeCrawler/1.0 and the IcloneU server's outbound IP. The other three: the site returned a very small response with no links, which usually means a challenge or captcha page; the page loaded but we found no links (the page may need JavaScript to show its menu, or may not link to other pages); or we could not reach the site, or it did not return a web page. The last one, which asks you to check the address, is what's left when nothing else fits. After a sync, it usually means the starting page never loaded or couldn't be read: the site didn't answer, the page doesn't exist (404) or sent back another error, or it isn't a web page. When you're adding a site, IcloneU has already checked that the address answers before crawling, so if the address is right, the likelier cause is a starting page that isn't a web page, or an address that sends visitors on to another one (example.com to www.example.com, say), where IcloneU can't always tell why the crawl came back empty: start from the address your site ends up on. Each notice ends with what to try next; once that's done, click Synchronize on the row to try again. If the site keeps the crawler out, use a different ingestion route, such as exporting the pages (to PDF, say) and uploading them as file sources — see Adding documents and files.
IcloneU won't read a site whose security certificate doesn't check out, and it tells you which problem it found: the certificate has expired, was issued for a different address, is incomplete, or wasn't issued by a trusted authority. When you first add the site, the Add Website dialog says so and doesn't start the crawl. On a later sync, the run stops, Activity lists the reason under Errors, and the bell shows Your website's security certificate needs attention. The fix is on the site's side: ask whoever hosts your site to renew or reinstall the certificate, then sync again.
Start with the run's details in Activity: pages and linked files the crawl couldn't take are listed under Errors, each with a reason. No readable content on the page means a page your library already held had almost no text left even after the crawler opened it in a browser (a new page like that is kept as a Too little text row instead; see the next question), or a linked file had no text it could read (a password-protected document, for example); neither is retried. File too large is a linked file over 50 MB — upload it directly instead — and Could not extract content is a linked file that couldn't be converted. The site blocked access means the site refused that page; you can retry it. A linked file that doesn't show up at all is usually kept on another domain, such as a cloud-storage service, which the crawl skips — upload it directly instead. Pages that don't show up at all are usually outside the crawl's reach (see What a crawl covers) or behind a login: the crawler isn't known to authenticate against arbitrary sites, so export those pages manually and upload them as file sources instead.
Usually, yes. When a page is built with a framework such as React, Vue, Angular, Next.js, Nuxt, Svelte, Blazor or HubSpot, or its HTML carries very little text, the crawler opens it in a real browser and reads what a visitor would see. Since production 1.151.0 a page that still has almost no text is kept in the collection as an excluded row tagged Too little text, so later syncs don't fetch it again and nothing from it reaches your Clone; if the page later gets real content, delete that row and the next sync learns it. Only a page your library already held, which now has almost no text, is listed under Errors as No readable content on the page, and it keeps what was learned from it before. If a whole crawl comes back empty with a notice that the page loaded but no links were found, start from a page that lists your content.
Pages that vanish from the site aren't deleted outright. A re-sync moves them to the Recycle Bin and lists them on the Activity sub-tab as Removed (N) — no longer on the site. Open the Recycle Bin to restore the removed set. A page that's still on your site can land there too if the sync didn't reach it — because the site refused it that time, say, or the crawl hit its page limit.
Media that came out empty usually shows in the website's collection as an excluded row, with a short tag after its name saying why: No speech found, Couldn't transcribe, Not available on the platform or Longer than the allowed duration. Nothing from it reaches your Clone, and syncs don't fetch it again. The hosting platform may have changed its access policy, or the media may be geo-restricted. To try again, use the row's circular-arrow button, Retry: process this media again; if it still comes out empty, it stays excluded. If the content matters, you can also try adding the media directly via its own URL, where a source type accepts it — see Keeping your library up to date.
There's a better route: choose YouTube instead of Website in the Add Knowledge dropdown, give it a channel or playlist URL, and you choose all its videos or the latest ones before they're imported and transcribed. (YouTube videos embedded in pages you crawl are transcribed as part of the website source too.)
Last updated October 31, 2026 · Knowledge Library
