Browse docs

Crawling and indexing

How Morrison finds, crawls and indexes your pages, and how to manage sources and schedules under Data Sources.

Crawling is the foundation of everything in Morrison. Before you can chat with your content, score it or run workflows on it, Morrison visits your pages, extracts their content and indexes it so it can be searched.

You set this up once in the setup wizard. After that, everything is managed under Data Sources in the sidebar.

Where Morrison finds pages

A data source tells Morrison where to look. There are three types:

  • Sitemap: Morrison reads your XML sitemap, including sitemap indexes. Leave the sitemap URL empty and Morrison finds it for you. This is the best choice for most sites.
  • Spider: Morrison starts from one or more seed URLs and follows internal links. Use it when your sitemap is missing or incomplete.
  • Manual list: paste the exact URLs you want indexed, one per line.

A website can have several data sources, for example one for the main sitemap and one for a list of landing pages. Click Add source on the Data Sources page to add one.

Rules

Each source can have rules that decide which pages are indexed, written as Page path followed by starts with, contains, ends with or is exactly. Use them to keep sections such as tag archives, internal search results or PDFs out of the index. Leave rules empty to include the whole site.

What Morrison stores for each page

  • Clean content: the main body text, without navigation, ads and boilerplate
  • Title and meta description
  • Canonical URL and Open Graph tags
  • Internal and external links
  • Structured data (JSON-LD), when present
  • HTTP status and crawl times

Morrison handles JavaScript-rendered pages and complex layouts. After a page is crawled, the content analyzer scores it.

Indexing

Every crawled page is indexed, which means it can be found by meaning, not just by exact words. This powers chat, content scans and workflows.

Each indexed page counts toward your plan's indexed pages for the month. Checks on a schedule only count pages that are new or changed. See Credits and usage.

Keeping pages up to date

Under Scheduled crawling on the Data Sources page, choose:

  • Look for new pages: Daily, Twice daily or Weekly
  • and recrawl all pages: Weekly, Bi-weekly or Monthly

Discovery finds new URLs. Recrawling refreshes the content of pages you already have. The page shows when the next discovery and next recrawl will run.

On every plan, Morrison can look for new pages daily. How often all pages are recrawled depends on your plan: monthly on Starter; weekly, bi-weekly or monthly on Pro and Business. Only pages that are new or have changed use your indexed-page allowance. See pricing.

Crawling now

Click the crawl button at the top of Data Sources to start a crawl right away. If the crawl would use more indexed pages than you have left, Morrison tells you before it starts and offers Crawl what we can.

The overview shows:

  • Indexed: how many pages are in the index
  • Last crawl and Next crawl
  • New pages found by discovery
  • View log: a log of what each crawl did

Crawl settings

Open Crawl settings for advanced options, such as extracting the main content only, including or excluding specific HTML tags, and how fast Morrison crawls your site.

Removing pages

Use Remove pages from index to rebuild your index or clean it up. You can remove only pages whose data source has been deleted, or every indexed page on the website. Removed pages and their content are deleted permanently.

Content history

When a page is recrawled and its content has changed, Morrison keeps the previous version. Open a page's History tab to see how it changed. This is useful for:

  • Tracking the impact of content updates
  • Spotting unintended changes after a release
  • Auditing content drift over time

Best practices

  • Start with a sitemap: it is the most reliable way to find your full page inventory
  • Skip the noise: add rules for tag archives, internal search and other thin sections
  • Match the schedule to your site: a news site needs frequent checks, a brochure site does not
  • Watch the log: the crawl log shows pages that failed and why