HearthIndex

The robot that builds Hearth search: an independent index of the web, run on Hearth nodes rather than by a big search company.

If you've seen this in your server logs, it was us:

Mozilla/5.0 (compatible; HearthIndex/1; +https://hearth.software/index/)

How it behaves

It obeys robots.txt for HearthIndex, or for * when there's no HearthIndex section. That includes Crawl-delay (up to 30 seconds).
It's slow and polite. One request at a time per site, at least a second apart, and no more than 50 pages per site. Crawled sites are refreshed weekly.
It respects page tags. Pages marked noindex aren't kept, and links marked nofollow aren't followed.
It reads public pages only. Https on the standard port, no forms, no sign-ins, no private addresses. It keeps the title, description and some text from each page.
It crawls the independent web. Small businesses, makers, clubs, open-source and decentralised projects. It leaves government, big media and big tech alone: they have Google already.
If robots.txt can't be read because it's refused or the server is erroring, it stays out until it can.

Keep your site out

Add this to your robots.txt. Your pages leave the index on the robot's next visit.

User-agent: HearthIndex
Disallow: /

To keep out a single page, use <meta name="robots" content="noindex">, or <meta name="hearthindex" content="noindex"> to hide it from Hearth only.

Put your site in, and rank higher

Sites that opt in rank above crawled ones, can list up to 500 pages, are refreshed daily, and can include content on IPFS. To opt in, serve /.well-known/hearth-index.json:

{
  "version": "hearth-index/1",
  "name": "Your business name",
  "description": "What you do, in a sentence"
}

Only the version line is required: the robot reads your sitemap for the rest. Sites built with Hearth CMS do this automatically when they go live.

Nodes check for themselves

Hearth nodes share only the names of sites with each other. Every node fetches and checks each site itself, so one node can't slip false results into another's index.

Questions or a problem

If the robot is misbehaving on your site, use the contact form on the Hearth home page and include your domain. Blocking it in robots.txt works straight away and needs no reply from us.