IndexEcoBot

IndexEcoBot is the web crawler (also called a spider) operated by IndexEco.org, an experimental search engine and specialized index. It discovers and indexes publicly accessible pages so that people can find them — and so your pages get real visitors back. This page documents how the crawler identifies itself, how it behaves on your servers, and every way you can control it.

In short: IndexEcoBot identifies itself with a stable user-agent, follows robots.txt, crawls politely with per-host rate limiting, fetches public content only, and can be verified against our published IP list.

Identification

IndexEcoBot never hides what it is. Every request carries a stable user-agent string that contains the URL of this page, so you can always find this documentation from your access logs:

Mozilla/5.0 (compatible; IndexEcoBot/1.0; +https://indexeco.org/searchbot)

The product token to use in your robots.txt is IndexEcoBot.

User-agent strings can be spoofed by anyone, so don't trust the string alone — see Verifying IndexEcoBot below.

At a glance

Item	Value
Operator	IndexEco — indexeco.org
Purpose	Building the index of IndexEco, an experimental search engine and specialized index
Bot type	Web crawler (self-identifying)
Version	1.0
robots.txt token	`IndexEcoBot`
Follows robots.txt	Yes — checked before every fetch
Honors Crawl-delay	Yes
Rate limiting	Per host
HTTP compression	Yes — gzip
Executes JavaScript	No
Submits forms	No
Bypasses logins, paywalls or CAPTCHAs	Never
Verification	Check the source IP against our published IP list (JSON)

How IndexEcoBot behaves

Respects robots.txt

Your rules are checked before every fetch. Disallow and Crawl-delay directives are honored.

Crawls politely

Rate-limited per host to keep the load on your servers negligible.

Public content only

No bypassing of logins, paywalls or CAPTCHAs, and no form submissions.

Light on your servers

Supports gzip compression and executes no scripts.

Frequently asked questions

What is IndexEcoBot doing on my website?

IndexEcoBot follows links found on public pages in order to discover content for the IndexEco index. Any page that is publicly cited somewhere on the web may therefore be visited — including pages you never explicitly submitted. Pages that are indexed become discoverable through IndexEco.org, which sends visitors back to the original page.

Does IndexEcoBot respect my robots.txt file?

Yes. The file is checked before every fetch, and Disallow and Crawl-delay directives are honored. If you believe IndexEcoBot is not respecting your directives, contact us and we will investigate immediately.

Controlling IndexEcoBot

robots.txt gives you full control over what IndexEcoBot may crawl. The crawler looks for a rules group addressed to IndexEcoBot first; if you haven't defined one, it falls back to your general * rules.

Block IndexEcoBot from your entire site

User-agent: IndexEcoBot
Disallow: /

Keep it out of part of your site

User-agent: IndexEcoBot
Disallow: /private/
Disallow: /admin/

Slow IndexEcoBot down

User-agent: IndexEcoBot
Crawl-delay: 10

The crawl delay applies per host: if a single server hosts several domains, each domain gets its own budget. If you need a stronger guarantee — for example a lower rate across a whole data center — contact us.

Verifying IndexEcoBot

Anyone can put our user-agent string in a request. To tell a genuine IndexEcoBot request from an impostor, check that the source IP address appears in the list of IP addresses IndexEcoBot crawls from, published as JSON:

https://indexeco.org/searchbot/ip-ranges.json

The list may change over time — for example when we add crawler machines — so fetch it periodically rather than copying it once. If the source IP address is not on the list, the request is not ours: someone is impersonating our user-agent, and you can block it safely.

Your site blocks IndexEcoBot — how to fix it

Even though IndexEcoBot crawls politely, security plugins, firewalls or rate limiters sometimes mistake it for a DDoS or brute-force attack and blacklist it temporarily. If that happens:

Allow IndexEcoBot in your firewall or security plugin, using the user-agent or the IPs from our published list.
If you can't — or don't want to — contact us and we will slow down or stop crawling your site. No justification is required on your side.

Contact

For anything concerning IndexEcoBot — questions, requests to slow down, or to stop crawling your site — write to us at the address below.

A human reads every message. To help us act quickly, please include: the domain concerned, the source IP addresses you observed, and a few log lines showing the requests.

If you want IndexEcoBot to slow down, stay away from part of your site, or stop crawling you altogether, just say so — we will apply it.