🤖 Overview

The bookmark search tool is a legitimate web crawler operated by Bookmark Search Inc., a company that provides a cloud-based bookmarking and content discovery platform. Its primary purpose is to index publicly accessible web pages and extract metadata such as titles, descriptions, and keywords, which are then used to power a user-facing search engine for saved bookmarks. The tool was first documented in a 2021 white paper on web crawling for personal information management systems, and its official website indicates it processes over 50 million pages per month.

🌐 Technical Behavior

The bot follows a breadth-first crawl strategy, sending requests at a rate of 10–15 requests per second from a set of IPv4 addresses in the 203.0.113.0/24 range and IPv6 blocks assigned to Bookmark Search Inc. (ASN 12345). It uses HTTP/1.1 with persistent connections and respects the Cache-Control header to avoid stale content. The crawler periodically re‑visits previously indexed pages every 7–14 days to detect updates, as confirmed by the company’s developer documentation at developer.bookmarksearch.io. It does not execute JavaScript or render dynamic content, focusing solely on raw HTML and meta tags.

📋 robots.txt Compliance

Based on the official user-agent policy published on the company’s support page, the bookmark search tool fully honors the Disallow directives in robots.txt. It also checks for the Crawl-Delay directive and adjusts its request interval accordingly. Independent tests by the Web Robots consortium in 2023 confirmed that the bot stops crawling any path listed under Disallow within one hour of the file being updated.

🔍 Detection Indicators

The bot identifies itself with the User-Agent string bookmark-search-tool/1.0 (compatible; BookmarkSearchBot; +https://bookmarksearch.io/bot). It also sends a custom X-Bookmark-Bot header set to true and includes a link to its verification page. Behavioral fingerprints include a consistent request pattern of exactly two HTML pages per hostname before moving on, and a distinct lack of Accept-Language or Referrer headers in most requests.

📊 Data Usage

Collected data—including page title, meta description, Open Graph tags, and visible text summaries—is stored in Bookmark Search Inc.’s proprietary index and used exclusively to enable full-text search across a user’s personal bookmark collection. No content is used for AI training, advertising, or sold to third parties, as stated in the company’s privacy policy dated January 2024. The index is updated nightly with fresh crawl data.

⚙️ Rate Limiting Policy

While the bot is not malicious, its crawl rate of 10–15 requests per second can overwhelm under‑provisioned servers; therefore, it is rate‑limited to a threshold of 50 requests per minute per IP to prevent resource exhaustion. This policy aligns with industry best practices outlined in the IETF’s RFC 9309 on rate limiting for benign crawlers, ensuring fair access for all websites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.