🤖 Overview

Attributor is a web crawler operated by Attributor Corporation, now a subsidiary of Digimarc, used for automated content monitoring and plagiarism detection. The bot was first deployed in 2006 to scan publicly accessible web pages to identify unauthorized reproduction of copyrighted materials for publishers and content creators.

🌐 Technical Behavior

The Attributor crawler performs deep, recursive crawling of web domains, typically making requests at moderate rates of 1-3 requests per second per IP. It uses HTTP/1.1 and respects standard web protocols, with a known IP range associated with Digimarc’s infrastructure (e.g., 64.34.XX.XX). The bot identifies itself via the User-Agent string “Attributor/1.0” and sends a custom header “X-Robots-Tag: noindex” when indexing content. According to official documentation, the crawler supports both HTTP and HTTPS and follows links, but does not execute JavaScript.

📋 robots.txt Compliance

Attributor’s crawler is documented to fully respect robots.txt directives, including Disallow rules. The official Attributor FAQ (archived from attributor.com) states that site owners can block the bot entirely by adding “User-agent: Attributor Disallow: /” to their robots.txt file. There is no evidence of deliberate non-compliance.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string “Attributor/1.0” or “Attributor”. Additional fingerprints include a consistent request pattern with a short crawl delay (default 1 second) and a reverse DNS hostname ending in .attributor.com. The bot does not include a Referer header. Server logs may show repeated requests to the same URL over several days as the bot rechecks for updated content.

📊 Data Usage

Collected data is used for content attribution monitoring — comparing crawled web pages against a database of registered copyrighted works to detect unauthorized usage. The system generates reports for publishers highlighting where their content appears without permission. Attributor also provides analytics on content distribution across the web.

⚙️ Rate Limiting Policy

Although legitimate, the Attributor crawler can be aggressive when monitoring large sites, potentially impacting server resources. Rate limiting is recommended to prevent excessive load, with a threshold-based policy that blocks requests exceeding 10 queries per second from a single IP. This ensures site performance while still allowing the bot to fulfill its monitoring role.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.