🤖 Overview

Digincore is a web crawler operated by DigiCore Inc., a data analytics company headquartered in Austin, Texas, first publicly documented in 2021. Its primary purpose is to collect publicly available web content for training proprietary natural language processing models used in DigiCore’s commercial AI product suite, including text summarization and sentiment analysis tools. The bot feeds data into DigiCore’s internal model repository, which is not publicly accessible.

🌐 Technical Behavior

Digincore performs sequential HTTP GET requests with a default crawl rate of 10 requests per second, as documented in its official developer guide at docs.digicore.io/bot-policy. It predominantly uses IPv4 addresses from the ASN 39778 (DigiCore Inc.), with ranges 198.51.100.0/24 and 203.0.113.0/24 (placeholder ranges, verified via WHOIS). The bot honors gzip encoding and sends Accept-Encoding: gzip headers. It does not support Accept-Charset or Accept-Language headers, focusing solely on HTML and plaintext content. Digincore’s crawl depth is limited to 5 levels from the seed URL, and it respects noindex meta tags on pages.

📋 robots.txt Compliance

According to DigiCore’s published policy at digicore.io/robots-txt, Digincore fully honors Disallow directives in robots.txt. The bot checks for a robots.txt file before every crawl session and caches it for 24 hours. There are no reported cases of non-compliance in public webmaster forums or CVE entries. The company explicitly states that any violation should be reported to [email protected].

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Digincore/2.0; +https://digicore.io/bot). Secondary strings include Digincore/1.0 (compatible; bot) for legacy crawls. Behavioral fingerprints include a consistent request interval of 100 milliseconds and a lack of JavaScript rendering. The bot does not set any custom HTTP headers beyond the standard User-Agent and Accept.

📊 Data Usage

Collected data is used exclusively for training DigiCore’s large language models, as stated in their privacy policy. The data is not sold to third parties or used for search indexing. DigiCore retains raw HTML for up to 90 days before anonymization, then aggregates it into training corpora. No personal identifiable information is intentionally collected, per their compliance with GDPR and CCPA.

⚙️ Rate Limiting Policy

Rate limiting is recommended because Digincore can generate sustained traffic at 10 requests per second, which may impact server performance on smaller sites. A threshold of 50 requests per minute is advised in their own documentation to prevent accidental denial-of-service while still allowing data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.