🤖 Overview

Goodjelly is a web crawler operated by GoodJelly Inc., a data aggregation company founded in 2022. Its primary purpose is to systematically index publicly accessible web content to train and improve GoodJelly's generative AI models, specifically their JellyGPT family of large language models.

🌐 Technical Behavior

Goodjelly performs asynchronous crawling using a distributed fleet of headless Chromium instances, typically scaling to multiple concurrent requests per domain. It adheres to a crawl delay of 5 seconds between requests by default, but may increase to 10 seconds if server response times exceed 2 seconds. The bot primarily crawls via HTTPS on ports 80 and 443, following HTTP redirects up to 5 hops and supporting gzip and deflate content encoding. It respects 429 Too Many Requests responses by backing off for at least 60 seconds before retrying. IP address ranges are published in the official Goodjelly documentation: 104.16.0.0/12 and 35.190.0.0/16, sourced from Cloudflare and Google Cloud respectively. Crawl sessions are identified by a unique session ID in the X-Goodjelly-Session header. The bot uses a rotating user-agent string and may present as Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Goodjelly for JavaScript rendering.

📋 robots.txt Compliance

According to GoodJelly's official documentation at https://docs.goodjelly.com/crawler-policy, Goodjelly fully respects the Robots Exclusion Standard. It reads and caches robots.txt for up to 24 hours before each crawl session. The bot will cease crawling any path with a Disallow directive and will also respect Crawl-Delay directives.

🔍 Detection Indicators

The standard User-Agent string is Goodjelly/1.0 (compatible; GoodjellyBot; +https://goodjelly.com/bot). Behavioral fingerprints include a consistent request pattern with a custom header X-Goodjelly-RequestId and a referrer of https://goodjelly.com/. The bot also sets a cookie gid for session tracking and may present a rotated IP from the published ranges. In some cases, the User-Agent may appear as Mozilla/5.0 Goodjelly/2.0 when JavaScript rendering is required. The official IP range JSON is available at https://goodjelly.com/ip-ranges.json.

📊 Data Usage

Collected data is used exclusively for training GoodJelly's AI models, including text generation, summarization, and code synthesis. GoodJelly also uses the crawled data to improve its proprietary search engine, JellySearch, and to train a recommendation algorithm called JellyRank. GoodJelly publishes an opt-out page at https://goodjelly.com/optout where site owners can request their content be excluded from future training sets. The company claims not to store raw content longer than 90 days for compliance with GDPR and CCPA.

⚙️ Rate Limiting Policy

While Goodjelly is a legitimate bot, its aggressive concurrent crawling can impact server resources. Rate limiting is recommended at thresholds of 100 requests per minute per IP, or by implementing a 5-second crawl delay via robots.txt. GoodJelly does not bypass blocks; they recommend rate limiting as a standard operational practice. Administrators should consider returning a 429 status code to enforce limits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.