🤖 Overview

twinuffbot is a legitimate educational web crawler operated by Twinkl, a United Kingdom–based publisher of teaching resources founded in 2010, first publicly identified in January 2023 through server logs and discussions on the WebmasterWorld forum. Its purpose is to systematically index publicly available educational content—lesson plans, worksheets, and curriculum guides—to populate Twinkl’s internal recommendation engine and resource aggregation platform, as stated on Twinkl’s official bot information page at https://www.twinkl.com/bot.

🌐 Technical Behavior

twinuffbot uses a breadth‑first crawling strategy with a default request delay of five to ten seconds, though community reports on Stack Overflow indicate the bot sometimes sends bursts of 20–30 requests within a minute when encountering a new domain. It communicates via HTTP/1.1 with persistent connections and respects 429 Too Many Requests responses by temporarily backing off. The bot originates from IP addresses within the Twinkl‑owned ASN (AS396982), though exact ranges are not publicly documented; it fetches only HTML pages and ignores images, PDFs, and other binary files unless they are directly linked in crawled HTML. It does not support HTTPS‑only sites unless the certificate is valid, and it follows redirects up to three hops.

📋 robots.txt Compliance

Twinkl’s official policy (published at https://www.twinkl.com/robots.txt) states that twinuffbot fully honours Disallow directives and Crawl‑Delay instructions. However, anecdotal evidence from webmasters on the Google Webmasters Help forums notes that the bot occasionally ignores Crawl‑Delay values below two seconds, suggesting a practical lower bound. Twinkl recommends using the Disallow: / directive to entirely block the bot if needed.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; twinuffbot/1.0; +https://www.twinkl.com/bot) — note the lowercase “twinuffbot” variant often appears in logs. No additional identifying HTTP headers (e.g., From or X‑Robot‑Tags) are consistently sent, making server‑side detection reliant solely on the User‑Agent field. The bot does not spoof its identity; it always carries the documented Twinkl URL in the comment field.

📊 Data Usage

Collected data is used internally by Twinkl to improve search relevance within its platform, generate metadata for resource discovery, and train content‑matching algorithms for teachers and students. The data is not licensed to third parties nor used to train general‑purpose large language models, according to Twinkl’s privacy policy archived at https://www.twinkl.com/privacy. Twinkl also uses the crawl results to audit the availability of linked external resources.

⚙️ Rate Limiting Policy

Because twinuffbot can send up to 30 requests in a short burst during initial scans—exceeding its declared delay—webmasters are advised to enforce rate limiting with a per‑IP threshold of 100 requests per minute. This policy prevents server strain while still allowing the bot’s legitimate indexing activity, aligning with the broad principle of protecting origin servers from unintended load without completely blocking a well‑intentioned crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.