metagloss
Metagloss is a web crawler operated by Metagloss Inc., a company that maintains a centralized repository of specialized glossaries, terminology databases, and domain-specific definition catalogs. Its primary purpose is to index publicly accessible glossary pages, technical documentation, and educational resources to feed into the Metagloss API and public website, which serve researchers, translators, and content managers seeking precise term definitions.
The crawler follows a breadth-first crawl strategy with a default delay of 3 seconds between requests, though it may burst up to 50 requests per minute on large sites. It uses HTTP/1.1 with persistent connections and respects the Cache-Control header to avoid stale content. The bot typically sources its IP ranges from a pool managed by Amazon Web Services (AWS) (ASN 16509) and occasionally from Google Cloud (ASN 15169), though the exact ranges are not publicly listed. It sends requests with the Accept-Encoding: gzip header and a custom User-Agent that includes a contact email for feedback. The crawl depth is limited to 10 hops by default, and it avoids binary file types like PDF and images unless explicitly included in the page markup.
According to the official Metagloss bot policy published at metagloss.com/bot-policy, the crawler fully honors Disallow directives in robots.txt and will not access any path or resource explicitly forbidden. It also supports the Crawl-Delay directive, adjusting its request interval to the specified number of seconds. There are no known violations or documented incidents of ignoring such rules.
The primary User-Agent string is Metagloss/1.0 (https://metagloss.com; [email protected]), with an alternative string MetaglossBot/1.0 observed in older logs. Behavioral fingerprints include sequential request patterns without randomized delays, and a consistently applied From email header. Some administrators also report a missing Referer header and an unusually low request-level variation in HTTP user-agent version numbers.
Collected glossary terms, definitions, and contextual usage examples are indexed, normalized, and cross-referenced within the Metagloss platform. The data powers the company’s AI‑assisted translation tool, API‑based term lookup for enterprise clients, and a public-facing search engine that returns curated definitions from multiple authoritative sources. No user‑generated content is permanently stored beyond the snippet required for term extraction.
Although Metaglass is a legitimate agent, it is rate‑limited to prevent excessive bandwidth consumption on shared web servers. A threshold of 10 requests per 30 seconds is recommended; exceeding this triggers a temporary block that still allows the bot to resume after a 60‑second cooldown, ensuring fair use for all crawlers.
🛡️
Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.
✅ Start Free ProtectionSetup takes under a minute · Free trial available
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.