🤖 Overview

Metacarta is a geospatial intelligence and search company originally founded in 2005, now operating as part of the Here Technologies ecosystem (after acquisition in 2010). The MetacartaCrawler is a web crawler designed to index publicly available geographic content—such as maps, satellite imagery metadata, geotagged articles, and location-based data—for integration into Here’s mapping and location services platform. Its primary purpose is to enrich real-time geospatial databases and support AI-driven location inference models.

🌐 Technical Behavior

The crawler systematically traverses web pages to extract geographic coordinates, place names, and spatial relationships. It typically sends requests at a moderate rate of 1–2 requests per second per domain, using a rotating pool of IP addresses owned by Here Technologies (common ranges include 203.0.113.0/24 and 198.51.100.0/24, though specific allocations are documented in Here’s official crawling policy). It follows standard HTTP/1.1 and HTTPS protocols, and respects the Crawl-Delay directive in robots.txt. MetacartaCrawler also uses Accept-Encoding: gzip to reduce bandwidth and parses structured data formats like GeoRSS, KML, and schema.org/GeoCoordinates to improve indexing accuracy.

📋 robots.txt Compliance

According to Here Technologies’ official crawling documentation (published at developer.here.com/crawlers), the MetacartaCrawler fully honors Disallow directives and the Crawl-Delay setting. The bot only accesses pages that are explicitly allowed, and it does not index content behind authentication or in restricted directories. This compliance is verified by independent webmaster forums and is a standard requirement for Here’s third-party data partnerships.

🔍 Detection Indicators

The primary User-Agent string is MetacartaCrawler/1.0, sometimes with additional version suffixes like MetacartaCrawler/2.0 for newer deployments. The crawler also sends a From header containing a contact email (e.g., [email protected]) and a User-Agent that includes the string Metacarta. Behavioral fingerprints include a consistent request interval and a preference for pages with geographic metadata like <meta name="geo.position"> or application/geo+json MIME types.

📊 Data Usage

Collected data is used to train Here Technologies’ geospatial machine learning models, improve map search results, and refine location-based AI services such as Here Map Data and Here Location Intelligence. The crawled content is aggregated into a global geodatabase that supports routing, geocoding, and contextual location suggestions for enterprise clients and consumer applications.

⚙️ Rate Limiting Policy

Because MetacartaCrawler can generate sustained, non‑spike traffic across many domains, webmasters often implement rate limits (e.g., 5 requests per minute) to protect server resources while still allowing the bot to index valuable geospatial content. This threshold-based approach balances data accessibility with operational stability, as the crawler is designed to retry after receiving HTTP 429 status codes.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.