🤖 Overview

odp entries t_st is a web crawler operated by the Open Directory Project (ODP), also known as DMOZ, a community-edited web directory that was one of the largest human-curated indexes of the Internet until its shutdown in March 2017. Its purpose was to periodically crawl submitted websites to verify their existence, collect metadata such as titles and descriptions, and update the directory listings. The bot fed data into the ODP directory, which was licensed to major search engines like Google and Yahoo for categorized search results.

🌐 Technical Behavior

The crawler typically operated from IP ranges associated with AOL and later Netscape Communications, as ODP was acquired by AOL. It used HTTP/1.1 requests with a default crawl interval of several seconds to avoid overloading servers, following links within submitted sites to a limited depth of usually no more than three levels. The bot respected the robots.txt file and used distinct User-Agent strings: Mozilla/5.0 (compatible; DMOZ/1.0; +http://www.dmoz.org/) or Mozilla/5.0 (compatible; ODP/1.0; +http://www.dmoz.org/). It did not index full page content, instead extracting only the title, meta description, and keyword tags from the HTML head section of each crawled page. The bot also validated that submitted URLs were accessible and returned a valid HTTP status code before adding them to the directory.

📋 robots.txt Compliance

Official documentation from the DMOZ website confirmed that the bot honored Disallow directives exactly as specified in the Robots Exclusion Protocol. Webmasters could block the crawler by adding User-agent: DMOZ or User-agent: ODP to their robots.txt file. There are no known incidents of this bot ignoring disallow rules, and its compliant behavior is cited in several webmaster resources from the early 2000s.

🔍 Detection Indicators

The primary User-Agent string was Mozilla/5.0 (compatible; DMOZ/1.0; +http://www.dmoz.org/), with a secondary variant Mozilla/5.0 (compatible; ODP/1.0; +http://www.dmoz.org/). The bot also included a From header in some implementations, set to [email protected]. Its request pattern was characterized by a low frequency (one request every 10–30 seconds) and a focus on URLs submitted through the ODP submission form or linked from already-indexed pages. It did not crawl randomly across the web, making its traffic easily distinguishable from broader search engine crawlers.

📊 Data Usage

The collected metadata—URLs, titles, descriptions, and assigned categories—was used exclusively for the Open Directory Project’s human-edited directory. This directory was then licensed to third-party search engines and portals to enhance their categorized search results. No personal data or full-page content was stored; the bot only extracted meta-information and validated site availability to maintain the directory’s accuracy. After DMOZ shut down in 2017, no further data collection occurred, and the bot is considered defunct.

⚙️ Rate Limiting Policy

Rate limiting for the ODP crawler is historically irrelevant as the service has been discontinued. When active, administrators were advised to apply moderate rate limits to prevent resource exhaustion, though the bot’s built-in politeness made aggressive throttling unnecessary. Any modern detection of this bot should be treated as a legacy artifact—it no longer operates on the live internet.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.