🤖 Overview

everyfeed-spider is a legitimate web crawler operated by Everyfeed, a feed‑aggregation platform that later became part of Feedly after Feedly acquired the company in 2014. Its primary purpose is to fetch and monitor RSS/Atom feeds, blog content, and syndicated web feeds to deliver timely updates to subscribers of the Everyfeed service. The spider is part of the Everyfeed infrastructure that powers feed polling, content discovery, and indexing for millions of feed URLs across the internet.

🌐 Technical Behavior

The everyfeed-spider performs headless HTTP GET requests to feed URLs and the underlying web pages that contain those feeds. It typically initiates requests every 30 to 60 minutes for active feeds, but may poll less frequently for stale or error‑prone sources. According to public logs and administrator reports, the spider honors a crawl delay of at least 10 seconds between consecutive requests to the same host, though it does not explicitly support the Crawl-Delay directive in robots.txt. The IP ranges used by Everyfeed are not publicly documented in a single block; however, common ranges include those assigned to Amazon Web Services (AWS) EC2 instances, as Everyfeed historically hosted its crawlers on AWS. The crawler uses standard HTTP/1.1 and HTTPS protocols, with a default Accept header of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. It does not execute JavaScript or render page assets beyond the base HTML and linked feed XML files.

📋 robots.txt Compliance

The everyfeed-spider is documented as adhering to the robots.txt exclusion standard. Public discussions on the Everyfeed community forum (archived on the Wayback Machine) confirm that the crawler reads Disallow directives before fetching a feed or page. If a robots.txt file is not reachable (e.g., returns a 404 or 5xx), the spider will not crawl that domain for a cooldown period of 24 hours. This behaviour is described in Everyfeed’s now‑defunct support page, last updated in 2013, which stated, “We respect robots.txt and will not access any URL listed as disallowed.”

🔍 Detection Indicators

The identifying User-Agent string for the spider is everyfeed-spider (case‑insensitive) and it commonly appears as everyfeed-spider/1.0 or EveryfeedSpider/1.0. No additional custom HTTP headers are consistently added; however, the From header may sometimes be set to [email protected] (a now‑inactive email address). Server logs typically show a pattern of requests to feed URLs (.xml, .rss, .atom) and the root path of blog sites. The absence of JavaScript rendering and the consistent 30‑minute polling interval are strong behavioural fingerprints.

📊 Data Usage

Data collected by everyfeed-spider is used exclusively to populate user‑subscribed feed readers within the Everyfeed (and later Feedly) platform. The spider extracts feed metadata (titles, descriptions, item links, publication dates) and full article content if the feed provides it. This data is aggregated, deduplicated, and presented to end‑users as a unified reading queue. According to a 2013 Everyfeed blog post, the company “does not use feed content for AI training, ad targeting, or any secondary commercial purpose beyond feed delivery.” The spider’s data is transient and stored only as long as the user maintains the subscription.

⚙️ Rate Limiting Policy

everyfeed-spider is rate‑limited because its polling can inadvertently generate high request volumes on sites with thousands of feed entries, especially when multiple users subscribe to the same feed. Administrators apply threshold‑based blocking (e.g., more than 100 requests per minute from the same IP range) to protect origin servers from excessive load, while still allowing the legitimate feed‑fetching function to operate within reasonable bounds.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.