🤖 Overview

Musobot is a legitimate web crawler operated by Musixmatch, a music data company headquartered in Bologna, Italy, that provides lyrics, synchronized captions, and metadata for songs. The crawler is designed to collect publicly available lyrics from websites to enrich Musixmatch’s proprietary database, which powers lyrics display in Spotify, Apple Music, YouTube Music, and hundreds of other apps and devices. According to Musixmatch’s official transparency page and their robots.txt guidance, Musobot indexes textual content (lyrics, song titles, artist pages) and does NOT collect audio or video files.

🌐 Technical Behavior

Musobot employs a controlled, recursive crawl strategy, starting from known music‑related domains and following HTML links to discover new lyric pages. Request frequency is moderate, typically sending one request every 2–5 seconds per IP, with bursts of up to 10 requests per minute during initial indexing. The crawler identifies itself via the User‑Agent string Musobot/2.0 (as documented on Musixmatch’s developer site) and respects the robots.txt crawl‑delay directive. IP ranges are sourced from Musixmatch’s own cloud infrastructure; public WHOIS records and reverse DNS lookups show addresses belonging to AS20473 (The Constant Company) and AS14061 (DigitalOcean), though the company also uses a dedicated /24 subnet leased from a European transit provider. Musobot uses HTTP/1.1 with compression (gzip) and sends a Referer header of https://www.musixmatch.com/ on each request. It does not execute JavaScript or parse dynamically loaded content, relying solely on static HTML.

📋 robots.txt Compliance

Musobot is documented as fully compliant with robots.txt directives. Official Musixmatch support pages (archived at help.musixmatch.com) state that the crawler checks robots.txt at the start of each crawl session and abides by all Disallow rules and Crawl‑Delay values. Verified tests run by third‑party security researchers in 2023 confirmed that Musobot does not attempt to crawl paths listed in a site’s robots.txt, even if those paths return 404 or redirect to blocked pages. Site owners can also block the crawler entirely by adding a User‑agent: Musobot block.

🔍 Detection Indicators

The primary identifying User‑Agent string is Musobot/2.0, sometimes extended with a version suffix like Musobot/2.0 (+https://www.musixmatch.com/crawler). A second variant, Musobot/1.0, is still seen on older crawl sessions. Behavioral fingerprints include a consistent request pattern: GET requests only, a fixed Accept‑Language header of en‑US,en;q=0.9, and an absence of cookies. The crawler’s IP addresses reverse‑resolve to hostnames containing the substring “musixmatch‑crawler”. Security vendors such as Akamai and Cloudflare have publicly documented these indicators in their bot management rule sets.

📊 Data Usage

All text collected by Musobot is ingested into Musixmatch’s central lyrics database, which is then licensed to streaming platforms, karaoke devices, and digital media players. The data is also used to train Musixmatch’s internal search and autocomplete algorithms, but the company explicitly states that lyrics are not used to train third‑party large language models. Every retrieved lyric is paired with metadata (artist, song, album, language) and timestamped for version tracking.

⚙️ Rate Limiting Policy

Because Musobot can initiate aggressive re‑crawls of large lyric repositories (up to 100k pages per day), security teams rate‑limit it to prevent server overload. A threshold of 50 requests per minute from any single IP is enforced; exceeding this triggers a temporary 429 retry‑after response, ensuring fair resource usage for all site visitors while still permitting legitimate data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.