Contao crawler leaks auth credentials to external hosts
Summary
Contao's crawler tries to prevent confidential HTTP client options from being sent to external domains by creating a scoped client: full options for root page origins, cleaned options for everything else. The cleaner removes Cookie and Authorization headers, but it removes the non-Symfony option names basic_auth and bearer_auth instead of Symfony HttpClient's real auth_basic and auth_bearer options.
When contao.crawl.default_http_client_options contains Basic or Bearer authentication for a protected staging/production site, those credentials remain in the "clean" client used for external links or configured additional URIs. An attacker who can get an external URL crawled, for example through a link on a crawled page while the broken-link checker is enabled, can receive the crawler credentials.
Technical Detail
Root Cause
```php
// core-bundle/src/Crawl/Escargot/Factory.php:175-209 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
private function createHttpClient(array $options = []): HttpClientInterface
{
$options = array_merge_recursive(
[
'headers' => [
'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'user-agent' => self::USER_AGENT,
],
'max_duration' => 10,
],
array_merge_recursive($this->getDefaultHttpClientOptions(), $options),
);
$cleanOptions = $this->cleanOptionsFromConfidentialData($options);
if ($options === $cleanOptions) {
return ($this->httpClientFactory)($options);
}
$scopedOptionsByRegex = [];
foreach ($this->getRootPageUriCollection()->all() as $rootPageUri) {
$scopedOptionsByRegex[preg_quote($this->getOriginFromUri($rootPageUri))] = $options;
}
return new ScopingHttpClient(($this->httpClientFactory)($cleanOptions), $scopedOptionsByRegex);
}
```
```php
// core-bundle/src/Crawl/Escargot/Factory.php:226-247 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
foreach ($options as $k => $v) {
if ('headers' === $k) {
foreach ($v as $header => $value) {
if (\in_array(strtolower($header), ['authorization', 'cookie'], true)) {
continue;
}
$cleanOptions['headers'][$header] = $value;
}
continue;
}
if ('basic_auth' === $k || 'bearer_auth' === $k) {
continue;
}
$cleanOptions[$k] = $v;
}
```
Symfony HttpClient authentication options are auth_basic and auth_bearer; Contao's own manual documents auth_basic for crawler Basic Authentication. Because the cleaner only strips basic_auth and bearer_auth, the "clean" default client for non-root-page hosts still carries the real auth options. The existing factory test intends to assert that Authorization is not sent to www.foreign-domain.com, but its mock client factory ignores the $defaultOptions argument, so it does not catch auth options that survive into HttpClient::create($cleanOptions).
Suggested Mitigation
Strip the actual Symfony HttpClient authentication option keys from the clean client. Include NTLM as a defensive extension because Symfony documents it as another auth option.
```diff
- if ('basic_auth' === $k || 'bearer_auth' === $k) {
+ if (\in_array($k, ['auth_basic', 'auth_bearer', 'auth_ntlm', 'basic_auth', 'bearer_auth'], true)) {
continue;
}
```
Also update the factory test so the mock factory records or preserves $defaultOptions; otherwise the test does not verify what HttpClient::create($cleanOptions) receives in production.
Impact
- Direct primitive: disclosure of crawler Basic/Bearer credentials to an external host reached by the crawler.
- Chain potential: if those credentials protect a staging or pre-publication environment, an attacker can use them to access that environment. The impact depends on what the leaked credential unlocks.
- Realistic exploitation: a content editor adds a link to
https://attacker.example/probe on a page that the crawler visits. When an administrator or scheduled maintenance run starts the broken-link checker with crawler Basic/Bearer authentication configured, the request to the attacker URL includes the generated Authorization header.
