Crawler options 0.7.13+

📝 Name: crawlerOptions · 🖥️ Option: --crawler-options

Additional options for configurable crawlers.

INFO

These options only apply to configurable crawlers. If the configured crawler does not implement the required interface, a warning is shown.

Example

Pass crawler options in the expected input format.

IMPORTANT

When passing crawler options as command parameter or environment variable, make sure to pass them as JSON-encoded string.

CLI

./cache-warmup.phar --crawler-options '{"concurrency": 3, "request_options": {"delay": 3000}}'

JSON

{
    "crawlerOptions": {
        "concurrency": 3,
        "request_options": {
            "delay": 3000
        }
    }
}

PHP

use EliasHaeussler\CacheWarmup;

return static function (CacheWarmup\Config\CacheWarmupConfig $config) {
    $config->setCrawlerOption('concurrency', 3);
    $config->setCrawlerOption('request_options', [
        'delay' => 3000,
    ]);

    return $config;
};

YAML

crawlerOptions:
  concurrency: 3
  request_options:
    delay: 3000

.env

CACHE_WARMUP_CRAWLER_OPTIONS='{"concurrency": 3, "request_options": {"delay": 3000}}'

Option Reference

Both default crawlers are implemented as configurable crawlers:

• Crawler\ConcurrentCrawler
• Crawler\OutputtingCrawler

The following configuration options are currently available for both crawlers:

concurrency 0.7.13+

🎨 Type: integer · 🐝 Default: 3

Define how many URLs are crawled concurrently.

INFO

Internally, Guzzle's Pool feature is used to send multiple requests concurrently using asynchronous requests. You may also have a look at how this is implemented in the library's RequestPoolFactory.

CLI

./cache-warmup.phar --crawler-options '{"concurrency": 5}'

JSON

{
    "crawlerOptions": {
        "concurrency": 5
    }
}

PHP

use EliasHaeussler\CacheWarmup;

return static function (CacheWarmup\Config\CacheWarmupConfig $config) {
    $config->setCrawlerOption('concurrency', 5);

    return $config;
};

YAML

crawlerOptions:
  concurrency: 5

.env

CACHE_WARMUP_CRAWLER_OPTIONS='{"concurrency": 5}'

request_headers 0.7.13+

🎨 Type: array<string, mixed> · 🐝 Default: ['User-Agent' => '<default user-agent>']

A list of HTTP headers to send with each cache warmup request.

INFO

The default User-Agent is built in ConcurrentCrawlerTrait::getRequestHeaders().

CLI

./cache-warmup.phar --crawler-options '{"request_headers": {"X-Foo": "bar", "User-Agent": "Foo-Crawler/1.0"}}'

JSON

{
    "crawlerOptions": {
        "request_headers": {
            "X-Foo": "bar",
            "User-Agent": "Foo-Crawler/1.0"
        }
    }
}

PHP

use EliasHaeussler\CacheWarmup;

return static function (CacheWarmup\Config\CacheWarmupConfig $config) {
    $config->setCrawlerOption('request_headers', [
        'X-Foo' => 'bar',
        'User-Agent' => 'Foo-Crawler/1.0',
    ]);

    return $config;
};

YAML

crawlerOptions:
  request_headers:
    X-Foo: bar
    User-Agent: 'Foo-Crawler/1.0'

.env

CACHE_WARMUP_CRAWLER_OPTIONS='{"request_headers": {"X-Foo": "bar", "User-Agent": "Foo-Crawler/1.0"}}'

request_method 0.7.13+

🎨 Type: string · 🐝 Default: HEAD

The HTTP method used to perform cache warmup requests.

CLI

./cache-warmup.phar --crawler-options '{"request_method": "GET"}'

JSON

{
    "crawlerOptions": {
        "request_method": "GET"
    }
}

PHP

use EliasHaeussler\CacheWarmup;

return static function (CacheWarmup\Config\CacheWarmupConfig $config) {
    $config->setCrawlerOption('request_method', 'GET');

    return $config;
};

YAML

crawlerOptions:
  request_method: GET

.env

CACHE_WARMUP_CRAWLER_OPTIONS='{"request_method": "GET"}'

request_options 2.0+

🎨 Type: array<string, mixed> · 🐝 Default: []

Additional request options used for each cache warmup request.

CLI

./cache-warmup.phar --crawler-options '{"request_options": {"delay": 500, "timeout": 10}}'

JSON

{
    "crawlerOptions": {
        "request_options": {
            "delay": 500,
            "timeout": 10
        }
    }
}

PHP

use EliasHaeussler\CacheWarmup;

return static function (CacheWarmup\Config\CacheWarmupConfig $config) {
    $config->setCrawlerOption('request_options', [
        'delay' => 500,
        'timeout' => 10,
    ]);

    return $config;
};

YAML

crawlerOptions:
  request_options:
    delay: 500
    timeout: 10

.env

CACHE_WARMUP_CRAWLER_OPTIONS='{"request_options": {"delay": 500, "timeout": 10}}'

write_response_body 4.0+

🎨 Type: bool · 🐝 Default: false

Define whether or not to write response body of crawled URLs to the corresponding response object.

WARNING

Enabling this option may significantly increase memory consumption during cache warmup.

CLI

./cache-warmup.phar --crawler-options '{"write_response_body": true}'

JSON

{
    "crawlerOptions": {
        "write_response_body": true
    }
}

PHP

use EliasHaeussler\CacheWarmup;

return static function (CacheWarmup\Config\CacheWarmupConfig $config) {
    $config->setCrawlerOption('write_response_body', true);

    return $config;
};

YAML

crawlerOptions:
  write_response_body: true

.env

CACHE_WARMUP_CRAWLER_OPTIONS='{"write_response_body": true}'