eliashaeussler/typo3-warming

..  include:: /Includes.rst.txt

..  _crawlers:

========
Crawlers
========

The extension uses crawlers to visit all URLs configured for cache
warmup. While visiting a URL, the appropriate page cache gets warmed.
Learn more about which crawlers are available by default and how to
implement a custom crawler on this page.

..  php:namespace:: EliasHaeussler\CacheWarmup\Crawler

..  php:interface:: CrawlerInterface

    Interface for crawlers used to crawl and warm up URLs.

    ..  php:method:: crawl($urls)

        Crawl a given list of URLs.

        :param array $urls: List of URLs to be crawled.
        :returntype: EliasHaeussler\\CacheWarmup\\Result\\CacheWarmupResult

..  _default-crawlers:

Default crawlers
================

The extension ships with two default crawlers:

-   :php:class:`EliasHaeussler\\Typo3Warming\\Crawler\\ConcurrentUserAgentCrawler`:
    Used for cache warmup triggered within the **TYPO3 backend**
-   :php:class:`EliasHaeussler\\Typo3Warming\\Crawler\\OutputtingUserAgentCrawler`:
    Used for cache warmup executed from the **command-line**

Both crawlers use a custom `User-Agent` header for all warmup
requests. By using this custom header, it is possible to exclude
warmup requests from the statistics of analysis tools, for example.
The header is generated by a HMAC hash of the string
`TYPO3/tx_warming_crawler`.

The generated header value can be copied form the cache warmup modal
in the TYPO3 backend. Alternatively, a command `warming:showuseragent`
is available which can be used to read the current `User-Agent` header.

..  _implement-a-custom-crawler:

Implement a custom crawler
==========================

..  _available-interfaces:

Available interfaces
--------------------

The actual cache warmup is done via the library
`eliashaeussler/cache-warmup <https://github.com/eliashaeussler/cache-warmup>`__.
It provides the
:php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\CrawlerInterface`,
which must be implemented when developing your own crawler. There is
also a
:php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\VerboseCrawlerInterface`
that redirects user-oriented output to an instance of
:php:interface:`Symfony\\Component\\Console\\Output\\OutputInterface`.

..  _configurable-crawlers:

Configurable crawlers
---------------------

Custom crawlers can also implement the
:php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\ConfigurableCrawlerInterface`,
allowing users to configure warmup requests themselves.

..  seealso::

    `Feature #59 - Introduce configurable crawlers <https://github.com/eliashaeussler/cache-warmup/pull/59>`__
    of `eliashaeussler/cache-warmup` library

..  _logging-crawlers:

Logging crawlers
----------------

Crawling results can be logged using a dedicated PSR-3 logger. For this, crawlers
must implement :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\LoggingCrawlerInterface`
and inject an appropriate PSR-3 logger. In TYPO3 context, this is mostly done using
TYPO3's log manager. Read more about logging in the :ref:`official documentation <t3coreapi:logging>`.

..  seealso::

    `Feature #271 - Introduce support for PSR-3 loggers <https://github.com/eliashaeussler/cache-warmup/pull/271>`__
    of `eliashaeussler/cache-warmup` library

..  _stoppable-crawlers:

Stoppable crawlers
------------------

Crawlers implementing :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\StoppableCrawlerInterface`
may cancel a cache warmup prematurely if any crawling failure occurs. This can be
especially useful for validation purposes to check whether any page within an XML
sitemap is inaccessible or failing.

..  seealso::

    `Feature #302 - Introduce stoppable crawler and --stop-on-failure option <https://github.com/eliashaeussler/cache-warmup/pull/302>`__
    of `eliashaeussler/cache-warmup` library

..  _streamable-crawlers:

Streamable crawlers
-------------------

When running cache warmup from the TYPO3 backend, the current crawling progress is
streamed to the cache warmup progress modal. However, this is only supported for
crawlers implementing :php:interface:`EliasHaeussler\\Typo3Warming\\Crawler\\StreamableCrawler`.

Those crawlers will then get a :php:class:`EliasHaeussler\\SSE\\Stream\\EventStream`
injected. It can be used to send events to the current event stream. The following
events are currently available:

-   :php:class:`EliasHaeussler\\Typo3Warming\\Http\\Message\\Event\\WarmupFinishedEvent`
-   :php:class:`EliasHaeussler\\Typo3Warming\\Http\\Message\\Event\\WarmupProgressEvent`

By default, when implementing a streamable crawler, there's no need to trigger these
events by your own. Instead, it's better to use the provided
:php:class:`EliasHaeussler\\Typo3Warming\\Http\\Message\\Handler\\StreamResponseHandler`
which takes care of sending appropriate events.

..  _steps-to-implement-a-new-crawler:

Steps to implement a new crawler
--------------------------------

..  rst-class:: bignums

1.  Create a new crawler

    The new crawler must implement at least one of the following interfaces:

    -   :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\CrawlerInterface`
    -   :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\ConfigurableCrawlerInterface`
    -   :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\LoggingCrawlerInterface`
    -   :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\StoppableCrawlerInterface`
    -   :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\VerboseCrawlerInterface`
    -   :php:interface:`EliasHaeussler\\Typo3Warming\\Crawler\\StreamableCrawler`

2.  Configure the new crawler

    Add the new crawler to the :ref:`extension configuration <extension-configuration>`.
    Note that you should configure either the `crawler` or
    `verboseCrawler` option, depending on what interface you have
    implemented.

3.  Flush system caches

    Finally, flush all system caches to ensure the correct crawler
    class is used for further cache warmup requests.

..  seealso::
    View the sources on GitHub:

    -   `CrawlerInterface <https://github.com/eliashaeussler/cache-warmup/blob/main/src/Crawler/CrawlerInterface.php>`__
    -   `ConfigurableCrawlerInterface <https://github.com/eliashaeussler/cache-warmup/blob/main/src/Crawler/ConfigurableCrawlerInterface.php>`__
    -   `LoggingCrawlerInterface <https://github.com/eliashaeussler/cache-warmup/blob/main/src/Crawler/LoggingCrawlerInterface.php>`__
    -   `StoppableCrawlerInterface <https://github.com/eliashaeussler/cache-warmup/blob/main/src/Crawler/StoppableCrawlerInterface.php>`__
    -   `StreamableCrawler <https://github.com/eliashaeussler/typo3-warming/blob/main/Classes/Crawler/StreamableCrawler.php>`__
    -   `VerboseCrawlerInterface <https://github.com/eliashaeussler/cache-warmup/blob/main/src/Crawler/VerboseCrawlerInterface.php>`__
    -   `ConcurrentUserAgentCrawler <https://github.com/eliashaeussler/typo3-warming/blob/main/Classes/Crawler/ConcurrentUserAgentCrawler.php>`__
    -   `OutputtingUserAgentCrawler <https://github.com/eliashaeussler/typo3-warming/blob/main/Classes/Crawler/OutputtingUserAgentCrawler.php>`__
    -   `StreamResponseHandler <https://github.com/eliashaeussler/typo3-warming/blob/main/Classes/Http/Message/Handler/StreamResponseHandler.php>`__
    -   `WarmupFinishedEvent <https://github.com/eliashaeussler/typo3-warming/blob/main/Classes/Http/Message/Event/WarmupFinishedEvent.php>`__
    -   `WarmupProgressEvent <https://github.com/eliashaeussler/typo3-warming/blob/main/Classes/Http/Message/Event/WarmupProgressEvent.php>`__