VIPnytt/RobotsTxtParser

# Overview
- [TxtClient](TxtClient.md)
    - [Clean-param](CleanParamClient.md)
    - [Host](HostClient.md)
    - [Sitemap](SitemapClient.md)
    - [UserAgent](UserAgentClient.md)
      - [Allow](AllowClient.md)
      - [Cache-delay](DelayClient.md)
      - [Comment](CommentClient.md)
      - [Crawl-delay](DelayClient.md)
      - [Disallow](AllowClient.md)
      - [NoIndex](AllowClient.md)
      - [Request-rate](RequestRateClient.md)
      - [Robot-version](RobotVersionClient.md)
      - [Visit-time](VisitTimeClient.md)

__Other available classes:__

- [Cache manage](CacheManageInterface.md)
  - [Cache base](CacheBaseInterface.md)
- [Delay manage](DelayManageInterface.md)
  - [Delay base](DelayBaseInterface.md)
- [Import](Import.md)
- [RenderClient](RenderClient.md)
- [UriClient](UriClient.md)