source/RobotsTxtParser.php
Showing 527 of 527 total issues
File RobotsTxtParser.php has 353 lines of code (exceeds 250 allowed). Consider refactoring. Open
Open
<?php declare(strict_types=1); namespace t1gor\RobotsTxtParser; use Psr\Log\LoggerAwareInterface;The class RobotsTxtParser has an overall complexity of 103 which is very high. The configured complexity threshold is 50. Open
Open
class RobotsTxtParser implements LoggerAwareInterface { use LogsIfAvailableTrait; // default encoding- Exclude checks
RobotsTxtParser has 27 functions (exceeds 20 allowed). Consider refactoring. Open
Open
class RobotsTxtParser implements LoggerAwareInterface { use LogsIfAvailableTrait; // default encodingFunction parseURL has a Cognitive Complexity of 15 (exceeds 5 allowed). Consider refactoring. Open
Open
protected function parseURL($url) { $parsed = parse_url($url); if ($parsed === false) { return false; } elseif (!isset($parsed['scheme']) || !$this->isValidScheme($parsed['scheme'])) {- Read upRead up
Function render has a Cognitive Complexity of 13 (exceeds 5 allowed). Consider refactoring. Open
Open
public function render($eol = "\r\n") { $input = $this->getRules(); krsort($input); $output = []; foreach ($input as $userAgent => $rules) {- Read upRead up
Method render has 30 lines of code (exceeds 25 allowed). Consider refactoring. Open
Open
public function render($eol = "\r\n") { $input = $this->getRules(); krsort($input); $output = []; foreach ($input as $userAgent => $rules) {Function getSitemaps has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring. Open
Open
public function getSitemaps(?string $userAgent = null): array { $this->buildTree(); $maps = []; if (!is_null($userAgent)) {- Read upRead up
Function checkRules has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring. Open
Open
protected function checkRules(string $rule, string $path, string $userAgent = '*'): bool { // check for disallowed http status code if ($this->checkHttpStatusCodeRule()) { return ($rule === Directive::DISALLOW); }- Read upRead up
Method __construct has 5 arguments (exceeds 4 allowed). Consider refactoring. Open
Open
$content, string $encoding = self::DEFAULT_ENCODING, ?TreeBuilderInterface $treeBuilder = null, ?ReaderInterface $reader = null, ?UserAgentMatcherInterface $userAgentMatcher = nullFunction getHost has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring. Open
Open
public function getHost(?string $userAgent = null) { $this->buildTree(); if (!is_null($userAgent)) { $userAgent = $this->userAgentMatcher->getMatching($userAgent, array_keys($this->tree));- Read upRead up
Avoid too many return statements within this method. Open
Open
return $parsed;The method parseURL() has a Cyclomatic Complexity of 10. The configured cyclomatic complexity threshold is 10. Open
Open
protected function parseURL($url) { $parsed = parse_url($url); if ($parsed === false) { return false; } elseif (!isset($parsed['scheme']) || !$this->isValidScheme($parsed['scheme'])) {- Read upRead up
- Exclude checks
The class RobotsTxtParser has a coupling between objects value of 15. Consider to reduce the number of dependencies under 13. Open
Open
class RobotsTxtParser implements LoggerAwareInterface { use LogsIfAvailableTrait; // default encoding- Read upRead up
- Exclude checks
Missing class import via use statement (line '236', column '13'). Open
Open
throw new \RuntimeException(WarmingMessages::SET_UA_DEPRECATED);- Read upRead up
- Exclude checks
Avoid using static access to class '\t1gor\RobotsTxtParser\Parser\HostName' in method 'isValidHostName'. Open
Open
return HostName::isValid($host);- Read upRead up
- Exclude checks
The method parseURL uses an else expression. Else clauses are basically not necessary and you can simplify the code by not using them. Open
Open
} else { if (!isset($parsed['host']) || !$this->isValidHostName($parsed['host'])) { return false; } else { if (!isset($parsed['port'])) {- Read upRead up
- Exclude checks
Avoid using static access to class 't1gor\RobotsTxtParser\Directive' in method 'checkRuleSwitch'. Open
Open
switch (Directive::attemptGetInline($rule)) {- Read upRead up
- Exclude checks
The method parseURL uses an else expression. Else clauses are basically not necessary and you can simplify the code by not using them. Open
Open
} else { if (!isset($parsed['port'])) { $parsed['port'] = getservbyname($parsed['scheme'], 'tcp'); if (!is_int($parsed['port'])) { return false;- Read upRead up
- Exclude checks
Avoid using static access to class 't1gor\RobotsTxtParser\Directive' in method 'checkRuleSwitch'. Open
Open
if ($this->checkCleanParamRule(Directive::stripInline($rule), $path)) {- Read upRead up
- Exclude checks
Avoid using static access to class 't1gor\RobotsTxtParser\Directive' in method 'checkRuleSwitch'. Open
Open
if ($this->checkHostRule(Directive::stripInline($rule))) {- Read upRead up
- Exclude checks
Avoid using static access to class '\t1gor\RobotsTxtParser\Stream\GeneratorBasedReader' in method '__construct'. Open
Open
? GeneratorBasedReader::fromStream($content)- Read upRead up
- Exclude checks
The method getSitemaps uses an else expression. Else clauses are basically not necessary and you can simplify the code by not using them. Open
Open
} else { foreach ($this->tree as $userAgentBased) { if (isset($userAgentBased[Directive::SITEMAP]) && !empty($userAgentBased[Directive::SITEMAP])) { $maps = array_merge($maps, $userAgentBased[Directive::SITEMAP]); }- Read upRead up
- Exclude checks
The method render uses an else expression. Else clauses are basically not necessary and you can simplify the code by not using them. Open
Open
} else { $output[] = $directive . ': ' . $value; }- Read upRead up
- Exclude checks
Avoid using static access to class '\t1gor\RobotsTxtParser\Parser\DirectiveProcessorsFactory' in method 'buildTree'. Open
Open
DirectiveProcessorsFactory::getDefault($this->logger),- Read upRead up
- Exclude checks
Avoid using static access to class '\t1gor\RobotsTxtParser\Stream\GeneratorBasedReader' in method '__construct'. Open
Open
: GeneratorBasedReader::fromString($content);- Read upRead up
- Exclude checks
Avoid using static access to class '\t1gor\RobotsTxtParser\Parser\Url' in method 'isValidScheme'. Open
Open
return Url::isValidScheme($scheme);- Read upRead up
- Exclude checks
Avoid unused private fields such as '$userAgent'. Open
Open
private $userAgent = '*';- Read upRead up
- Exclude checks
Avoid unused private fields such as '$content'. Open
Open
private $content = '';- Read upRead up
- Exclude checks
Avoid unused parameters such as '$userAgent'. Open
Open
public function setUserAgent(string $userAgent) {- Read upRead up
- Exclude checks
Avoid variables with short names like $b. Configured minimum length is 3. Open
Open
usort($value, function ($a, $b) {- Read upRead up
- Exclude checks
Avoid variables with short names like $a. Configured minimum length is 3. Open
Open
usort($value, function ($a, $b) {- Read upRead up
- Exclude checks
Blank line found at start of control structure Open
Open
switch (Directive::attemptGetInline($rule)) {- Exclude checks
Opening brace of a class must be on the line after the definition Open
Open
class RobotsTxtParser implements LoggerAwareInterface {- Exclude checks
Scope keyword "private" must be followed by a single space Open
Open
private $content = '';- Exclude checks
CASE statements must be defined using a colon Open
Open
case Directive::HOST;- Exclude checks
Only one argument is allowed per line in a multi-line function call Open
Open
$host, [- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private string $encoding = '';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
?TreeBuilderInterface $treeBuilder = null,- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($this->encoding !== static::DEFAULT_ENCODING) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
DirectiveProcessorsFactory::getDefault($this->logger),- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->logger;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($this->reader instanceof LoggerAwareInterface) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!isset($parsed['host']) || !$this->isValidHostName($parsed['host'])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!isset($parsed['port'])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $parsed;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private function explodeCleanParamRule($rule) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// strip any invalid characters from path prefix- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param int $code- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// Check each directive for rules, allowed by default- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function __construct(- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->treeBuilder = $treeBuilder;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->reader = is_resource($content)- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
: GeneratorBasedReader::fromString($content);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->logger- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->reader->setLogger($this->logger);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
} elseif (!isset($parsed['scheme']) || !$this->isValidScheme($parsed['scheme'])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!is_int($code) || $code < 100 || $code > 599) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return true;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// robots.txt file content- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->userAgentMatcher = new UserAgentMatcher();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->reader->setEncoding($this->encoding);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// construct a tree builder if not passed- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->logger = $logger;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($parsed === false) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Explode Clean-Param rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return bool- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function setHttpStatusCode(int $code): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->buildTree();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
protected function checkRules(string $rule, string $path, string $userAgent = '*'): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$userAgent = $this->userAgentMatcher->getMatching($userAgent, array_keys($this->tree));- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private function checkHttpStatusCodeRule(): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
use LogsIfAvailableTrait;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private $userAgent = '*';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->treeBuilder = new TreeBuilder(- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$parsed['port'] = getservbyname($parsed['scheme'], 'tcp');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// check for disallowed http status code- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$content,- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
?ReaderInterface $reader = null,- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log('UserAgentMatcher is not passed, using a default one...');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
protected function checkRuleSwitch(string $rule, string $path): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function setLogger(LoggerInterface $logger): void {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return true;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private static function isValidHostName(string $host): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->httpStatusCode = $code;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $userAgent - which robot to check for- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
protected $rules = [];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// robots.txt http status code- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private $url = null;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
string $encoding = self::DEFAULT_ENCODING,- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
?UserAgentMatcherInterface $userAgentMatcher = null- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->encoding = $encoding;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getLogger(): ?LoggerInterface {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return Url::isValidScheme($scheme);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!is_int($parsed['port'])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$cleanParam['path'] = isset($array[1]) ? $this->encode_url(preg_replace('/[^A-Za-z0-9\.-\/\*\_]/', '', $array[1])) : '/*';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$cleanParam['param'][] = trim($key);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->checkRules(Directive::ALLOW, $url->getPath(), $userAgent);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// rule match- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
protected ?int $httpStatusCode;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->reader = $reader;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// split into parameter and path- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function isAllowed(string $url, ?string $userAgent = '*'): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return void- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return ($rule === Directive::DISALLOW);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$result = ($rule === Directive::ALLOW);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ([Directive::DISALLOW, Directive::ALLOW] as $directive) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// rules set- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
? GeneratorBasedReader::fromStream($content)- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return bool- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private static function isValidScheme($scheme) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Parse URL- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $url- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return true;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $path - path to check- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($this->checkCleanParamRule(Directive::stripInline($rule), $path)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// url- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->userAgentMatcher = $userAgentMatcher;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!empty($this->tree)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return HostName::isValid($host);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// strip multi-spaces- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($param as $key) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Set UserAgent- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function setUserAgent(string $userAgent) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return bool- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
continue;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return bool- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log("Disallowed by HTTP status code {$this->httpStatusCode}");- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
case Directive::CLEAN_PARAM:- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private ?UserAgentMatcherInterface $userAgentMatcher;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (is_null($this->reader)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log('Reader is not passed, using a default one...');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (is_null($this->treeBuilder)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $scheme- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $userAgent- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $rule - rule to check- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($this->tree[$userAgent][$directive] as $robotRule) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Check HTTP status code rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
switch (Directive::attemptGetInline($rule)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($this->userAgentMatcher instanceof LoggerAwareInterface) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Line exceeds 120 characters; contains 135 characters Open
Open
$parsed['custom'] = (isset($parsed['path']) ? $parsed['path'] : '/') . (isset($parsed['query']) ? '?' . $parsed['query'] : '');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// check rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// host set- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private ?TreeBuilderInterface $treeBuilder;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (is_null($this->userAgentMatcher)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Validate URL scheme- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return array|false- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$rule = preg_replace('/\s+/S', ' ', $rule);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $cleanParam;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Set the HTTP status code- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Check rules- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $result;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// default encoding- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private array $tree = [];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$url = new Url($url);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
protected $host = null;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private function buildTree() {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->userAgentMatcher->setLogger($this->logger);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$parsed = parse_url($url);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
} else {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$parsed['custom'] = (isset($parsed['path']) ? $parsed['path'] : '/') . (isset($parsed['query']) ? '?' . $parsed['query'] : '');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$array = explode(' ', $rule, 2);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$cleanParam = [];- Exclude checks
Line exceeds 120 characters; contains 130 characters Open
Open
$cleanParam['path'] = isset($array[1]) ? $this->encode_url(preg_replace('/[^A-Za-z0-9\.-\/\*\_]/', '', $array[1])) : '/*';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log('Invalid HTTP status code, not taken into account.', ['code' => $code], LogLevel::WARNING);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
!is_null($this->logger) && $url->setLogger($this->logger);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!isset($this->tree[$userAgent][$directive])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$result = ($rule === $directive);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
const DEFAULT_ENCODING = 'UTF-8';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// UserAgent- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->treeBuilder->setContent($this->reader->getContentIterated());- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->tree = $this->treeBuilder->build();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
protected function parseURL($url) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
} else {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$param = explode('&', $array[0]);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
throw new \RuntimeException(WarmingMessages::SET_UA_DEPRECATED);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($this->checkHttpStatusCodeRule()) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($this->checkRuleSwitch($robotRule, $path)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private $content = '';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private ?ReaderInterface $reader;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log('Creating a default tree builder as none passed...');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return array- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @deprecated please check rules for exact user agent instead- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($this->httpStatusCode) && $this->httpStatusCode >= 500 && $this->httpStatusCode <= 599) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($cleanParam['param'] as $param) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return true;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (mb_strrpos($value, '/') == (mb_strlen($value) - 1)- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!isset($this->url)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$url['scheme'] . '://' . $url['host'] . ':' . $url['port'],- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log('Rule match: ' . Directive::HOST . ' directive');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Check url wrapper- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->buildTree();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log("{$directive} directive (unofficial): Not found, fallback to " . Directive::CRAWL_DELAY . " directive");- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return 0;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getCleanParam(): array {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!isset($this->tree[Directive::CLEAN_PARAM]) || empty($this->tree[Directive::CLEAN_PARAM])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
break;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (preg_match('@' . $escaped . '@', $path)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$value = '^' . $value;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private function checkHostRule($rule) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$url['host'],- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log(Directive::CLEAN_PARAM . ' directive: Not found');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
});- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($value as $subValue) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log(sprintf("Rules not found for the given User-Agent '%s'", $userAgent));- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @note NULL is returned to public API compatibility reasons. Will be removed in the future.- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!is_null($userAgent)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!strpos($path, "?$param=")- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log('Rule match: ' . Directive::CLEAN_PARAM . ' directive');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
krsort($input);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
} else {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output[] = '';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log(sprintf("No direct match found for '%s', fallback to *", $userAgent));- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return string[]|string|null- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getSitemaps(?string $userAgent = null): array {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->checkBasicRule($rule, $path);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return bool- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (in_array(- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree[$userAgent];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getHost(?string $userAgent = null) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->buildTree();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!$this->checkBasicRule($cleanParam['path'], $path)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Check basic rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$escaped = strtr($this->prepareRegexRule($rule), ['@' => '\@']);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string|null $userAgent - which robot to check for- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return array- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @deprecated- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return string- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// return all rules- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param ?string $userAgent- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
} else {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($this->tree as $userAgentBased) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Check Clean-Param rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// change @ to \@- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getDelay(string $userAgent = "*", string $type = Directive::CRAWL_DELAY) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($this->tree[$userAgent][Directive::CRAWL_DELAY])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$directive = ucfirst($directive);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (is_array($value)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return [];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$userAgent = $this->userAgentMatcher->getMatching($userAgent, array_keys($this->tree));- Exclude checks
Line exceeds 120 characters; contains 124 characters Open
Open
if (isset($this->tree[$userAgent][Directive::SITEMAP]) && !empty($this->tree[$userAgent][Directive::SITEMAP])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
case Directive::HOST;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (substr($value, 0, 2) != '.*') {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$url = $this->parseURL($this->url);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return bool- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$url = new Url($url);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return [];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Render- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output = [];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// Not multibyte- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// Shorter paths later- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree['*'];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return null;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$maps = array_merge($maps, $userAgentBased[Directive::SITEMAP]);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return true;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log($error_msg, [], LogLevel::ERROR);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
? Directive::CACHE_DELAY- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
: Directive::CRAWL_DELAY;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->buildTree();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree[Directive::CLEAN_PARAM];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $eol- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($rules as $directive => $value) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $maps;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private function checkCleanParamRule($rule, $path) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
protected function prepareRegexRule(string $value): string {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (mb_strlen($value) > 2 && mb_substr($value, -2) == '\$') {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$value .= '.*';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$url['host'] . ':' . $url['port'],- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->buildTree();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree[$userAgent][$directive];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree[$userAgent][Directive::CRAWL_DELAY];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output[] = $directive . ': ' . $value;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output[] = '';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->buildTree();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($this->tree['*'])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($this->tree[$userAgent][Directive::HOST]) && !empty($this->tree[$userAgent][Directive::HOST])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
private function checkBasicRule(string $rule, string $path): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Line exceeds 120 characters; contains 127 characters Open
Open
$this->log("{$directive} directive (unofficial): Not found, fallback to " . Directive::CRAWL_DELAY . " directive");- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output[] = $directive . ': ' . $subValue;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// direct match- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (!is_null($userAgent)) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $path- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$escape = ['$' => '\$', '?' => '\?', '.' => '\.', '*' => '.*', '[' => '\[', ']' => '\]'];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$url['scheme'] . '://' . $url['host'],- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->checkRules(Directive::DISALLOW, $url->getPath(), $userAgent);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($sitemaps as $sitemap) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getRules(?string $userAgent = null) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($this->tree[$userAgent])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($this->tree as $userAgentBased) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return !empty($hosts) ? $hosts : null;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return true;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
break;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
default:- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$cleanParam = $this->explodeCleanParamRule($rule);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
|| mb_strrpos($value, '=') == (mb_strlen($value) - 1)- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $value;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
]- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
!is_null($this->logger) && $url->setLogger($this->logger);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$directive = in_array($type, [Directive::CACHE, Directive::CACHE_DELAY])- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getLog(): array {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$maps = [];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// check if path prefix matches the path of the url we're checking- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @return bool- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$error_msg = WarmingMessages::INLINED_HOST;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$host, [- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @deprecated- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
foreach ($input as $userAgent => $rules) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$host = $this->getHost();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output[] = 'Sitemap: ' . $sitemap;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->buildTree();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$userAgent = $this->userAgentMatcher->getMatching($userAgent, array_keys($this->tree));- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($this->checkHostRule(Directive::stripInline($rule))) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
&& !strpos($path, "&$param=")- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log('Rule match: Path');- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$value = substr($value, 0, -2) . '$';- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
|| mb_strrpos($value, '?') == (mb_strlen($value) - 1)- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* Check Host rule- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$host = trim(str_ireplace(Directive::HOST . ':', '', mb_strtolower($rule)));- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @param string $url - url to check- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($this->tree[$userAgent][$directive])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->reader->getContentRaw();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
usort($value, function ($a, $b) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($userAgent === null) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($userAgentBased[Directive::SITEMAP]) && !empty($userAgentBased[Directive::SITEMAP])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return false;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// match result- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return true;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function isDisallowed(string $url, string $userAgent = '*'): bool {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// return delay for requested directive- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$this->log("$directive directive: Not found");- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function getContent(): string {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
* @see RobotsTxtParser::getLogger()- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output[] = 'User-agent: ' . $userAgent;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return mb_strlen($a) < mb_strlen($b);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
// fallback for *- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree[$userAgent][Directive::HOST];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($userAgentBased[Directive::HOST]) && !empty($userAgentBased[Directive::HOST])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if (isset($this->tree[$userAgent][Directive::SITEMAP]) && !empty($this->tree[$userAgent][Directive::SITEMAP])) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$value = str_replace(array_keys($escape), array_values($escape), $value);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
/**- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$input = $this->getRules();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$output[] = 'Host: ' . $host;- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$sitemaps = $this->getSitemaps();- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return implode($eol, $output);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$userAgent = $this->userAgentMatcher->getMatching($userAgent, array_keys($this->tree));- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
*/- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
$hosts = [];- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
array_push($hosts, $userAgentBased[Directive::HOST]);- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
public function render($eol = "\r\n") {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
if ($host !== null) {- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
}- Exclude checks
Spaces must be used to indent lines; tabs are not allowed Open
Open
return $this->tree[$userAgent][Directive::SITEMAP];- Exclude checks
Opening brace should be on a new line Open
Open
public function isAllowed(string $url, ?string $userAgent = '*'): bool {- Exclude checks
Opening brace should be on a new line Open
Open
public function setLogger(LoggerInterface $logger): void {- Exclude checks
Opening brace should be on a new line Open
Open
public function setHttpStatusCode(int $code): bool {- Exclude checks
Opening brace should be on a new line Open
Open
private function checkHttpStatusCodeRule(): bool {- Exclude checks
Opening brace should be on a new line Open
Open
protected function checkRules(string $rule, string $path, string $userAgent = '*'): bool {- Exclude checks
Opening brace should be on a new line Open
Open
private function buildTree() {- Exclude checks
Opening brace should be on a new line Open
Open
protected function parseURL($url) {- Exclude checks
Opening brace should be on a new line Open
Open
public function setUserAgent(string $userAgent) {- Exclude checks
Opening brace should be on a new line Open
Open
public function getLogger(): ?LoggerInterface {- Exclude checks
Opening brace should be on a new line Open
Open
private static function isValidHostName(string $host): bool {- Exclude checks
Opening brace should be on a new line Open
Open
private function explodeCleanParamRule($rule) {- Exclude checks
Opening brace should be on a new line Open
Open
protected function checkRuleSwitch(string $rule, string $path): bool {- Exclude checks
Opening brace should be on a new line Open
Open
private static function isValidScheme($scheme) {- Exclude checks
Opening brace should be on a new line Open
Open
public function getCleanParam(): array {- Exclude checks
Opening brace should be on a new line Open
Open
private function checkHostRule($rule) {- Exclude checks
Opening brace should be on a new line Open
Open
public function getHost(?string $userAgent = null) {- Exclude checks
Opening brace should be on a new line Open
Open
public function getSitemaps(?string $userAgent = null): array {- Exclude checks
Opening brace should be on a new line Open
Open
private function checkBasicRule(string $rule, string $path): bool {- Exclude checks
Opening brace should be on a new line Open
Open
public function getDelay(string $userAgent = "*", string $type = Directive::CRAWL_DELAY) {- Exclude checks
Opening brace should be on a new line Open
Open
public function render($eol = "\r\n") {- Exclude checks
Opening brace should be on a new line Open
Open
private function checkCleanParamRule($rule, $path) {- Exclude checks
Opening brace should be on a new line Open
Open
protected function prepareRegexRule(string $value): string {- Exclude checks
Opening brace should be on a new line Open
Open
public function getLog(): array {- Exclude checks
Opening brace should be on a new line Open
Open
public function getRules(?string $userAgent = null) {- Exclude checks
Opening brace should be on a new line Open
Open
public function getContent(): string {- Exclude checks
Opening brace should be on a new line Open
Open
public function isDisallowed(string $url, string $userAgent = '*'): bool {- Exclude checks
The variable $error_msg is not named in camelCase. Open
Open
private function checkHostRule($rule) { if (!isset($this->url)) { $error_msg = WarmingMessages::INLINED_HOST; $this->log($error_msg, [], LogLevel::ERROR); return false;- Read upRead up
- Exclude checks
The variable $error_msg is not named in camelCase. Open
Open
private function checkHostRule($rule) { if (!isset($this->url)) { $error_msg = WarmingMessages::INLINED_HOST; $this->log($error_msg, [], LogLevel::ERROR); return false;- Read upRead up
- Exclude checks