Sign upLogin

LucaCappelletti94/tinycrawler

View on GitHub
Star

Showing 81 of 90 total issues

Missing whitespace around operator
Open

    def Log(self, log_filename: str)->Log:
Severity: Minor
Found in tinycrawler/managers/tinycrawler_manager.py by pep8

Surround operators with a single space on either side.

- Always surround these binary operators with a single space on
  either side: assignment (=), augmented assignment (+=, -= etc.),
  comparisons (==, <, >, !=, <=, >=, in, not in, is, is not),
  Booleans (and, or, not).

- If operators with different priorities are used, consider adding
  whitespace around the operators with the lowest priorities.

Okay: i = i + 1
Okay: submitted += 1
Okay: x = x * 2 - 1
Okay: hypot2 = x * x + y * y
Okay: c = (a + b) * (a - b)
Okay: foo(bar, key='word', *args, **kwargs)
Okay: alpha[:-i]

E225: i=i+1
E225: submitted +=1
E225: x = x /2 - 1
E225: z = x **y
E225: z = 1and 1
E226: c = (a+b) * (a-b)
E226: hypot2 = x*x + y*y
E227: c = a|b
E228: msg = fmt%(errno, errmsg)

Line too long (126 > 79 characters)
Open

            self, file_parser: Callable[[str, BeautifulSoup, Set[str], Log], None], url_validator: Callable[[str, Log], bool],
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (159 > 79 characters)
Open

            use_cli: bool=True, bloom_filters_capacity: int=1e9, responses_queue_max_size: int= 10000, download_attempts: int=10, follow_robots_txt: bool=True,
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (83 > 79 characters)
Open

                           custom_domains_timeout, follow_robots_txt, self._robots)
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Missing whitespace around parameter equals
Open

            use_cli: bool=True, bloom_filters_capacity: int=1e9, responses_queue_max_size: int= 10000, download_attempts: int=10, follow_robots_txt: bool=True,
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Don't use spaces around the '=' sign in function arguments.

Don't use spaces around the '=' sign when used to indicate a
keyword argument or a default parameter value, except when
using a type annotation.

Okay: def complex(real, imag=0.0):
Okay: return magic(r=real, i=imag)
Okay: boolean(a == b)
Okay: boolean(a != b)
Okay: boolean(a <= b)
Okay: boolean(a >= b)
Okay: def foo(arg: int = 42):
Okay: async def foo(arg: int = 42):

E251: def complex(real, imag = 0.0):
E251: return magic(r = real, i = imag)
E252: def complex(real, image: float=0.0):

Missing whitespace around parameter equals
Open

            parser_library: str="html5lib",
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Don't use spaces around the '=' sign in function arguments.

Don't use spaces around the '=' sign when used to indicate a
keyword argument or a default parameter value, except when
using a type annotation.

Okay: def complex(real, imag=0.0):
Okay: return magic(r=real, i=imag)
Okay: boolean(a == b)
Okay: boolean(a != b)
Okay: boolean(a <= b)
Okay: boolean(a >= b)
Okay: def foo(arg: int = 42):
Okay: async def foo(arg: int = 42):

E251: def complex(real, imag = 0.0):
E251: return magic(r = real, i = imag)
E252: def complex(real, image: float=0.0):

Missing whitespace around parameter equals
Open

    def set(self, category: str, name: str, value: int=1):
Severity: Minor
Found in tinycrawler/statistics/statistics.py by pep8

Don't use spaces around the '=' sign in function arguments.

Don't use spaces around the '=' sign when used to indicate a
keyword argument or a default parameter value, except when
using a type annotation.

Okay: def complex(real, imag=0.0):
Okay: return magic(r=real, i=imag)
Okay: boolean(a == b)
Okay: boolean(a != b)
Okay: boolean(a <= b)
Okay: boolean(a >= b)
Okay: def foo(arg: int = 42):
Okay: async def foo(arg: int = 42):

E251: def complex(real, imag = 0.0):
E251: return magic(r = real, i = imag)
E252: def complex(real, image: float=0.0):

Line too long (159 > 79 characters)
Open

            if sum([self._downloader.alive_processes_number(), self._parser.alive_processes_number(), self._urls_number.value, self._pages_number.value]) == 0:
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (393 > 79 characters)
Open

    def __init__(self, process_spawn_event: Event, process_callback_event: Event, pages_number: Value, urls_number: Value, urls: Urls, local: Local, proxies: Queue, responses: Queue, statistics: Statistics, connection_timeout: float, custom_connection_timeout: Callable[[str], float], maximal_failure_proxy_rate: float, download_attempts: int, cooldown_time_beetween_download_attempts: float):
Severity: Minor
Found in tinycrawler/process/downloader.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Missing whitespace around operator
Open

    def Urls(self, statistics: Statistics, bloom_filters_capacity: int)->Urls:
Severity: Minor
Found in tinycrawler/managers/tinycrawler_manager.py by pep8

Surround operators with a single space on either side.

- Always surround these binary operators with a single space on
  either side: assignment (=), augmented assignment (+=, -= etc.),
  comparisons (==, <, >, !=, <=, >=, in, not in, is, is not),
  Booleans (and, or, not).

- If operators with different priorities are used, consider adding
  whitespace around the operators with the lowest priorities.

Okay: i = i + 1
Okay: submitted += 1
Okay: x = x * 2 - 1
Okay: hypot2 = x * x + y * y
Okay: c = (a + b) * (a - b)
Okay: foo(bar, key='word', *args, **kwargs)
Okay: alpha[:-i]

E225: i=i+1
E225: submitted +=1
E225: x = x /2 - 1
E225: z = x **y
E225: z = 1and 1
E226: c = (a+b) * (a-b)
E226: hypot2 = x*x + y*y
E227: c = a|b
E228: msg = fmt%(errno, errmsg)

Line too long (96 > 79 characters)
Open

            if not proxy.is_local() and proxy.failure_rate() < self._maximal_failure_proxy_rate:
Severity: Minor
Found in tinycrawler/process/downloader.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Missing whitespace around operator
Open

    def can_fetch(self, url: str)->bool:
Severity: Minor
Found in tinycrawler/robots/robots.py by pep8

Surround operators with a single space on either side.

- Always surround these binary operators with a single space on
  either side: assignment (=), augmented assignment (+=, -= etc.),
  comparisons (==, <, >, !=, <=, >=, in, not in, is, is not),
  Booleans (and, or, not).

- If operators with different priorities are used, consider adding
  whitespace around the operators with the lowest priorities.

Okay: i = i + 1
Okay: submitted += 1
Okay: x = x * 2 - 1
Okay: hypot2 = x * x + y * y
Okay: c = (a + b) * (a - b)
Okay: foo(bar, key='word', *args, **kwargs)
Okay: alpha[:-i]

E225: i=i+1
E225: submitted +=1
E225: x = x /2 - 1
E225: z = x **y
E225: z = 1and 1
E226: c = (a+b) * (a-b)
E226: hypot2 = x*x + y*y
E227: c = a|b
E228: msg = fmt%(errno, errmsg)

Missing whitespace around parameter equals
Open

            proxy_timeout: float=10, domains_timeout: float=10, robots_timeout: float=60*60*24, connection_timeout: float=5, cooldown_time_beetween_download_attempts: float=1,
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Don't use spaces around the '=' sign in function arguments.

Don't use spaces around the '=' sign when used to indicate a
keyword argument or a default parameter value, except when
using a type annotation.

Okay: def complex(real, imag=0.0):
Okay: return magic(r=real, i=imag)
Okay: boolean(a == b)
Okay: boolean(a != b)
Okay: boolean(a <= b)
Okay: boolean(a >= b)
Okay: def foo(arg: int = 42):
Okay: async def foo(arg: int = 42):

E251: def complex(real, imag = 0.0):
E251: return magic(r = real, i = imag)
E252: def complex(real, image: float=0.0):

Missing whitespace around operator
Open

    def timeout(self, domain: str)->float:
Severity: Minor
Found in tinycrawler/robots/robots.py by pep8

Surround operators with a single space on either side.

- Always surround these binary operators with a single space on
  either side: assignment (=), augmented assignment (+=, -= etc.),
  comparisons (==, <, >, !=, <=, >=, in, not in, is, is not),
  Booleans (and, or, not).

- If operators with different priorities are used, consider adding
  whitespace around the operators with the lowest priorities.

Okay: i = i + 1
Okay: submitted += 1
Okay: x = x * 2 - 1
Okay: hypot2 = x * x + y * y
Okay: c = (a + b) * (a - b)
Okay: foo(bar, key='word', *args, **kwargs)
Okay: alpha[:-i]

E225: i=i+1
E225: submitted +=1
E225: x = x /2 - 1
E225: z = x **y
E225: z = 1and 1
E226: c = (a+b) * (a-b)
E226: hypot2 = x*x + y*y
E227: c = a|b
E228: msg = fmt%(errno, errmsg)

Line too long (107 > 79 characters)
Open

            custom_timeout: Callable[[object], float], function to call to get the timeout for given value.
Severity: Minor
Found in tinycrawler/eta/eta.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (80 > 79 characters)
Open

            requests_rate_delay = requests_rate.seconds / requests_rate.requests
Severity: Minor
Found in tinycrawler/robots/robots.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Missing whitespace around operator
Open

    def failure_rate(self)->float:
Severity: Minor
Found in tinycrawler/proxy/proxy.py by pep8

Surround operators with a single space on either side.

- Always surround these binary operators with a single space on
  either side: assignment (=), augmented assignment (+=, -= etc.),
  comparisons (==, <, >, !=, <=, >=, in, not in, is, is not),
  Booleans (and, or, not).

- If operators with different priorities are used, consider adding
  whitespace around the operators with the lowest priorities.

Okay: i = i + 1
Okay: submitted += 1
Okay: x = x * 2 - 1
Okay: hypot2 = x * x + y * y
Okay: c = (a + b) * (a - b)
Okay: foo(bar, key='word', *args, **kwargs)
Okay: alpha[:-i]

E225: i=i+1
E225: submitted +=1
E225: x = x /2 - 1
E225: z = x **y
E225: z = 1and 1
E226: c = (a+b) * (a-b)
E226: hypot2 = x*x + y*y
E227: c = a|b
E228: msg = fmt%(errno, errmsg)

Missing whitespace around parameter equals
Open

            use_cli: bool=True, bloom_filters_capacity: int=1e9, responses_queue_max_size: int= 10000, download_attempts: int=10, follow_robots_txt: bool=True,
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Don't use spaces around the '=' sign in function arguments.

Don't use spaces around the '=' sign when used to indicate a
keyword argument or a default parameter value, except when
using a type annotation.

Okay: def complex(real, imag=0.0):
Okay: return magic(r=real, i=imag)
Okay: boolean(a == b)
Okay: boolean(a != b)
Okay: boolean(a <= b)
Okay: boolean(a >= b)
Okay: def foo(arg: int = 42):
Okay: async def foo(arg: int = 42):

E251: def complex(real, imag = 0.0):
E251: return magic(r = real, i = imag)
E252: def complex(real, image: float=0.0):

Line too long (120 > 79 characters)
Open

    def __init__(self, timeout: float, custom_timeout: Callable[[str], float], follow_robots_txt: bool, robots: Robots):
Severity: Minor
Found in tinycrawler/domains_eta/domains_eta.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Missing whitespace around parameter equals
Open

            proxy_timeout: float=10, domains_timeout: float=10, robots_timeout: float=60*60*24, connection_timeout: float=5, cooldown_time_beetween_download_attempts: float=1,
Severity: Minor
Found in tinycrawler/tinycrawler.py by pep8

Don't use spaces around the '=' sign in function arguments.

Don't use spaces around the '=' sign when used to indicate a
keyword argument or a default parameter value, except when
using a type annotation.

Okay: def complex(real, imag=0.0):
Okay: return magic(r=real, i=imag)
Okay: boolean(a == b)
Okay: boolean(a != b)
Okay: boolean(a <= b)
Okay: boolean(a >= b)
Okay: def foo(arg: int = 42):
Okay: async def foo(arg: int = 42):

E251: def complex(real, imag = 0.0):
E251: return magic(r = real, i = imag)
E252: def complex(real, image: float=0.0):
Severity
Category
Status
Source
Language