File splash_browser.py has 319 lines of code (exceeds 250 allowed). Consider refactoring.

Open

# -*- coding: utf-8 -*-
"""
transistor.browsers.splash_browser
~~~~~~~~~~~~
This module implements the SplashBrowser class.

Function _get_usd_row_cells has a Cognitive Complexity of 21 (exceeds 5 allowed). Consider refactoring.

Open

def _get_usd_row_cells(trs):
    """
    Get the table cells from the webpage we need for the exchange rate data.
    :param trs:  browser.get_current_page().find_all("tr")
    :return: list(): a list of cells

• Read up
  Read up

Function manage has a Cognitive Complexity of 19 (exceeds 5 allowed). Consider refactoring.

Open

    def manage(self):
        """"
        Manage will hand out work when the appropriate Worker is free.
        The manager timeout must be less than worker timeout, or else, the
        workers will be idled and shutdown.

• Read up
  Read up

File splash_scraper_abc.py has 264 lines of code (exceeds 250 allowed). Consider refactoring.

Open

# -*- coding: utf-8 -*-
"""
transistor.scrapers.splash_scraper_abc
~~~~~~~~~~~~
This module implements an abstract base class for a scraper which assumes the

SplashBrowser has 22 functions (exceeds 20 allowed). Consider refactoring.

Open

class SplashBrowser(StatefulBrowser, SplashBrowserMixin):
    """
    Make a few changes and overrides to enable a mechanicalsoup StatefulBrowser to
    work with Splash alone or Splash + Crawlera.

File splash_crawler_abc.py has 258 lines of code (exceeds 250 allowed). Consider refactoring.

Open

# -*- coding: utf-8 -*-
"""
transistor.crawlers.splash_crawler_abc
~~~~~~~~~~~~
This module implements an abstract base class for a crawler which assumes the

Function _get_serialized_fields has a Cognitive Complexity of 16 (exceeds 5 allowed). Consider refactoring.

Open

    def _get_serialized_fields(self, item, default_value=None, include_empty=None):
        """Return the fields to export as an iterable of tuples
        (name, serialized_value)
        """
        if include_empty is None:

• Read up
  Read up

Function write has 28 lines of code (exceeds 25 allowed). Consider refactoring.

Open

    def write(self):
        """
        Create the new class::Item() container object. This is
        the Transistor equivalent to the Scrapy API for Item Loaders.

Function _init_workers has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring.

Open

    def _init_workers(self, kwargs):
        """
        Create the WorkGroups by tracker name and assign them by name to the
        workgroups dict.

• Read up
  Read up

Function select_form has a Cognitive Complexity of 9 (exceeds 5 allowed). Consider refactoring.

Open

    def select_form(self, selector="form", nr=0):
        """Select a form in the current page.

        :param selector: CSS selector or a bs4.element.Tag object to identify
            the form to select.

• Read up
  Read up

Function default has a Cognitive Complexity of 9 (exceeds 5 allowed). Consider refactoring.

Open

    def default(self, o):
        if isinstance(o, set):
            return list(o)
        elif isinstance(o, datetime.datetime):
            return o.strftime("%s %s" % (self.DATE_FORMAT, self.TIME_FORMAT))

• Read up
  Read up

Function __init__ has 7 arguments (exceeds 4 allowed). Consider refactoring.

Open

    def __init__(self, job_id, tasks: Type[Union[Type[StatefulBook],

Function __init__ has 6 arguments (exceeds 4 allowed). Consider refactoring.

Open

    def __init__(self, records=None, sheet=None, to_do=None, in_proc=None,

Function _stateful_post has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def _stateful_post(self, url, *args, **kwargs):
        """
        Execute sending the post request to the local running Splash service with our
        self.browser object.

• Read up
  Read up

Avoid deeply nested control flow statements.

Open

                    for cell in usd_row:
                        if '\n' not in cell:
                            cells.append(cell)
    return cells

Function _stateful_post has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def _stateful_post(self, url, *args, **kwargs):
        """
        Execute sending the post request to the local running Splash service with our
        self.browser object.

• Read up
  Read up

Avoid deeply nested control flow statements.

Open

                            for worker in workgroup:
                                one_task = q.get(timeout=self.mgr_qtimeout)
                                worker.tasks.put(one_task)
                gevent.sleep(0)

Function _export_xml_field has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def _export_xml_field(self, name, serialized_value, depth):
        self._beautify_indent(depth=depth)
        self.xg.startElement(name, {})
        if hasattr(serialized_value, 'items'):
            self._beautify_newline()

• Read up
  Read up

Function process_task has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def process_task(self, body, message):
        """
        Process messages to extract the task keywords and then
        load them into a gevent Queue for each tracker.

• Read up
  Read up

Function send_as_task has 5 arguments (exceeds 4 allowed). Consider refactoring.

Open

def send_as_task(connection, keywords, routing_key, exchange, kwargs={}):