File fetch_sex_places.py has 411 lines of code (exceeds 250 allowed). Consider refactoring.

Open

import asyncio
import json
import logging
import math
import os

File fetch_tse_data.py has 279 lines of code (exceeds 250 allowed). Consider refactoring.

Open

"""
This script downloads and format some data from TSE website.
The first objective with this data is to obtain a list of all politicians in Brazil.
In march 2017, the data available in TSE website contained information about elected people from the year 1994 to 2016.
Data before 1994 does not contains name of the politicians.

File fetch_cnpj_info.py has 267 lines of code (exceeds 250 allowed). Consider refactoring.

Open

from concurrent import futures
import json
import argparse
import time
import random

File translation_table.py has 261 lines of code (exceeds 250 allowed). Consider refactoring.

Open

from bs4 import BeautifulSoup


EN = (
    {

Function transform_and_translate_data has a Cognitive Complexity of 12 (exceeds 5 allowed). Consider refactoring.

Open

def transform_and_translate_data(json_data):
    """
    Transform main activity, secondary activity and partners list in
    multi columns and translate column names.
    """

• Read up
  Read up

Function run has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring.

Open

def run(target, limit=None):
    """
    :param target: (string) path to the directory to save the receipts images
    :param limit: (int) limit the amount of receipts to fecth (default: None)
    """

• Read up
  Read up

Function handle has a Cognitive Complexity of 9 (exceeds 5 allowed). Consider refactoring.

Open

    def handle(self, *args, **options):
        if not all(self.credentials):
            self.log.warning('Twitter API credentials not set.')
            return

• Read up
  Read up

Avoid deeply nested control flow statements.

Open

                if future.exception() is None and future.result() is not None and future.result()['status'] == 'OK':
                    result_translated = transform_and_translate_data(
                        future.result())
                    temp_dataset = pd.concat([temp_dataset, result_translated])
                    if last_saving_point < divmod(len(temp_dataset.index), 100)[0]:

Function handle has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def handle(self, *args, **options):
        self.batch, self.pause = options['batch_size'], options['pause']
        self.count = 0
        self.queue = []

• Read up
  Read up

Function organize_deputy_data has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

def organize_deputy_data(deputy, advisors):
    """
    Organizes all the deputies information in a list. Use this function to
    prepare data to be written to CSV format.
    :deputy: (dict) A dict with keys `deputy_name` and `deputy_number`

• Read up
  Read up

Function get has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def get(self, request):

        # get filtering parameters from query string
        params = (
            'applicant_id',

• Read up
  Read up

Function translate_impeded_non_profit_entities_dataset has 5 arguments (exceeds 4 allowed). Consider refactoring.

Open

def translate_impeded_non_profit_entities_dataset(filepath, dataset_name, year, month, day):

Function translate_national_register_punished_companies_dataset has 5 arguments (exceeds 4 allowed). Consider refactoring.

Open

def translate_national_register_punished_companies_dataset(filepath, dataset_name, year, month, day):

Function translate_inident_and_suspended_companies_dataset has 5 arguments (exceeds 4 allowed). Consider refactoring.

Open

def translate_inident_and_suspended_companies_dataset(filepath, dataset_name, year, month, day):

Function dummy_translation_dataset has 5 arguments (exceeds 4 allowed). Consider refactoring.

Open

def dummy_translation_dataset(filepath, dataset_name, year, month, day):

Function send_requests has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

Open

def send_requests(reqs):
    """
    Send all the requests in :reqs: and reads the response data to extract the
    deputies data.  It will check if a deputy has more than one page of
    advisors and send new requests if True

• Read up
  Read up

Function read_geocoding_info has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

Open

def read_geocoding_info(company):
    cnpj = re.sub(CNPJ_REGEX, '', company['cnpj'])
    filename = '%s/%s.pkl' % (TEMP_PATH, cnpj)
    if os.path.isfile(filename):
        with open(filename, 'rb') as f:

• Read up
  Read up

Function find_sum_of_values has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

Open

def find_sum_of_values(df, aggregator, value):
    '''
    Return a dataframe with the statistics of values from "value" property
    aggregated by unique values from the column "aggregator"

• Read up
  Read up

Function get_civil_names has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

Open

    def get_civil_names(self):
        congresspeople_ids = self.get_all_congresspeople_ids()
        for i, congress_id in enumerate(congresspeople_ids):
            if not np.math.isnan(float(congress_id)):
                percentage = (i / self.total * 100)

• Read up
  Read up

Function to_date has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

Open

    def to_date(text):

        ddmmyyyy = match(r'^[\d]{1,2}/[\d]{1,2}/[\d]{2,4}$', text)
        yyyymmdd = match(r'^[\d]{2,4}-[\d]{1,2}-[\d]{2,4}', text)

• Read up
  Read up