File train_neural_net.py has 549 lines of code (exceeds 250 allowed). Consider refactoring.

Open

from glob import glob
import json
import logging
import os
import random

Function handle has a Cognitive Complexity of 21 (exceeds 5 allowed). Consider refactoring.

Open

    def handle(self, *args, **options):
        start_time = now()

        maxfiles = options['maxfiles']
        pattern = re.compile('1721.1\-(\d+)\.txt')

• Read up
  Read up

Function handle has a Cognitive Complexity of 15 (exceeds 5 allowed). Consider refactoring.

Open

    def handle(self, *args, **options):
        orig_count = Citation.objects.count()
        deleted = 0
        loopcount = 0
        for c in Citation.objects.all()[0:orig_count]:

• Read up
  Read up

Function train_model has a Cognitive Complexity of 11 (exceeds 5 allowed). Consider refactoring.

Open

    def train_model(self, filename, queryset=Thesis.objects.all()):
        # Don't bother with theses when we know we can't get text from them.
        queryset = queryset.filter(unextractable=False)
        print('About to extract all text')
        for thesis in queryset:

• Read up
  Read up

Function rewrite has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

def rewrite(max=0):
    count = 0
    fetcher = DocFetcher()

    items = fetcher.get_record_list(DSPACE_OAI_URI)

• Read up
  Read up

Function add_people has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def add_people(self, people, author=True):
        """Given a list of person name strings, add Person relations."""
        if author:
            role = Contribution.AUTHOR
        else:

• Read up
  Read up

Function extract_degree has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

    def extract_degree(self, metadata, item_sets):
        if item_sets:
            degree = self.extract_degree_from_sets(item_sets)
            if degree:
                return degree

• Read up
  Read up

Function inner_train_model has 5 arguments (exceeds 4 allowed). Consider refactoring.

Open

    def inner_train_model(self, window, size, iterator, filename,

Function get_network_files has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

Open

    def get_network_files(self, args, start_date=None, end_date=None):
        print('Network files!!!!!!')
        items = self.get_record_list(DSPACE_OAI_URI, start_date, end_date)
        parsed_items = self.parse_record_list(items)
        total_items_processed = 0

• Read up
  Read up

Function extract_degree has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

Open

    def extract_degree(self, degree_statement):
        """Takes METS format metadata and finds degrees."""
        result = []
        try:
            degree = re.findall(r'[A-Z][a-z]{,4}\.? ?[A-Z][a-z]{,3}\.?[A-Z]?\.?'

• Read up
  Read up

Function _reprocess_bad has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

Open

def _reprocess_bad(good, bad):
    for handle in bad.keys():
        filteredbad = [refdict for refdict in bad[handle]
                       if refdict and len(refdict['raw_ref'][0]) < 200]

• Read up
  Read up

Function handle has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

Open

    def handle(self, *args, **options):
        for c in Citation.objects.all():
            if re.match(r'\d+\.', c.raw_ref):
                c.raw_ref = re.sub(r'^\d+\. ', '', c.raw_ref).lstrip()
                c.save()

• Read up
  Read up

Avoid too many return statements within this function.

Open

            return

Function encoding has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

Open

    def encoding(self):
        if not self._encoding:
            detector = UniversalDetector()
            for line in self.doc:
                detector.feed(line)

• Read up
  Read up

Function extract_date has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

Open

    def extract_date(self, metadata):
        # There will be several (representing copyright, accessioning, etc.)
        # The copyright date will be a four-digit year. Find the earliest
        # year (there may be a substantial difference between copyright year
        # and archival processing years).

• Read up
  Read up

Function extract_contributors has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

Open

    def extract_contributors(self, metadata):
        # Includes advisor and department.
        contributors = metadata.findall('.//dc:contributor', METS_NAMESPACE)
        advisors = []
        departments = []

• Read up
  Read up

Function _find_candidate_refs has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

Open

def _find_candidate_refs(reftuples):
    # Given citation data associated with various filenames, classify the
    # citations as good or bad.
    #
    # :param reftuples: list of tuples of (filename, list of dictd of citation

• Read up
  Read up