Sign upLogin

CenterForOpenScience/scrapi

View on GitHub
Star

Showing 45 of 45 total issues

Function format_tags has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.
Open

def format_tags(all_tags, sep=','):
    tags = []
    if isinstance(all_tags, six.string_types):
        tags = all_tags.split(sep)
    elif isinstance(all_tags, list):
Severity: Minor
Found in scrapi/base/helpers.py - About 45 mins to fix

Cognitive Complexity

Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

A method's cognitive complexity is based on a few simple rules:

  • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
  • Code is considered more complex for each "break in the linear flow of the code"
  • Code is considered more complex when "flow breaking structures are nested"

Further reading

Function migrate has 6 arguments (exceeds 4 allowed). Consider refactoring.
Open

def migrate(migration, sources=None, kwargs_string=None, dry=True, async=False, group_size=1000):
Severity: Minor
Found in tasks.py - About 45 mins to fix

    Function populate has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.
    Open

    def populate(grid_file):
    transformer = JSONTransformer(schema)
    for doc in get_jsons(grid_file):
        transformed = transformer.transform(doc, load=False)
        try:
    Severity: Minor
    Found in institutions/grid.py - About 45 mins to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Function migrate has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.
    Open

    def migrate(migration, sources=None, kwargs_string=None, dry=True, async=False, group_size=1000):
    ''' Task to run a migration.

    :param migration: The migration function to run. This is passed in
    as a string then interpreted as a function by the invoke task.
    Severity: Minor
    Found in tasks.py - About 45 mins to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Function record_or_load_response has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.
    Open

    def record_or_load_response(method, url, throttle=None, force=False, params=None, expected=(200,), **kwargs):

    resp = _maybe_load_response(method, url)

    if not force and resp and resp.ok:
    Severity: Minor
    Found in scrapi/requests.py - About 45 mins to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Function cross_db has 6 arguments (exceeds 4 allowed). Consider refactoring.
    Open

    def cross_db(docs, source_db=None, target_db=None, index=None, versions=False, **kwargs):
    Severity: Minor
    Found in scrapi/migrations.py - About 45 mins to fix

      Function migrate has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.
      Open

      def migrate(migration, source_db=None, sources=tuple(), async=False, dry=True, group_size=1000, **kwargs):

    source_db = source_db or settings.CANONICAL_PROCESSOR
    documents = processing.get_processor(source_db).documents
      Severity: Minor
      Found in scrapi/tasks.py - About 45 mins to fix

      Cognitive Complexity

      Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

      A method's cognitive complexity is based on a few simple rules:

      • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
      • Code is considered more complex for each "break in the linear flow of the code"
      • Code is considered more complex when "flow breaking structures are nested"

      Further reading

      Similar blocks of code found in 2 locations. Consider refactoring.
      Open

              ('dataCenter', ('//dif:Data_Center/node()', compose(
            list,
            lambda x: map(element_to_dict, x),
            lambda x: filter(non_string, x)
      Severity: Minor
      Found in scrapi/base/schemas.py and 1 other location - About 40 mins to fix
      scrapi/base/schemas.py on lines 85..88

      Duplicated Code

      Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

      Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

      When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

      Tuning

      This issue has a mass of 34.

      We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

      The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

      If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

      See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

      Refactorings

      Further Reading

      Similar blocks of code found in 2 locations. Consider refactoring.
      Open

              ('relatedUrl', ('//dif:Related_URL/node()', compose(
            list,
            lambda x: map(element_to_dict, x),
            lambda x: filter(non_string, x)
      Severity: Minor
      Found in scrapi/base/schemas.py and 1 other location - About 40 mins to fix
      scrapi/base/schemas.py on lines 80..83

      Duplicated Code

      Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

      Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

      When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

      Tuning

      This issue has a mass of 34.

      We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

      The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

      If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

      See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

      Refactorings

      Further Reading

      Function oai_get_records_and_token has 5 arguments (exceeds 4 allowed). Consider refactoring.
      Open

      def oai_get_records_and_token(url, throttle, force, namespaces, verify):
      Severity: Minor
      Found in scrapi/base/helpers.py - About 35 mins to fix

        Function test has 5 arguments (exceeds 4 allowed). Consider refactoring.
        Open

        def test(cov=True, doctests=True, verbose=False, debug=False, pdb=False):
        Severity: Minor
        Found in tasks.py - About 35 mins to fix

          Function gather_identifiers has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.
          Open

          def gather_identifiers(args):
    identifiers = []
    for arg in args:
        if isinstance(arg, list):
            for identifier in arg:
          Severity: Minor
          Found in scrapi/base/helpers.py - About 35 mins to fix

          Cognitive Complexity

          Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

          A method's cognitive complexity is based on a few simple rules:

          • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
          • Code is considered more complex for each "break in the linear flow of the code"
          • Code is considered more complex when "flow breaking structures are nested"

          Further reading

          Function oai_extract_dois has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.
          Open

          def oai_extract_dois(*args):
    identifiers = gather_identifiers(args)
    dois = []
    for item in identifiers:
        if 'doi' in item.lower():
          Severity: Minor
          Found in scrapi/base/helpers.py - About 35 mins to fix

          Cognitive Complexity

          Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

          A method's cognitive complexity is based on a few simple rules:

          • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
          • Code is considered more complex for each "break in the linear flow of the code"
          • Code is considered more complex when "flow breaking structures are nested"

          Further reading

          Function _transform_value has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.
          Open

              def _transform_value(self, value, doc, fail=False):
        if isinstance(value, dict):
            return self._transform_dict(value, doc, fail=fail)
        elif isinstance(value, list):
            return self._transform_list(value, doc, fail=fail)
          Severity: Minor
          Found in scrapi/base/transformer.py - About 35 mins to fix

          Cognitive Complexity

          Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

          A method's cognitive complexity is based on a few simple rules:

          • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
          • Code is considered more complex for each "break in the linear flow of the code"
          • Code is considered more complex when "flow breaking structures are nested"

          Further reading

          Function documents has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.
          Open

              def documents(self, *sources):
        q = DocumentModel.objects.timeout(500).allow_filtering().all().limit(100)
        querysets = (q.filter(source=source) for source in sources) if sources else [q]
        for query in querysets:
            page = try_n_times(5, list, query)
          Severity: Minor
          Found in scrapi/processing/cassandra.py - About 35 mins to fix

          Cognitive Complexity

          Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

          A method's cognitive complexity is based on a few simple rules:

          • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
          • Code is considered more complex for each "break in the linear flow of the code"
          • Code is considered more complex when "flow breaking structures are nested"

          Further reading

          Function copy_to_unicode has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.
          Open

          def copy_to_unicode(element):
    """ used to transform the lxml version of unicode to a
    standard version of unicode that can be pickalable -
    necessary for linting """
          Severity: Minor
          Found in scrapi/util.py - About 35 mins to fix

          Cognitive Complexity

          Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

          A method's cognitive complexity is based on a few simple rules:

          • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
          • Code is considered more complex for each "break in the linear flow of the code"
          • Code is considered more complex when "flow breaking structures are nested"

          Further reading

          Function oai_process_uris has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.
          Open

          def oai_process_uris(*args, **kwargs):
    use_doi = kwargs.get('use_doi', False)

    identifiers = gather_identifiers(args)
    provider_uris, object_uris = seperate_provider_object_uris(identifiers)
          Severity: Minor
          Found in scrapi/base/helpers.py - About 35 mins to fix

          Cognitive Complexity

          Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

          A method's cognitive complexity is based on a few simple rules:

          • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
          • Code is considered more complex for each "break in the linear flow of the code"
          • Code is considered more complex when "flow breaking structures are nested"

          Further reading

          Avoid too many return statements within this function.
          Open

                      return self._transform_string(value, doc)
          Severity: Major
          Found in scrapi/base/transformer.py - About 30 mins to fix

            Avoid too many return statements within this function.
            Open

                        return value(doc)
            Severity: Major
            Found in scrapi/base/transformer.py - About 30 mins to fix

              Function setup has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.
              Open

                  def setup(self, force=False, sync=True):
        if self._setup and not force:
            return True

        try:
              Severity: Minor
              Found in scrapi/processing/cassandra.py - About 25 mins to fix

              Cognitive Complexity

              Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

              A method's cognitive complexity is based on a few simple rules:

              • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
              • Code is considered more complex for each "break in the linear flow of the code"
              • Code is considered more complex when "flow breaking structures are nested"

              Further reading

              Severity
              Category
              Status
              Source
              Language