Sign upLogin

fedspendingtransparency/usaspending-api

View on GitHub
Star

Showing 156 of 218 total issues

Identical blocks of code found in 2 locations. Consider refactoring.
Open

def build_spending_update_query(query_base, update_data):
    values_string = ""
    for count, row in enumerate(update_data, 1):
        values_string += "("
        values_string += ",".join(["%s"] * len(row))
Severity: Major
Found in usaspending_api/database_scripts/job_archive/backfill_per_transaction_exec_comp.py and 1 other location - About 5 hrs to fix
usaspending_api/database_scripts/job_archive/backfill_solicitation_date.py on lines 99..107

Duplicated Code

Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

Tuning

This issue has a mass of 95.

We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

Refactorings

Further Reading

Identical blocks of code found in 2 locations. Consider refactoring.
Open

def build_spending_update_query(query_base, update_data):
    values_string = ""
    for count, row in enumerate(update_data, 1):
        values_string += "("
        values_string += ",".join(["%s"] * len(row))
Severity: Major
Found in usaspending_api/database_scripts/job_archive/backfill_solicitation_date.py and 1 other location - About 5 hrs to fix
usaspending_api/database_scripts/job_archive/backfill_per_transaction_exec_comp.py on lines 203..211

Duplicated Code

Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

Tuning

This issue has a mass of 95.

We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

Refactorings

Further Reading

File recipients.py has 405 lines of code (exceeds 250 allowed). Consider refactoring.
Open

import logging
import uuid

from decimal import Decimal
Severity: Minor
Found in usaspending_api/recipient/v2/views/recipients.py - About 5 hrs to fix

    File transaction_fpds.py has 402 lines of code (exceeds 250 allowed). Consider refactoring.
    Open

    from usaspending_api.common.data_classes import TransactionColumn

TRANSACTION_FPDS_COLUMN_INFO = [
    TransactionColumn("a_76_fair_act_action", "a_76_fair_act_action", "STRING"),
    TransactionColumn("a_76_fair_act_action_desc", "a_76_fair_act_action_desc", "STRING"),
    Severity: Minor
    Found in usaspending_api/transactions/delta_models/transaction_fpds.py - About 5 hrs to fix

      File disaster_covid19_file_d1_awards.py has 391 lines of code (exceeds 250 allowed). Consider refactoring.
      Open

      d1_awards_sql_string = """
SELECT
    award_search.generated_unique_award_id AS contract_award_unique_key,
    award_search.piid AS award_id_piid,
    latest_transaction.referenced_idv_agency_iden AS parent_award_agency_id,
      Severity: Minor
      Found in usaspending_api/disaster/management/sql/spark/disaster_covid19_file_d1_awards.py - About 5 hrs to fix

        File test_award_index_elasticsearch.py has 388 lines of code (exceeds 250 allowed). Consider refactoring.
        Open

        import pytest

from model_bakery import baker
        Severity: Minor
        Found in usaspending_api/tests/integration/test_award_index_elasticsearch.py - About 5 hrs to fix

          File disaster_base.py has 382 lines of code (exceeds 250 allowed). Consider refactoring.
          Open

          import json

from datetime import date
from django.db.models import Case, Count, DecimalField, F, Max, Q, Sum, Value, When
from django.db.models.functions import Coalesce
          Severity: Minor
          Found in usaspending_api/disaster/v2/views/disaster_base.py - About 5 hrs to fix

            Cyclomatic complexity is too high in function type_filter. (41)
            Open

            def type_filter(_type, filters, limit=None):
    _types = [
        "budget_function",
        "budget_subfunction",
        "federal_account",
            Severity: Minor
            Found in usaspending_api/spending_explorer/v2/filters/type_filter.py by radon

            Cyclomatic Complexity

            Cyclomatic Complexity corresponds to the number of decisions a block of code contains plus 1. This number (also called McCabe number) is equal to the number of linearly independent paths through the code. This number can be used as a guide when testing conditional logic in blocks.

            Radon analyzes the AST tree of a Python program to compute Cyclomatic Complexity. Statements have the following effects on Cyclomatic Complexity:

            Construct Effect on CC Reasoning
            if +1 An if statement is a single decision.
            elif +1 The elif statement adds another decision.
            else +0 The else statement does not cause a new decision. The decision is at the if.
            for +1 There is a decision at the start of the loop.
            while +1 There is a decision at the while statement.
            except +1 Each except branch adds a new conditional path of execution.
            finally +0 The finally block is unconditionally executed.
            with +1 The with statement roughly corresponds to a try/except block (see PEP 343 for details).
            assert +1 The assert statement internally roughly equals a conditional statement.
            Comprehension +1 A list/set/dict comprehension of generator expression is equivalent to a for loop.
            Boolean Operator +1 Every boolean operator (and, or) adds a decision point.

            Source: http://radon.readthedocs.org/en/latest/intro.html

            File daims_maps.py has 381 lines of code (exceeds 250 allowed). Consider refactoring.
            Open

            """
This is a map of code enumerations, extracted from the DAIMS.

Each should be named following this pattern:
    - For a field that is unique among all models, <field>_map
            Severity: Minor
            Found in usaspending_api/data/daims_maps.py - About 5 hrs to fix

              File field_mappings_fpds.py has 378 lines of code (exceeds 250 allowed). Consider refactoring.
              Open

              from usaspending_api.broker.helpers.award_category_helper import award_types
from usaspending_api.etl.transaction_loaders.derived_field_functions_fpds import (
    calculate_fiscal_year,
    calculate_awarding_agency,
    calculate_funding_agency,
              Severity: Minor
              Found in usaspending_api/etl/transaction_loaders/field_mappings_fpds.py - About 5 hrs to fix

                Function get_download_sources has a Cognitive Complexity of 43 (exceeds 15 allowed). Consider refactoring.
                Open

                def get_download_sources(
    json_request: dict, download_job: DownloadJob = None, origination: Optional[str] = None
) -> List[DownloadSource]:
    download_sources = []
    for download_type in json_request["download_types"]:
                Severity: Minor
                Found in usaspending_api/download/filestreaming/download_generation.py - About 4 hrs to fix

                Cognitive Complexity

                Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

                A method's cognitive complexity is based on a few simple rules:

                • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
                • Code is considered more complex for each "break in the linear flow of the code"
                • Code is considered more complex when "flow breaking structures are nested"

                Further reading

                Cyclomatic complexity is too high in method build_elasticsearch_result. (39)
                Open

                    def build_elasticsearch_result(self, response: dict) -> Dict[str, dict]:
        results = {}
        geo_info_buckets = response.get("group_by_agg_key", {}).get("buckets", [])

        for bucket in geo_info_buckets:
                Severity: Minor
                Found in usaspending_api/disaster/v2/views/spending_by_geography.py by radon

                Cyclomatic Complexity

                Cyclomatic Complexity corresponds to the number of decisions a block of code contains plus 1. This number (also called McCabe number) is equal to the number of linearly independent paths through the code. This number can be used as a guide when testing conditional logic in blocks.

                Radon analyzes the AST tree of a Python program to compute Cyclomatic Complexity. Statements have the following effects on Cyclomatic Complexity:

                Construct Effect on CC Reasoning
                if +1 An if statement is a single decision.
                elif +1 The elif statement adds another decision.
                else +0 The else statement does not cause a new decision. The decision is at the if.
                for +1 There is a decision at the start of the loop.
                while +1 There is a decision at the while statement.
                except +1 Each except branch adds a new conditional path of execution.
                finally +0 The finally block is unconditionally executed.
                with +1 The with statement roughly corresponds to a try/except block (see PEP 343 for details).
                assert +1 The assert statement internally roughly equals a conditional statement.
                Comprehension +1 A list/set/dict comprehension of generator expression is equivalent to a for loop.
                Boolean Operator +1 Every boolean operator (and, or) adds a decision point.

                Source: http://radon.readthedocs.org/en/latest/intro.html

                Function process_data has a Cognitive Complexity of 42 (exceeds 15 allowed). Consider refactoring.
                Open

                    def process_data(self):
        # Make sure we weren't flagged to exit
        while True:
            row = self.data_queue.get()
            # If row is none, we need to die.
                Severity: Minor
                Found in usaspending_api/common/threaded_data_loader.py - About 4 hrs to fix

                Cognitive Complexity

                Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

                A method's cognitive complexity is based on a few simple rules:

                • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
                • Code is considered more complex for each "break in the linear flow of the code"
                • Code is considered more complex when "flow breaking structures are nested"

                Further reading

                Cyclomatic complexity is too high in method construct_es_response_for_prime_awards. (38)
                Open

                    def construct_es_response_for_prime_awards(self, response) -> dict:
        results = []
        should_return_display_award_id = "Award ID" in self.fields
        should_return_recipient_id = "recipient_id" in self.fields
        for res in response:
                Severity: Minor
                Found in usaspending_api/search/v2/views/spending_by_award.py by radon

                Cyclomatic Complexity

                Cyclomatic Complexity corresponds to the number of decisions a block of code contains plus 1. This number (also called McCabe number) is equal to the number of linearly independent paths through the code. This number can be used as a guide when testing conditional logic in blocks.

                Radon analyzes the AST tree of a Python program to compute Cyclomatic Complexity. Statements have the following effects on Cyclomatic Complexity:

                Construct Effect on CC Reasoning
                if +1 An if statement is a single decision.
                elif +1 The elif statement adds another decision.
                else +0 The else statement does not cause a new decision. The decision is at the if.
                for +1 There is a decision at the start of the loop.
                while +1 There is a decision at the while statement.
                except +1 Each except branch adds a new conditional path of execution.
                finally +0 The finally block is unconditionally executed.
                with +1 The with statement roughly corresponds to a try/except block (see PEP 343 for details).
                assert +1 The assert statement internally roughly equals a conditional statement.
                Comprehension +1 A list/set/dict comprehension of generator expression is equivalent to a for loop.
                Boolean Operator +1 Every boolean operator (and, or) adds a decision point.

                Source: http://radon.readthedocs.org/en/latest/intro.html

                Cyclomatic complexity is too high in function _build_usas_data_for_spark. (37)
                Open

                def _build_usas_data_for_spark():
    """Create Django model data to be saved to Postgres, which will then get ingested into Delta Lake"""
    # Create recipient data for two transactions; the other two will generate ad hoc
    baker.make(
        "recipient.RecipientLookup",
                Severity: Minor
                Found in usaspending_api/tests/conftest_spark.py by radon

                Cyclomatic Complexity

                Cyclomatic Complexity corresponds to the number of decisions a block of code contains plus 1. This number (also called McCabe number) is equal to the number of linearly independent paths through the code. This number can be used as a guide when testing conditional logic in blocks.

                Radon analyzes the AST tree of a Python program to compute Cyclomatic Complexity. Statements have the following effects on Cyclomatic Complexity:

                Construct Effect on CC Reasoning
                if +1 An if statement is a single decision.
                elif +1 The elif statement adds another decision.
                else +0 The else statement does not cause a new decision. The decision is at the if.
                for +1 There is a decision at the start of the loop.
                while +1 There is a decision at the while statement.
                except +1 Each except branch adds a new conditional path of execution.
                finally +0 The finally block is unconditionally executed.
                with +1 The with statement roughly corresponds to a try/except block (see PEP 343 for details).
                assert +1 The assert statement internally roughly equals a conditional statement.
                Comprehension +1 A list/set/dict comprehension of generator expression is equivalent to a for loop.
                Boolean Operator +1 Every boolean operator (and, or) adds a decision point.

                Source: http://radon.readthedocs.org/en/latest/intro.html

                File settings.py has 353 lines of code (exceeds 250 allowed). Consider refactoring.
                Open

                """
For more information on this file: https://docs.djangoproject.com/en/3.2/topics/settings/
For the full list of settings and their values: https://docs.djangoproject.com/en/3.2/ref/settings/
"""
                Severity: Minor
                Found in usaspending_api/settings.py - About 4 hrs to fix

                  Function handle has a Cognitive Complexity of 40 (exceeds 15 allowed). Consider refactoring.
                  Open

                      def handle(self, *args, **options):
        """Run the application."""

        # Make sure
        #   settings.BULK_DOWNLOAD_S3_BUCKET_NAME
                  Severity: Minor
                  Found in usaspending_api/download/management/commands/populate_monthly_files.py - About 4 hrs to fix

                  Cognitive Complexity

                  Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

                  A method's cognitive complexity is based on a few simple rules:

                  • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
                  • Code is considered more complex for each "break in the linear flow of the code"
                  • Code is considered more complex when "flow breaking structures are nested"

                  Further reading

                  Cyclomatic complexity is too high in method post. (36)
                  Open

                      @cache_response()
    def post(self, request, pk, format=None):
        # create response
        response = {"results": {}}
                  Severity: Minor
                  Found in usaspending_api/accounts/views/federal_accounts_v2.py by radon

                  Cyclomatic Complexity

                  Cyclomatic Complexity corresponds to the number of decisions a block of code contains plus 1. This number (also called McCabe number) is equal to the number of linearly independent paths through the code. This number can be used as a guide when testing conditional logic in blocks.

                  Radon analyzes the AST tree of a Python program to compute Cyclomatic Complexity. Statements have the following effects on Cyclomatic Complexity:

                  Construct Effect on CC Reasoning
                  if +1 An if statement is a single decision.
                  elif +1 The elif statement adds another decision.
                  else +0 The else statement does not cause a new decision. The decision is at the if.
                  for +1 There is a decision at the start of the loop.
                  while +1 There is a decision at the while statement.
                  except +1 Each except branch adds a new conditional path of execution.
                  finally +0 The finally block is unconditionally executed.
                  with +1 The with statement roughly corresponds to a try/except block (see PEP 343 for details).
                  assert +1 The assert statement internally roughly equals a conditional statement.
                  Comprehension +1 A list/set/dict comprehension of generator expression is equivalent to a for loop.
                  Boolean Operator +1 Every boolean operator (and, or) adds a decision point.

                  Source: http://radon.readthedocs.org/en/latest/intro.html

                  Cyclomatic complexity is too high in class SpendingOverTimeFederalAccountsViewSet. (36)
                  Open

                  class SpendingOverTimeFederalAccountsViewSet(APIView):
    """
    This route takes a federal_account DB ID and returns the data required to visualized the spending over time graphic.
    """
                  Severity: Minor
                  Found in usaspending_api/accounts/views/federal_accounts_v2.py by radon

                  Cyclomatic Complexity

                  Cyclomatic Complexity corresponds to the number of decisions a block of code contains plus 1. This number (also called McCabe number) is equal to the number of linearly independent paths through the code. This number can be used as a guide when testing conditional logic in blocks.

                  Radon analyzes the AST tree of a Python program to compute Cyclomatic Complexity. Statements have the following effects on Cyclomatic Complexity:

                  Construct Effect on CC Reasoning
                  if +1 An if statement is a single decision.
                  elif +1 The elif statement adds another decision.
                  else +0 The else statement does not cause a new decision. The decision is at the if.
                  for +1 There is a decision at the start of the loop.
                  while +1 There is a decision at the while statement.
                  except +1 Each except branch adds a new conditional path of execution.
                  finally +0 The finally block is unconditionally executed.
                  with +1 The with statement roughly corresponds to a try/except block (see PEP 343 for details).
                  assert +1 The assert statement internally roughly equals a conditional statement.
                  Comprehension +1 A list/set/dict comprehension of generator expression is equivalent to a for loop.
                  Boolean Operator +1 Every boolean operator (and, or) adds a decision point.

                  Source: http://radon.readthedocs.org/en/latest/intro.html

                  File elasticsearch_indexer.py has 347 lines of code (exceeds 250 allowed). Consider refactoring.
                  Open

                  import logging
from abc import ABC, abstractmethod
from datetime import datetime, timezone
from time import perf_counter
                  Severity: Minor
                  Found in usaspending_api/etl/management/commands/elasticsearch_indexer.py - About 4 hrs to fix
                    Severity
                    Category
                    Status
                    Source
                    Language