Sign upLogin

fedspendingtransparency/usaspending-api

View on GitHub
Star
usaspending_api/common/etl/spark.py

Summary

Maintainability
C
1 day
Test Coverage
D
69%

File spark.py has 639 lines of code (exceeds 250 allowed). Consider refactoring.
Open

"""
Spark utility functions that could be used as stages or steps of an ETL job (aka "data pipeline")

NOTE: This is distinguished from the usaspending_api.common.helpers.spark_helpers module, which holds mostly boilerplate
functions for setup and configuration of the spark environment
Severity: Major
Found in usaspending_api/common/etl/spark.py - About 1 day to fix

    Function extract_db_data_frame has 10 arguments (exceeds 6 allowed). Consider refactoring.
    Wontfix

    def extract_db_data_frame(
    Severity: Major
    Found in usaspending_api/common/etl/spark.py - About 1 hr to fix

      Function write_csv_file has 7 arguments (exceeds 6 allowed). Consider refactoring.
      Open

      def write_csv_file(
      Severity: Major
      Found in usaspending_api/common/etl/spark.py - About 35 mins to fix

        There are no issues that match your filters.

        Category
        Status