NationalGenomicsInfrastructure/ngi_pipeline

View on GitHub
ngi_pipeline/engines/piper_ngi/local_process_tracking.py

Summary

Maintainability
F
6 days
Test Coverage

Function update_charon_with_local_jobs_status has a Cognitive Complexity of 100 (exceeds 5 allowed). Consider refactoring.
Open

def update_charon_with_local_jobs_status(quiet=False, config=None, config_file_path=None):
    """Check the status of all locally-tracked jobs and update Charon accordingly.
    """
    if quiet and not config.get("quiet"):
        config['quiet'] = True
Severity: Minor
Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 2 days to fix

Cognitive Complexity

Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

A method's cognitive complexity is based on a few simple rules:

  • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
  • Code is considered more complex for each "break in the linear flow of the code"
  • Code is considered more complex when "flow breaking structures are nested"

Further reading

File local_process_tracking.py has 542 lines of code (exceeds 250 allowed). Consider refactoring.
Open

import glob
import inspect
import os
import psutil
import re
Severity: Major
Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 1 day to fix

    Function update_coverage_for_sample_seqruns has a Cognitive Complexity of 21 (exceeds 5 allowed). Consider refactoring.
    Open

    def update_coverage_for_sample_seqruns(project_id, sample_id, piper_qc_dir,
                                           config=None, config_file_path=None):
        """Find all the valid seqruns for a particular sample, parse their
        qualimap output files, and update Charon with the mean autosomal
        coverage for each.
    Severity: Minor
    Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 2 hrs to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Function kill_running_sample_analysis has a Cognitive Complexity of 14 (exceeds 5 allowed). Consider refactoring.
    Open

    def kill_running_sample_analysis(workflow_subtask, project_id, sample_id):
        """Determine if a sample is currently being analyzed by accessing the local
        process tracking database."""
        sample_run_name = "{}/{}".format(project_id, sample_id)
        LOG.info('Attempting to kill sample analysis run "{}"'.format(sample_run_name))
    Severity: Minor
    Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 1 hr to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Function record_process_sample has a Cognitive Complexity of 11 (exceeds 5 allowed). Consider refactoring.
    Open

    def record_process_sample(project, sample, workflow_subtask, analysis_module_name,
                              process_id=None, slurm_job_id=None, config=None, config_file_path=None):
        LOG.info('Recording slurm job id "{}" for project "{}", sample "{}", '
                 'workflow "{}"'.format(slurm_job_id, project, sample, workflow_subtask))
        with get_db_session() as session:
    Severity: Minor
    Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 1 hr to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Function record_process_sample has 8 arguments (exceeds 4 allowed). Consider refactoring.
    Open

    def record_process_sample(project, sample, workflow_subtask, analysis_module_name,
    Severity: Major
    Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 1 hr to fix

      Function get_exit_code has 7 arguments (exceeds 4 allowed). Consider refactoring.
      Open

      def get_exit_code(workflow_name, project_base_path, project_name, project_id,
      Severity: Major
      Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 50 mins to fix

        Avoid deeply nested control flow statements.
        Open

                                    if not config.get('quiet'):
                                        mail_analysis(project_name=project_name, sample_name=sample_id,
                                                      engine_name=engine, level="ERROR",
                                                      workflow=workflow, info_text=error_text)
                    except CharonError as e:
        Severity: Major
        Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

          Avoid deeply nested control flow statements.
          Open

                                  if not config.get('quiet'):
                                      mail_analysis(project_name=project_name,
                                                    sample_name=sample_id,
                                                    engine_name=engine, level="ERROR",
                                                    info_text=error_text,
          Severity: Major
          Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

            Avoid deeply nested control flow statements.
            Open

                                    if not psutil.pid_exists(process_id):
                                        # Job did not write an exit code and is also not running
                                        JOB_FAILED = True
                                if JOB_FAILED:
            Severity: Major
            Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

              Avoid deeply nested control flow statements.
              Open

                                      if workflow == "merge_process_variantcall":
                                          sample_status_field = "analysis_status"
                                          seqrun_status_field = "alignment_status"
                                          recurse_status = "RUNNING"
                                      elif workflow == "genotype_concordance":
              Severity: Major
              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

                Avoid deeply nested control flow statements.
                Open

                                        if slurm_exit_code is not None: # "None" indicates job is still running
                                            JOB_FAILED = True
                                    else:
                Severity: Major
                Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

                  Avoid deeply nested control flow statements.
                  Open

                                              if charon_status and not charon_status == set_status:
                                                  LOG.warning('Tracking inconsistency for {}: Charon status '
                                                           'for field "{}" is "{}" but local process tracking '
                                                           'database indicates it is running. Setting value '
                                                           'in Charon to {}.'.format(label, sample_status_field,
                  Severity: Major
                  Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

                    Avoid deeply nested control flow statements.
                    Open

                                            if slurm_job_id:
                                                exit_code_file_path = \
                                                    create_exit_code_file_path(workflow_subtask=workflow,
                                                                               project_base_path=project_base_path,
                                                                               project_name=project_name,
                    Severity: Major
                    Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

                      Avoid deeply nested control flow statements.
                      Open

                                              if workflow == "merge_process_variantcall":
                                                  sample_status_field = "analysis_status"
                                                  seqrun_status_field = "alignment_status"
                                              elif workflow == "genotype_concordance":
                                                  sample_status_field = seqrun_status_field = "genotype_status"
                      Severity: Major
                      Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 45 mins to fix

                        Function update_sample_duplication_and_coverage has 5 arguments (exceeds 4 allowed). Consider refactoring.
                        Open

                        def update_sample_duplication_and_coverage(project_id, sample_id, project_base_path,
                        Severity: Minor
                        Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 35 mins to fix

                          Function update_gtc_for_sample has 5 arguments (exceeds 4 allowed). Consider refactoring.
                          Open

                          def update_gtc_for_sample(project_id, sample_id, piper_gtc_path, config=None, config_file_path=None):
                          Severity: Minor
                          Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 35 mins to fix

                            Function update_coverage_for_sample_seqruns has 5 arguments (exceeds 4 allowed). Consider refactoring.
                            Open

                            def update_coverage_for_sample_seqruns(project_id, sample_id, piper_qc_dir,
                            Severity: Minor
                            Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py - About 35 mins to fix

                              Similar blocks of code found in 2 locations. Consider refactoring.
                              Open

                                                      try:
                                                          remote_sample=charon_session.sample_get(projectid=project_id, sampleid=sample_id)
                                                          charon_status = remote_sample.get(sample_status_field)
                                                          if charon_status and not charon_status == set_status:
                                                              LOG.warning('Tracking inconsistency for {}: Charon status '
                              Severity: Major
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 3 hrs to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 254..259

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 65.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              Similar blocks of code found in 2 locations. Consider refactoring.
                              Open

                                          except CharonError as e:
                                              error_text = ('Unable to update Charon for {}: '
                                                            '{}'.format(label, e))
                                              LOG.error(error_text)
                                              if not config.get('quiet'):
                              Severity: Major
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 3 hrs to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 230..251

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 65.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              Similar blocks of code found in 2 locations. Consider refactoring.
                              Open

                                                      if workflow == "merge_process_variantcall":
                                                          sample_status_field = "analysis_status"
                                                          seqrun_status_field = "alignment_status"
                                                          recurse_status = "RUNNING"
                                                      elif workflow == "genotype_concordance":
                              Severity: Major
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 1 hr to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 90..96

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 47.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              Similar blocks of code found in 2 locations. Consider refactoring.
                              Open

                                                  if workflow == "merge_process_variantcall":
                                                      sample_status_field = "analysis_status"
                                                      seqrun_status_field = "alignment_status"
                                                      set_status = "ANALYZED" # sample level
                                                  elif workflow == "genotype_concordance":
                              Severity: Major
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 1 hr to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 223..229

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 47.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              Identical blocks of code found in 2 locations. Consider refactoring.
                              Open

                                                      if workflow == "merge_process_variantcall":
                                                          sample_status_field = "analysis_status"
                                                          seqrun_status_field = "alignment_status"
                                                      elif workflow == "genotype_concordance":
                                                          sample_status_field = seqrun_status_field = "genotype_status"
                              Severity: Minor
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 55 mins to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 159..163

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 37.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              Identical blocks of code found in 2 locations. Consider refactoring.
                              Open

                                                  if workflow == "merge_process_variantcall":
                                                      sample_status_field = "analysis_status"
                                                      seqrun_status_field = "alignment_status"
                                                  elif workflow == "genotype_concordance":
                                                      sample_status_field = seqrun_status_field = "genotype_status"
                              Severity: Minor
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 55 mins to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 206..210

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 37.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              Identical blocks of code found in 2 locations. Consider refactoring.
                              Open

                                              if not config.get('quiet'):
                                                  mail_analysis(project_name=project_id, sample_name=sample_id,
                              Severity: Minor
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 45 mins to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 349..350

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 35.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              Identical blocks of code found in 2 locations. Consider refactoring.
                              Open

                                      if not config.get('quiet'):
                                          mail_analysis(project_name=project_id, sample_name=sample_id,
                              Severity: Minor
                              Found in ngi_pipeline/engines/piper_ngi/local_process_tracking.py and 1 other location - About 45 mins to fix
                              ngi_pipeline/engines/piper_ngi/local_process_tracking.py on lines 402..403

                              Duplicated Code

                              Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

                              Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

                              When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

                              Tuning

                              This issue has a mass of 35.

                              We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

                              The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

                              If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

                              See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

                              Refactorings

                              Further Reading

                              There are no issues that match your filters.

                              Category
                              Status