Sign upLogin

apel/apel

View on GitHub
Star
bin/dbunloader.py

Summary

Maintainability
C
1 day
Test Coverage

Avoid too many return statements within this function.
Open

        return RECORDS_PER_MESSAGE_DEFAULT
Severity: Major
Found in bin/dbunloader.py - About 30 mins to fix

    Similar blocks of code found in 2 locations. Consider refactoring.
    Open

            if interval == 'latest':
            msgs, recs = unloader.unload_latest(table_name, send_ur)
        elif interval == 'gap':
            start = cp.get('unloader', 'gap_start')
            end = cp.get('unloader', 'gap_end')
    Severity: Major
    Found in bin/dbunloader.py and 1 other location - About 1 day to fix
    bin/client.py on lines 241..251

    Duplicated Code

    Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

    Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

    When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

    Tuning

    This issue has a mass of 123.

    We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

    The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

    If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

    See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

    Refactorings

    Further Reading

    Identical blocks of code found in 2 locations. Consider refactoring.
    Open

        try:
        if os.path.exists(options.log_config):
            logging.config.fileConfig(options.log_config)
        else:
            set_up_logging(cp.get('logging', 'logfile'),
    Severity: Major
    Found in bin/dbunloader.py and 1 other location - About 1 hr to fix
    bin/dbloader.py on lines 56..68

    Duplicated Code

    Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

    Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

    When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

    Tuning

    This issue has a mass of 41.

    We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

    The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

    If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

    See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

    Refactorings

    Further Reading

    Similar blocks of code found in 4 locations. Consider refactoring.
    Open

        try:
        include      = cp.get('unloader', 'include_vos')
        include_vos  = [ vo.strip() for vo in include.split(',') ]
    Severity: Major
    Found in bin/dbunloader.py and 3 other locations - About 50 mins to fix
    bin/client.py on lines 116..118
    bin/client.py on lines 122..124
    bin/dbunloader.py on lines 156..158

    Duplicated Code

    Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

    Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

    When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

    Tuning

    This issue has a mass of 36.

    We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

    The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

    If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

    See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

    Refactorings

    Further Reading

    Similar blocks of code found in 4 locations. Consider refactoring.
    Open

            try:
            exclude      = cp.get('unloader', 'exclude_vos')
            exclude_vos  = [ vo.strip() for vo in exclude.split(',') ]
    Severity: Major
    Found in bin/dbunloader.py and 3 other locations - About 50 mins to fix
    bin/client.py on lines 116..118
    bin/client.py on lines 122..124
    bin/dbunloader.py on lines 151..153

    Duplicated Code

    Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

    Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

    When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

    Tuning

    This issue has a mass of 36.

    We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

    The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

    If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

    See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

    Refactorings

    Further Reading

    Line too long (97 > 79 characters)
    Open

        opt_parser.add_option('-c', '--config', help='Location of configuration file for dbunloader',
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Module level import not at top of file
    Open

    import logging.config
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Module level import not at top of file
    Open

    from apel.common import set_up_logging
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Module level import not at top of file
    Open

    from apel import __version__
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Multiple spaces before operator
    Open

            include      = cp.get('unloader', 'include_vos')
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Line too long (87 > 79 characters)
    Open

            records_per_message = int(config_object.get('unloader', 'records_per_message'))
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Line too long (107 > 79 characters)
    Open

        opt_parser.add_option('-l', '--log_config', help='Location of logging configuration file for dbloader',
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Multiple spaces before operator
    Open

            withhold_dns     = cp.getboolean('unloader', 'withhold_dns')
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Module level import not at top of file
    Open

    import os
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Multiple spaces before operator
    Open

        table_name       = cp.get('unloader', 'table_name')
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Module level import not at top of file
    Open

    from optparse import OptionParser
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Module level import not at top of file
    Open

    from apel.db.unloader import DbUnloader
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Line too long (94 > 79 characters)
    Open

                'Invalid records_per_message value, must be a postive integer. Defaulting to %d.',
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Module level import not at top of file
    Open

    from apel.db import ApelDb, ApelDbException
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Multiple spaces before operator
    Open

        unload_dir       = cp.get('unloader', 'dir_location')
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Multiple spaces before operator
    Open

        include_vos      = None
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Line too long (81 > 79 characters)
    Open

        """Retrieve the records per message from config, keeping it within limits."""
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Line too long (91 > 79 characters)
    Open

        opt_parser.add_option('-d', '--db', help='location of configuration file for database',
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Module level import not at top of file
    Open

    import sys
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Place imports at the top of the file.

    Always put imports at the top of the file, just after any module
comments and docstrings, and before module globals and constants.

Okay: import os
Okay: # this is a comment\nimport os
Okay: '''this is a module docstring'''\nimport os
Okay: r'''this is a module docstring'''\nimport os
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nelse:\n\tpass\nimport y
Okay:
try:\n\timport x\nexcept ImportError:\n\tpass\nfinally:\n\tpass\nimport y
E402: a=1\nimport os
E402: 'One string'\n"Two string"\nimport os
E402: a=1\nfrom sys import x

Okay: if x:\n    import os

    Multiple spaces before operator
    Open

        exclude_vos      = None
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Line too long (86 > 79 characters)
    Open

            log.info('%d records in %d messages unloaded from %s', recs, msgs, table_name)
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Whitespace before ']'
    Open

                exclude_vos  = [ vo.strip() for vo in exclude.split(',') ]
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace.

    Avoid extraneous whitespace in these situations:
- Immediately inside parentheses, brackets or braces.
- Immediately before a comma, semicolon, or colon.

Okay: spam(ham[1], {eggs: 2})
E201: spam( ham[1], {eggs: 2})
E201: spam(ham[ 1], {eggs: 2})
E201: spam(ham[1], { eggs: 2})
E202: spam(ham[1], {eggs: 2} )
E202: spam(ham[1 ], {eggs: 2})
E202: spam(ham[1], {eggs: 2 })

E203: if x == 4: print x, y; x, y = y , x
E203: if x == 4: print x, y ; x, y = y, x
E203: if x == 4 : print x, y; x, y = y, x

    Line too long (93 > 79 characters)
    Open

        unloader = DbUnloader(db, unload_dir, include_vos, exclude_vos, local_jobs, withhold_dns)
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Limit all lines to a maximum of 79 characters.

    There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

    Whitespace after '['
    Open

            include_vos  = [ vo.strip() for vo in include.split(',') ]
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace.

    Avoid extraneous whitespace in these situations:
- Immediately inside parentheses, brackets or braces.
- Immediately before a comma, semicolon, or colon.

Okay: spam(ham[1], {eggs: 2})
E201: spam( ham[1], {eggs: 2})
E201: spam(ham[ 1], {eggs: 2})
E201: spam(ham[1], { eggs: 2})
E202: spam(ham[1], {eggs: 2} )
E202: spam(ham[1 ], {eggs: 2})
E202: spam(ham[1], {eggs: 2 })

E203: if x == 4: print x, y; x, y = y , x
E203: if x == 4: print x, y ; x, y = y, x
E203: if x == 4 : print x, y; x, y = y, x

    Whitespace after '['
    Open

                exclude_vos  = [ vo.strip() for vo in exclude.split(',') ]
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace.

    Avoid extraneous whitespace in these situations:
- Immediately inside parentheses, brackets or braces.
- Immediately before a comma, semicolon, or colon.

Okay: spam(ham[1], {eggs: 2})
E201: spam( ham[1], {eggs: 2})
E201: spam(ham[ 1], {eggs: 2})
E201: spam(ham[1], { eggs: 2})
E202: spam(ham[1], {eggs: 2} )
E202: spam(ham[1 ], {eggs: 2})
E202: spam(ham[1], {eggs: 2 })

E203: if x == 4: print x, y; x, y = y , x
E203: if x == 4: print x, y ; x, y = y, x
E203: if x == 4 : print x, y; x, y = y, x

    Multiple spaces before operator
    Open

                exclude      = cp.get('unloader', 'exclude_vos')
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Multiple spaces before operator
    Open

                exclude_vos  = [ vo.strip() for vo in exclude.split(',') ]
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Multiple spaces before operator
    Open

            include_vos  = [ vo.strip() for vo in include.split(',') ]
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace around an operator.

    Okay: a = 12 + 3
E221: a = 4  + 5
E222: a = 4 +  5
E223: a = 4\t+ 5
E224: a = 4 +\t5

    Whitespace before ']'
    Open

            include_vos  = [ vo.strip() for vo in include.split(',') ]
    Severity: Minor
    Found in bin/dbunloader.py by pep8

    Avoid extraneous whitespace.

    Avoid extraneous whitespace in these situations:
- Immediately inside parentheses, brackets or braces.
- Immediately before a comma, semicolon, or colon.

Okay: spam(ham[1], {eggs: 2})
E201: spam( ham[1], {eggs: 2})
E201: spam(ham[ 1], {eggs: 2})
E201: spam(ham[1], { eggs: 2})
E202: spam(ham[1], {eggs: 2} )
E202: spam(ham[1 ], {eggs: 2})
E202: spam(ham[1], {eggs: 2 })

E203: if x == 4: print x, y; x, y = y , x
E203: if x == 4: print x, y ; x, y = y, x
E203: if x == 4 : print x, y; x, y = y, x

    There are no issues that match your filters.

    Category
    Status