Sign upLogin

toodef/neural-pipeline

View on GitHub
Star
neural_pipeline/data_producer/data_producer.py

Summary

Maintainability
A
0 mins
Test Coverage

Line too long (92 > 79 characters)
Open

    def set_indices(self, indices: [int], remove_unused: bool = False) -> 'AbstractDataset':
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (94 > 79 characters)
Open

    def __init__(self, dataset: [AbstractDataset], batch_size: int = 1, num_workers: int = 0):
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (85 > 79 characters)
Open

            return dict(data, **{'data_idx': str(dataset_idx) + "_" + str(data_idx)})
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (125 > 79 characters)
Open

        If ``indices`` is specified - DataLoader will output data only by this indices. In this case indices will not passed.
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (104 > 79 characters)
Open

        Is need global shuffling. If global shuffling enable - batches will compile from random indices.
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (157 > 79 characters)
Open

        Is need to pin memory on loading. Pinning memory was increase data loading performance (especially when data loads to GPU) but incompatible with swap
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (96 > 79 characters)
Open

    Data Producer. Accumulate one or more datasets and pass it's data by batches for processing.
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (118 > 79 characters)
Open

        :param indices: list of indices. Each item of list is a string in format '{}_{}'.format(dataset_idx, data_idx)
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (100 > 79 characters)
Open

        return DataLoader(self._dataset, batch_size=self._batch_size, num_workers=self._num_workers,
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (133 > 79 characters)
Open

        Interpret one item from dataset. This method get index of item and returns interpreted data? that will be passed from dataset
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (88 > 79 characters)
Open

    def load_indices(self, path: str, remove_unused: bool = False) -> 'AbstractDataset':
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (111 > 79 characters)
Open

                          shuffle=self._glob_shuffle, pin_memory=self._pin_memory, collate_fn=self._collate_fn,
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Expected an indented block
Open

class BasicDataset(AbstractDataset, metaclass=ABCMeta):
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Use indent_size (PEP8 says 4) spaces per indentation level.

For really old code that you don't want to mess up, you can continue
to use 8-space tabs.

Okay: a = 1
Okay: if a == 0:\n    a = 1
E111:   a = 1
E114:   # a = 1

Okay: for item in items:\n    pass
E112: for item in items:\npass
E115: for item in items:\n# Hi\n    pass

Okay: a = 1\nb = 2
E113: a = 1\n    b = 2
E116: a = 1\n    # b = 2

Line too long (85 > 79 characters)
Open

        Get PyTorch :class:`DataLoader` object, that aggregate :class:`DataProducer`.
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (120 > 79 characters)
Open

    :param datasets: list of datasets. Every dataset might be iterable (contans methods ``__getitem__`` and ``__len__``)
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Missing whitespace after ':'
Open

    def __init__(self, dataset:AbstractDataset, indices: list):
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Each comma, semicolon or colon should be followed by whitespace.

Okay: [a, b]
Okay: (3,)
Okay: a[1:4]
Okay: a[:4]
Okay: a[1:]
Okay: a[1:4:2]
E231: ['a','b']
E231: foo(bar,baz)
E231: [{'a':'b'}]

Line too long (96 > 79 characters)
Open

    :param num_workers: number of processes, that load data from datasets and pass it for output
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (111 > 79 characters)
Open

                          shuffle=self._glob_shuffle, pin_memory=self._pin_memory, collate_fn=self._collate_fn,
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Expected 2 blank lines, found 1
Open

class BasicDataset(AbstractDataset, metaclass=ABCMeta):
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Separate top-level function and class definitions with two blank lines.

Method definitions inside a class are separated by a single blank
line.

Extra blank lines may be used (sparingly) to separate groups of
related functions.  Blank lines may be omitted between a bunch of
related one-liners (e.g. a set of dummy implementations).

Use blank lines in functions, sparingly, to indicate logical
sections.

Okay: def a():\n    pass\n\n\ndef b():\n    pass
Okay: def a():\n    pass\n\n\nasync def b():\n    pass
Okay: def a():\n    pass\n\n\n# Foo\n# Bar\n\ndef b():\n    pass
Okay: default = 1\nfoo = 1
Okay: classify = 1\nfoo = 1

E301: class Foo:\n    b = 0\n    def bar():\n        pass
E302: def a():\n    pass\n\ndef b(n):\n    pass
E302: def a():\n    pass\n\nasync def b(n):\n    pass
E303: def a():\n    pass\n\n\n\ndef b(n):\n    pass
E303: def a():\n\n\n\n    pass
E304: @decorator\n\ndef a():\n    pass
E305: def a():\n    pass\na()
E306: def a():\n    def b():\n        pass\n    def c():\n        pass

Line too long (121 > 79 characters)
Open

        return DataLoader(_ByIndices(self._dataset, indices), batch_size=self._batch_size, num_workers=self._num_workers,
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (103 > 79 characters)
Open

    Basic dataset get array of items and works with it. Array of items is just an array of shape [N, ?]
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

Line too long (91 > 79 characters)
Open

    This use PyTorch builtin :class:`DataLoader` for increase performance of data delivery.
Severity: Minor
Found in neural_pipeline/data_producer/data_producer.py by pep8

Limit all lines to a maximum of 79 characters.

There are still many devices around that are limited to 80 character
lines; plus, limiting windows to 80 characters makes it possible to
have several windows side-by-side.  The default wrapping on such
devices looks ugly.  Therefore, please limit all lines to a maximum
of 79 characters. For flowing long blocks of text (docstrings or
comments), limiting the length to 72 characters is recommended.

Reports error E501.

There are no issues that match your filters.

Category
Status