crawlkit/crawlkit

View on GitHub

Showing 22 of 22 total issues

Function exports has 127 lines of code (exceeds 25 allowed). Consider refactoring.
Open

module.exports = (scope, logger, runners, workerLogPrefix) => (cb) => {
  logger.debug('Trying to run page runners.');

  if (runners.size === 0) {
    logger.debug('No runners defined');
Severity: Major
Found in src/worker/steps/pageRunners.js - About 5 hrs to fix

    Function nextRunner has 111 lines of code (exceeds 25 allowed). Consider refactoring.
    Open

      const nextRunner = () => {
        if (done.called) {
          logger.debug('Callback was called already.');
          return;
        }
    Severity: Major
    Found in src/worker/steps/pageRunners.js - About 4 hrs to fix

      Function exports has a Cognitive Complexity of 25 (exceeds 5 allowed). Consider refactoring.
      Open

      module.exports = (scope, logger, runners, workerLogPrefix) => (cb) => {
        logger.debug('Trying to run page runners.');
      
        if (runners.size === 0) {
          logger.debug('No runners defined');
      Severity: Minor
      Found in src/worker/steps/pageRunners.js - About 3 hrs to fix

      Cognitive Complexity

      Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

      A method's cognitive complexity is based on a few simple rules:

      • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
      • Code is considered more complex for each "break in the linear flow of the code"
      • Code is considered more complex when "flow breaking structures are nested"

      Further reading

      Function exports has 64 lines of code (exceeds 25 allowed). Consider refactoring.
      Open

      module.exports = (scope, logger, finder, finderParameters, addUrl) => (cb) => {
        logger.debug('Trying to run finder.');
        if (!finder) {
          logger.debug('No finder defined.');
          cb();
      Severity: Major
      Found in src/worker/steps/findLinks.js - About 2 hrs to fix

        CrawlKit has 22 functions (exceeds 20 allowed). Consider refactoring.
        Open

        class CrawlKit {
        
          /**
           * Create a CrawlKit instance
           * @constructor
        Severity: Minor
        Found in src/index.js - About 2 hrs to fix

          Function exports has 57 lines of code (exceeds 25 allowed). Consider refactoring.
          Open

          module.exports = (crawlerInstance, writeResult, runnerKey, finderKey) => {
            const suffix = crawlerInstance.name ? `:${crawlerInstance.name}` : '';
            const prefix = `crawlkit${suffix}`;
            const logger = l(prefix);
            logger.info(`
          Severity: Major
          Found in src/crawl.js - About 2 hrs to fix

            Function exports has a Cognitive Complexity of 17 (exceeds 5 allowed). Consider refactoring.
            Open

            module.exports = (scope, logger, finder, finderParameters, addUrl) => (cb) => {
              logger.debug('Trying to run finder.');
              if (!finder) {
                logger.debug('No finder defined.');
                cb();
            Severity: Minor
            Found in src/worker/steps/findLinks.js - About 2 hrs to fix

            Cognitive Complexity

            Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

            A method's cognitive complexity is based on a few simple rules:

            • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
            • Code is considered more complex for each "break in the linear flow of the code"
            • Code is considered more complex when "flow breaking structures are nested"

            Further reading

            Function exports has 53 lines of code (exceeds 25 allowed). Consider refactoring.
            Open

            module.exports = (scope, logger, addUrl, crawlerInstance) => {
              const followRedirects = crawlerInstance.followRedirects;
              const redirectFilter = crawlerInstance.redirectFilter;
            
              return (cb) => {
            Severity: Major
            Found in src/worker/steps/openPage.js - About 2 hrs to fix

              Function exports has 53 lines of code (exceeds 25 allowed). Consider refactoring.
              Open

              module.exports = (crawlerInstance, runnerKey, finderKey, prefix, pool, addUrl, processResult) => {
                /**
                 * Gets a finder definition of a {@link CrawlKit} instance.
                 *
                 * @private
              Severity: Major
              Found in src/worker/index.js - About 2 hrs to fix

                Function exports has 53 lines of code (exceeds 25 allowed). Consider refactoring.
                Open

                module.exports = (logger, crawlerInstance, prefix) => {
                  const poolDebug = {};
                  const concurrency = crawlerInstance.concurrency;
                  const phantomParameters = crawlerInstance.phantomParameters;
                  const browserCookies = crawlerInstance.browserCookies;
                Severity: Major
                Found in src/createPhantomPool.js - About 2 hrs to fix

                  Function doneAndNext has 36 lines of code (exceeds 25 allowed). Consider refactoring.
                  Open

                      const doneAndNext = callbackTimeout(once((res) => {
                        logger.debug(`Runner '${runnerId}' finished.`);
                        let err;
                        let result;
                  
                  
                  Severity: Minor
                  Found in src/worker/steps/pageRunners.js - About 1 hr to fix

                    Function create has 34 lines of code (exceeds 25 allowed). Consider refactoring.
                    Open

                        create: (callback) => {
                          async.waterfall([
                            function createPhantom(done) {
                              logger.debug('Creating PhantomJS instance');
                              driver.create({
                    Severity: Minor
                    Found in src/createPhantomPool.js - About 1 hr to fix

                      Function phantomCallback has 31 lines of code (exceeds 25 allowed). Consider refactoring.
                      Open

                        function phantomCallback(err, urls) {
                          if (done.called) {
                            logger.debug('Callback alread called.');
                            return;
                          }
                      Severity: Minor
                      Found in src/worker/steps/findLinks.js - About 1 hr to fix

                        Function exports has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring.
                        Open

                        module.exports = (logger, crawlerInstance, prefix) => {
                          const poolDebug = {};
                          const concurrency = crawlerInstance.concurrency;
                          const phantomParameters = crawlerInstance.phantomParameters;
                          const browserCookies = crawlerInstance.browserCookies;
                        Severity: Minor
                        Found in src/createPhantomPool.js - About 1 hr to fix

                        Cognitive Complexity

                        Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

                        A method's cognitive complexity is based on a few simple rules:

                        • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
                        • Code is considered more complex for each "break in the linear flow of the code"
                        • Code is considered more complex when "flow breaking structures are nested"

                        Further reading

                        Function onNavigationRequested has 27 lines of code (exceeds 25 allowed). Consider refactoring.
                        Open

                            const onNavigationRequested = (redirectedToUrl, type, willNavigate, mainFrame) => {
                              if (urijs(scope.url).equals(redirectedToUrl)) {
                                // this is the initial open of the task URL, ignore
                                return;
                              }
                        Severity: Minor
                        Found in src/worker/steps/openPage.js - About 1 hr to fix

                          Function exports has 26 lines of code (exceeds 25 allowed). Consider refactoring.
                          Open

                          module.exports = (scope, logger, crawlerInstance) => {
                            const phantomPageSettings = crawlerInstance.phantomPageSettings;
                            const followRedirects = crawlerInstance.followRedirects;
                          
                            return (done) => {
                          Severity: Minor
                          Found in src/worker/steps/setPageSettings.js - About 1 hr to fix

                            Function exports has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.
                            Open

                            module.exports = (trace) => {
                              if (!(trace instanceof Array)) {
                                return false;
                              }
                              for (let i = 0; i < trace.length; i += 1) {
                            Severity: Minor
                            Found in src/isPhantomError.js - About 35 mins to fix

                            Cognitive Complexity

                            Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

                            A method's cognitive complexity is based on a few simple rules:

                            • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
                            • Code is considered more complex for each "break in the linear flow of the code"
                            • Code is considered more complex when "flow breaking structures are nested"

                            Further reading

                            Missing trailing comma.
                            Open

                                          addUrl
                            Severity: Minor
                            Found in src/worker/steps/openPage.js by eslint

                            require or disallow trailing commas (comma-dangle)

                            Trailing commas in object literals are valid according to the ECMAScript 5 (and ECMAScript 3!) spec. However, IE8 (when not in IE8 document mode) and below will throw an error when it encounters trailing commas in JavaScript.

                            var foo = {
                                bar: "baz",
                                qux: "quux",
                            };

                            Trailing commas simplify adding and removing items to objects and arrays, since only the lines you are modifying must be touched. Another argument in favor of trailing commas is that it improves the clarity of diffs when an item is added or removed from an object or array:

                            Less clear:

                            var foo = {
                            -    bar: "baz",
                            -    qux: "quux"
                            +    bar: "baz"
                             };

                            More clear:

                            var foo = {
                                 bar: "baz",
                            -    qux: "quux",
                             };

                            Rule Details

                            This rule enforces consistent use of trailing commas in object and array literals.

                            Options

                            This rule has a string option or an object option:

                            {
                                "comma-dangle": ["error", "never"],
                                // or
                                "comma-dangle": ["error", {
                                    "arrays": "never",
                                    "objects": "never",
                                    "imports": "never",
                                    "exports": "never",
                                    "functions": "ignore",
                                }]
                            }
                            • "never" (default) disallows trailing commas
                            • "always" requires trailing commas
                            • "always-multiline" requires trailing commas when the last element or property is in a different line than the closing ] or } and disallows trailing commas when the last element or property is on the same line as the closing ] or }
                            • "only-multiline" allows (but does not require) trailing commas when the last element or property is in a different line than the closing ] or } and disallows trailing commas when the last element or property is on the same line as the closing ] or }

                            Trailing commas in function declarations and function calls are valid syntax since ECMAScript 2017; however, the string option does not check these situations for backwards compatibility.

                            You can also use an object option to configure this rule for each type of syntax. Each of the following options can be set to "never", "always", "always-multiline", "only-multiline", or "ignore". The default for each option is "never" unless otherwise specified.

                            • arrays is for array literals and array patterns of destructuring. (e.g. let [a,] = [1,];)
                            • objects is for object literals and object patterns of destructuring. (e.g. let {a,} = {a: 1};)
                            • imports is for import declarations of ES Modules. (e.g. import {a,} from "foo";)
                            • exports is for export declarations of ES Modules. (e.g. export {a,};)
                            • functions is for function declarations and function calls. (e.g. (function(a,){ })(b,);)
                              functions is set to "ignore" by default for consistency with the string option.

                            never

                            Examples of incorrect code for this rule with the default "never" option:

                            /*eslint comma-dangle: ["error", "never"]*/
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux",
                            };
                            
                            var arr = [1,2,];
                            
                            foo({
                              bar: "baz",
                              qux: "quux",
                            });

                            Examples of correct code for this rule with the default "never" option:

                            /*eslint comma-dangle: ["error", "never"]*/
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux"
                            };
                            
                            var arr = [1,2];
                            
                            foo({
                              bar: "baz",
                              qux: "quux"
                            });

                            always

                            Examples of incorrect code for this rule with the "always" option:

                            /*eslint comma-dangle: ["error", "always"]*/
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux"
                            };
                            
                            var arr = [1,2];
                            
                            foo({
                              bar: "baz",
                              qux: "quux"
                            });

                            Examples of correct code for this rule with the "always" option:

                            /*eslint comma-dangle: ["error", "always"]*/
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux",
                            };
                            
                            var arr = [1,2,];
                            
                            foo({
                              bar: "baz",
                              qux: "quux",
                            });

                            always-multiline

                            Examples of incorrect code for this rule with the "always-multiline" option:

                            /*eslint comma-dangle: ["error", "always-multiline"]*/
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux"
                            };
                            
                            var foo = { bar: "baz", qux: "quux", };
                            
                            var arr = [1,2,];
                            
                            var arr = [1,
                                2,];
                            
                            var arr = [
                                1,
                                2
                            ];
                            
                            foo({
                              bar: "baz",
                              qux: "quux"
                            });

                            Examples of correct code for this rule with the "always-multiline" option:

                            /*eslint comma-dangle: ["error", "always-multiline"]*/
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux",
                            };
                            
                            var foo = {bar: "baz", qux: "quux"};
                            var arr = [1,2];
                            
                            var arr = [1,
                                2];
                            
                            var arr = [
                                1,
                                2,
                            ];
                            
                            foo({
                              bar: "baz",
                              qux: "quux",
                            });

                            only-multiline

                            Examples of incorrect code for this rule with the "only-multiline" option:

                            /*eslint comma-dangle: ["error", "only-multiline"]*/
                            
                            var foo = { bar: "baz", qux: "quux", };
                            
                            var arr = [1,2,];
                            
                            var arr = [1,
                                2,];

                            Examples of correct code for this rule with the "only-multiline" option:

                            /*eslint comma-dangle: ["error", "only-multiline"]*/
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux",
                            };
                            
                            var foo = {
                                bar: "baz",
                                qux: "quux"
                            };
                            
                            var foo = {bar: "baz", qux: "quux"};
                            var arr = [1,2];
                            
                            var arr = [1,
                                2];
                            
                            var arr = [
                                1,
                                2,
                            ];
                            
                            var arr = [
                                1,
                                2
                            ];
                            
                            foo({
                              bar: "baz",
                              qux: "quux",
                            });
                            
                            foo({
                              bar: "baz",
                              qux: "quux"
                            });

                            functions

                            Examples of incorrect code for this rule with the {"functions": "never"} option:

                            /*eslint comma-dangle: ["error", {"functions": "never"}]*/
                            
                            function foo(a, b,) {
                            }
                            
                            foo(a, b,);
                            new foo(a, b,);

                            Examples of correct code for this rule with the {"functions": "never"} option:

                            /*eslint comma-dangle: ["error", {"functions": "never"}]*/
                            
                            function foo(a, b) {
                            }
                            
                            foo(a, b);
                            new foo(a, b);

                            Examples of incorrect code for this rule with the {"functions": "always"} option:

                            /*eslint comma-dangle: ["error", {"functions": "always"}]*/
                            
                            function foo(a, b) {
                            }
                            
                            foo(a, b);
                            new foo(a, b);

                            Examples of correct code for this rule with the {"functions": "always"} option:

                            /*eslint comma-dangle: ["error", {"functions": "always"}]*/
                            
                            function foo(a, b,) {
                            }
                            
                            foo(a, b,);
                            new foo(a, b,);

                            When Not To Use It

                            You can turn this rule off if you are not concerned with dangling commas. Source: http://eslint.org/docs/rules/

                            Unnecessary return statement.
                            Open

                                    return;
                            Severity: Minor
                            Found in src/worker/steps/openPage.js by eslint

                            Disallow redundant return statements (no-useless-return)

                            A return; statement with nothing after it is redundant, and has no effect on the runtime behavior of a function. This can be confusing, so it's better to disallow these redundant statements.

                            Rule Details

                            This rule aims to report redundant return statements.

                            Examples of incorrect code for this rule:

                            /* eslint no-useless-return: "error" */
                            
                            function foo() { return; }
                            
                            function foo() {
                              doSomething();
                              return;
                            }
                            
                            function foo() {
                              if (condition) {
                                bar();
                                return;
                              } else {
                                baz();
                              }
                            }
                            
                            function foo() {
                              switch (bar) {
                                case 1:
                                  doSomething();
                                default:
                                  doSomethingElse();
                                  return;
                              }
                            }

                            Examples of correct code for this rule:

                            /* eslint no-useless-return: "error" */
                            
                            function foo() { return 5; }
                            
                            function foo() {
                              return doSomething();
                            }
                            
                            function foo() {
                              if (condition) {
                                bar();
                                return;
                              } else {
                                baz();
                              }
                              qux();
                            }
                            
                            function foo() {
                              switch (bar) {
                                case 1:
                                  doSomething();
                                  return;
                                default:
                                  doSomethingElse();
                              }
                            }
                            
                            function foo() {
                              for (const foo of bar) {
                                return;
                              }
                            }

                            When Not To Use It

                            If you don't care about disallowing redundant return statements, you can turn off this rule. Source: http://eslint.org/docs/rules/

                            Unexpected chained assignment.
                            Open

                              const results = scope.result.runners = {}; // eslint-disable-line no-param-reassign
                            Severity: Minor
                            Found in src/worker/steps/pageRunners.js by eslint

                            Disallow Use of Chained Assignment Expressions (no-multi-assign)

                            Chaining the assignment of variables can lead to unexpected results and be difficult to read.

                            a = b = c = d;

                            Rule Details

                            This rule disallows using multiple assignments within a single statement.

                            Examples of incorrect code for this rule:

                            /*eslint no-multi-assign: "error"*/
                            
                            var a = b = c = 5;
                            
                            var foo = bar = "baz";
                            
                            var a =
                                b =
                                c;

                            Examples of correct code for this rule:

                            /*eslint no-multi-assign: "error"*/
                            var a = 5;
                            var b = 5;
                            var c = 5;
                            
                            var foo = "baz";
                            var bar = "baz";
                            
                            var a = c;
                            var b = c;

                            Related Rules

                            Severity
                            Category
                            Status
                            Source
                            Language