ReanGD/go-web-search

View on GitHub

Showing 24 of 24 total issues

Method minificationHTML.parseElements has 222 lines of code (exceeds 50 allowed). Consider refactoring.
Open

func (m *minificationHTML) parseElements(node *html.Node) (*html.Node, error) {
    switch node.DataAtom {
    case atom.A:
        return m.openNode(node, false)
    case atom.Abbr:
Severity: Major
Found in crawler/minification_html.go - About 7 hrs to fix

    Method DBWorker.saveMeta has a Cognitive Complexity of 35 (exceeds 20 allowed). Consider refactoring.
    Open

    func (w *DBWorker) saveMeta(tr *DBrw, meta *proxy.Meta, origin sql.NullInt64) error {
        hostID := meta.GetHostID()
        urlStr := meta.GetURL()
        urlNullID, err := w.getURLIDByStr(tr, urlStr)
        if err != nil {
    Severity: Minor
    Found in content/db_worker.go - About 2 hrs to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Method Logger.Print has a Cognitive Complexity of 28 (exceeds 20 allowed). Consider refactoring.
    Open

    func (logger Logger) Print(values ...interface{}) {
        if len(values) > 1 {
            level := values[0]
            currentTime := "\n" + gorm.NowFunc().Format("2006-01-02 15:04:05")
            source := fmt.Sprintf("%v:", values[1])
    Severity: Minor
    Found in content/logger.go - About 1 hr to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Method DBWorker.saveMeta has 61 lines of code (exceeds 50 allowed). Consider refactoring.
    Open

    func (w *DBWorker) saveMeta(tr *DBrw, meta *proxy.Meta, origin sql.NullInt64) error {
        hostID := meta.GetHostID()
        urlStr := meta.GetURL()
        urlNullID, err := w.getURLIDByStr(tr, urlStr)
        if err != nil {
    Severity: Minor
    Found in content/db_worker.go - About 1 hr to fix

      Method DBWorker.saveMeta has 10 return statements (exceeds 4 allowed).
      Open

      func (w *DBWorker) saveMeta(tr *DBrw, meta *proxy.Meta, origin sql.NullInt64) error {
          hostID := meta.GetHostID()
          urlStr := meta.GetURL()
          urlNullID, err := w.getURLIDByStr(tr, urlStr)
          if err != nil {
      Severity: Major
      Found in content/db_worker.go - About 1 hr to fix

        Method dictParser.nextWords has 8 return statements (exceeds 4 allowed).
        Open

        func (p *dictParser) nextWords() ([]string, error) {
            if p.finish {
                return nil, nil
            }
            t, err := p.nextToken()
        Severity: Major
        Found in import_dict.go - About 50 mins to fix

          Method dictRead.nextGroup has 8 return statements (exceeds 4 allowed).
          Open

          func (d *dictRead) nextGroup() (*wordGroup, error) {
              if d.done {
                  return nil, errors.New("done")
              }
              t, err := d.tokens.nextToken()
          Severity: Major
          Found in dict/dict_reader.go - About 50 mins to fix

            Similar blocks of code found in 2 locations. Consider refactoring.
            Open

            func (m *minificationText) parseChildren(node *html.Node) (*html.Node, error) {
                var err error
                for it := node.FirstChild; it != nil; {
                    it, err = m.parseNode(it)
                    if err != nil {
            Severity: Minor
            Found in crawler/minification_text.go and 1 other location - About 45 mins to fix
            crawler/minification_html.go on lines 23..33

            Duplicated Code

            Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

            Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

            When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

            Tuning

            This issue has a mass of 116.

            We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

            The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

            If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

            See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

            Refactorings

            Further Reading

            Similar blocks of code found in 2 locations. Consider refactoring.
            Open

            func (m *minificationHTML) parseChildren(node *html.Node) (*html.Node, error) {
                var err error
                for it := node.FirstChild; it != nil; {
                    it, err = m.parseNode(it)
                    if err != nil {
            Severity: Minor
            Found in crawler/minification_html.go and 1 other location - About 45 mins to fix
            crawler/minification_text.go on lines 112..122

            Duplicated Code

            Duplicated code can lead to software that is hard to understand and difficult to change. The Don't Repeat Yourself (DRY) principle states:

            Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

            When you violate DRY, bugs and maintenance problems are sure to follow. Duplicated code has a tendency to both continue to replicate and also to diverge (leaving bugs as two similar implementations differ in subtle ways).

            Tuning

            This issue has a mass of 116.

            We set useful threshold defaults for the languages we support but you may want to adjust these settings based on your project guidelines.

            The threshold configuration represents the minimum mass a code block must have to be analyzed for duplication. The lower the threshold, the more fine-grained the comparison.

            If the engine is too easily reporting duplication, try raising the threshold. If you suspect that the engine isn't catching enough duplication, try lowering the threshold. The best setting tends to differ from language to language.

            See codeclimate-duplication's documentation for more information about tuning the mass threshold in your .codeclimate.yml.

            Refactorings

            Further Reading

            Avoid deeply nested control flow statements.
            Open

                    } else if t.ttype == tokenEmpty {
                        break
                    } else {
                        return nil, errors.New("unknown token")
                    }
            Severity: Major
            Found in import_dict.go - About 45 mins to fix

              Method tokenRead.nextToken has 7 return statements (exceeds 4 allowed).
              Open

              func (t *tokenRead) nextToken() (*token, error) {
                  if !t.scanner.Scan() {
                      return &token{ttype: tokenEnd}, nil
                  }
              
              
              Severity: Major
              Found in dict/token_reader.go - About 45 mins to fix

                Avoid deeply nested control flow statements.
                Open

                                    } else if b, ok := value.([]byte); ok {
                                        formatedValues = append(formatedValues, fmt.Sprintf("'{...}%d'", len(b)))
                                        // formatedValues = append(formatedValues, fmt.Sprintf("'%v'", string(b)))
                                    } else if r, ok := value.(driver.Valuer); ok {
                                        if value, err := r.Value(); err == nil && value != nil {
                Severity: Major
                Found in content/logger.go - About 45 mins to fix

                  Method responseParser.processBody has 7 return statements (exceeds 4 allowed).
                  Open

                  func (r *responseParser) processBody(body []byte, contentType string) (database.State, error) {
                      if !isHTML(body) {
                          return database.StateParseError, werrors.New(ErrBodyNotHTML)
                      }
                  
                  
                  Severity: Major
                  Found in crawler/response_parser.go - About 45 mins to fix

                    Method minificationHTML.parseElements has 6 return statements (exceeds 4 allowed).
                    Open

                    func (m *minificationHTML) parseElements(node *html.Node) (*html.Node, error) {
                        switch node.DataAtom {
                        case atom.A:
                            return m.openNode(node, false)
                        case atom.Abbr:
                    Severity: Major
                    Found in crawler/minification_html.go - About 40 mins to fix

                      Method dictParser.nextToken has 6 return statements (exceeds 4 allowed).
                      Open

                      func (p *dictParser) nextToken() (token, error) {
                          done, data, err := p.nextLine()
                      
                          if err != nil {
                              return token{}, err
                      Severity: Major
                      Found in import_dict.go - About 40 mins to fix

                        Function readBody has 5 return statements (exceeds 4 allowed).
                        Open

                        func readBody(contentEncoding string, body io.Reader) ([]byte, error) {
                            var err error
                            result := []byte{}
                            if contentEncoding == "gzip" {
                                reader, err := gzip.NewReader(body)
                        Severity: Major
                        Found in crawler/body_parser.go - About 35 mins to fix

                          Method Compressed.Scan has 5 return statements (exceeds 4 allowed).
                          Open

                          func (c *Compressed) Scan(value interface{}) error {
                              if value == nil {
                                  var nilResult []byte
                                  c.Data = nilResult
                                  return nil
                          Severity: Major
                          Found in database/compressed.go - About 35 mins to fix

                            Method DBrw.Transaction has 5 return statements (exceeds 4 allowed).
                            Open

                            func (db *DBrw) Transaction(fn func(*DBrw) error) error {
                                tr := db.DB.Begin()
                                if tr.Error != nil {
                                    return fmt.Errorf("create transaction, message: %s", tr.Error)
                                }
                            Severity: Major
                            Found in content/rw_func.go - About 35 mins to fix

                              Method DBrw.AddHost has 5 return statements (exceeds 4 allowed).
                              Open

                              func (db *DBrw) AddHost(host *proxy.Host, baseURL string) (int64, error) {
                                  var id int64
                                  err := db.Transaction(func(tr *DBrw) error {
                                      tblHost := host.GetTable()
                                      err := tr.Create(tblHost).Error
                              Severity: Major
                              Found in content/impl_host.go - About 35 mins to fix

                                Method responseParser.Run has 5 return statements (exceeds 4 allowed).
                                Open

                                func (r *responseParser) Run(response *http.Response) error {
                                    contentType, contentEncoding, err := r.processMeta(response.StatusCode, &response.Header)
                                    if err != nil {
                                        _ = response.Body.Close()
                                        return err
                                Severity: Major
                                Found in crawler/response_parser.go - About 35 mins to fix
                                  Severity
                                  Category
                                  Status
                                  Source
                                  Language