ReanGD/go-web-search

View on GitHub

Showing 62 of 62 total issues

Method minificationHTML.minificationHTML.minificationHTML.parseElements has 222 lines of code (exceeds 25 allowed). Consider refactoring.
Open

func (m *minificationHTML) parseElements(node *html.Node) (*html.Node, error) {
    switch node.DataAtom {
    case atom.A:
        return m.openNode(node, false)
    case atom.Abbr:
Severity: Major
Found in crawler/minification_html.go - About 1 day to fix

    Method DBWorker.DBWorker.DBWorker.saveMeta has a Cognitive Complexity of 35 (exceeds 5 allowed). Consider refactoring.
    Open

    func (w *DBWorker) saveMeta(tr *DBrw, meta *proxy.Meta, origin sql.NullInt64) error {
        hostID := meta.GetHostID()
        urlStr := meta.GetURL()
        urlNullID, err := w.getURLIDByStr(tr, urlStr)
        if err != nil {
    Severity: Minor
    Found in content/db_worker.go - About 5 hrs to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Method Logger.Logger.Logger.Print has a Cognitive Complexity of 28 (exceeds 5 allowed). Consider refactoring.
    Open

    func (logger Logger) Print(values ...interface{}) {
        if len(values) > 1 {
            level := values[0]
            currentTime := "\n" + gorm.NowFunc().Format("2006-01-02 15:04:05")
            source := fmt.Sprintf("%v:", values[1])
    Severity: Minor
    Found in content/logger.go - About 4 hrs to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    Method dictRead.dictRead.dictRead.nextGroup has a Cognitive Complexity of 20 (exceeds 5 allowed). Consider refactoring.
    Open

    func (d *dictRead) nextGroup() (*wordGroup, error) {
        if d.done {
            return nil, errors.New("done")
        }
        t, err := d.tokens.nextToken()
    Severity: Minor
    Found in dict/dict_reader.go - About 2 hrs to fix

    Cognitive Complexity

    Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

    A method's cognitive complexity is based on a few simple rules:

    • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
    • Code is considered more complex for each "break in the linear flow of the code"
    • Code is considered more complex when "flow breaking structures are nested"

    Further reading

    File minification_html.go has 269 lines of code (exceeds 250 allowed). Consider refactoring.
    Open

    package crawler
    
    // status: ok
    import (
        "github.com/ReanGD/go-web-search/werrors"
    Severity: Minor
    Found in crawler/minification_html.go - About 2 hrs to fix

      Method DBWorker.DBWorker.DBWorker.DBWorker.saveMeta has 61 lines of code (exceeds 25 allowed). Consider refactoring.
      Open

      func (w *DBWorker) saveMeta(tr *DBrw, meta *proxy.Meta, origin sql.NullInt64) error {
          hostID := meta.GetHostID()
          urlStr := meta.GetURL()
          urlNullID, err := w.getURLIDByStr(tr, urlStr)
          if err != nil {
      Severity: Major
      Found in content/db_worker.go - About 2 hrs to fix

        Function removeUtmFromQuery has a Cognitive Complexity of 16 (exceeds 5 allowed). Consider refactoring.
        Open

        func removeUtmFromQuery(u *url.URL) {
            q := u.Query()
        
            if len(q) > 0 {
                buf := new(bytes.Buffer)
        Severity: Minor
        Found in crawler/url_utils.go - About 2 hrs to fix

        Cognitive Complexity

        Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

        A method's cognitive complexity is based on a few simple rules:

        • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
        • Code is considered more complex for each "break in the linear flow of the code"
        • Code is considered more complex when "flow breaking structures are nested"

        Further reading

        Method request.request.request.request.get has 48 lines of code (exceeds 25 allowed). Consider refactoring.
        Open

        func (r *request) get(u *url.URL) (int64, error) {
            urlStr := u.String()
            r.urls = make(map[string]sql.NullInt64)
            hostID, robotOk := r.hostMng.CheckURL(u)
            r.meta = proxy.NewMeta(hostID, urlStr, nil)
        Severity: Minor
        Found in crawler/request.go - About 1 hr to fix

          Method dictRead.dictRead.dictRead.dictRead.nextGroup has 44 lines of code (exceeds 25 allowed). Consider refactoring.
          Open

          func (d *dictRead) nextGroup() (*wordGroup, error) {
              if d.done {
                  return nil, errors.New("done")
              }
              t, err := d.tokens.nextToken()
          Severity: Minor
          Found in dict/dict_reader.go - About 1 hr to fix

            Method DBWorker.DBWorker.DBWorker.Start has a Cognitive Complexity of 14 (exceeds 5 allowed). Consider refactoring.
            Open

            func (w *DBWorker) Start(wgParent *sync.WaitGroup) {
                defer wgParent.Done()
            
                finish := false
                for !finish {
            Severity: Minor
            Found in content/db_worker.go - About 1 hr to fix

            Cognitive Complexity

            Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

            A method's cognitive complexity is based on a few simple rules:

            • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
            • Code is considered more complex for each "break in the linear flow of the code"
            • Code is considered more complex when "flow breaking structures are nested"

            Further reading

            Function readBody has a Cognitive Complexity of 13 (exceeds 5 allowed). Consider refactoring.
            Open

            func readBody(contentEncoding string, body io.Reader) ([]byte, error) {
                var err error
                result := []byte{}
                if contentEncoding == "gzip" {
                    reader, err := gzip.NewReader(body)
            Severity: Minor
            Found in crawler/body_parser.go - About 1 hr to fix

            Cognitive Complexity

            Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

            A method's cognitive complexity is based on a few simple rules:

            • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
            • Code is considered more complex for each "break in the linear flow of the code"
            • Code is considered more complex when "flow breaking structures are nested"

            Further reading

            Method dictParser.dictParser.dictParser.nextWords has a Cognitive Complexity of 12 (exceeds 5 allowed). Consider refactoring.
            Open

            func (p *dictParser) nextWords() ([]string, error) {
                if p.finish {
                    return nil, nil
                }
                t, err := p.nextToken()
            Severity: Minor
            Found in import_dict.go - About 1 hr to fix

            Cognitive Complexity

            Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

            A method's cognitive complexity is based on a few simple rules:

            • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
            • Code is considered more complex for each "break in the linear flow of the code"
            • Code is considered more complex when "flow breaking structures are nested"

            Further reading

            Method Logger.Logger.Logger.Logger.Print has 35 lines of code (exceeds 25 allowed). Consider refactoring.
            Open

            func (logger Logger) Print(values ...interface{}) {
                if len(values) > 1 {
                    level := values[0]
                    currentTime := "\n" + gorm.NowFunc().Format("2006-01-02 15:04:05")
                    source := fmt.Sprintf("%v:", values[1])
            Severity: Minor
            Found in content/logger.go - About 1 hr to fix

              Method dictParser.dictParser.dictParser.dictParser.nextWords has 34 lines of code (exceeds 25 allowed). Consider refactoring.
              Open

              func (p *dictParser) nextWords() ([]string, error) {
                  if p.finish {
                      return nil, nil
                  }
                  t, err := p.nextToken()
              Severity: Minor
              Found in import_dict.go - About 1 hr to fix

                Method request.request.request.Init has a Cognitive Complexity of 11 (exceeds 5 allowed). Consider refactoring.
                Open

                func (r *request) Init(logger zap.Logger) {
                    r.client = new(http.Client)
                    r.logger = logger
                    r.client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
                        if len(via) >= 10 {
                Severity: Minor
                Found in crawler/request.go - About 1 hr to fix

                Cognitive Complexity

                Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

                A method's cognitive complexity is based on a few simple rules:

                • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
                • Code is considered more complex for each "break in the linear flow of the code"
                • Code is considered more complex when "flow breaking structures are nested"

                Further reading

                Function GetDBrw has 31 lines of code (exceeds 25 allowed). Consider refactoring.
                Open

                func GetDBrw() (*DBrw, error) {
                    db, err := gorm.Open("sqlite3", "file:content.db?cache=shared")
                    // db, err := gorm.Open("sqlite3", "content.db")
                    if err != nil {
                        return nil, err
                Severity: Minor
                Found in content/content.go - About 1 hr to fix

                  Function Convert has 30 lines of code (exceeds 25 allowed). Consider refactoring.
                  Open

                  func Convert(path string) error {
                      file, err := os.Open(path)
                      if err != nil {
                          return err
                      }
                  Severity: Minor
                  Found in dict/convert.go - About 1 hr to fix

                    Method responseParser.responseParser.responseParser.responseParser.processBody has 28 lines of code (exceeds 25 allowed). Consider refactoring.
                    Open

                    func (r *responseParser) processBody(body []byte, contentType string) (database.State, error) {
                        if !isHTML(body) {
                            return database.StateParseError, werrors.New(ErrBodyNotHTML)
                        }
                    
                    
                    Severity: Minor
                    Found in crawler/response_parser.go - About 1 hr to fix

                      Method dictParser.dictParser.dictParser.dictParser.nextToken has 28 lines of code (exceeds 25 allowed). Consider refactoring.
                      Open

                      func (p *dictParser) nextToken() (token, error) {
                          done, data, err := p.nextLine()
                      
                          if err != nil {
                              return token{}, err
                      Severity: Minor
                      Found in import_dict.go - About 1 hr to fix

                        Function createTables has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring.
                        Open

                        func createTables(db *gorm.DB, values ...interface{}) error {
                            for _, value := range values {
                                if !db.HasTable(value) {
                                    err := db.CreateTable(value).Error
                                    if err != nil {
                        Severity: Minor
                        Found in content/content.go - About 1 hr to fix

                        Cognitive Complexity

                        Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

                        A method's cognitive complexity is based on a few simple rules:

                        • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
                        • Code is considered more complex for each "break in the linear flow of the code"
                        • Code is considered more complex when "flow breaking structures are nested"

                        Further reading

                        Severity
                        Category
                        Status
                        Source
                        Language