Sign upLogin

nlpodyssey/gotokenizers

View on GitHub
Star
models/bpemodel/bpemodel.go

Summary

Maintainability
A
2 hrs
Test Coverage

Method BPEModel.mergeWord has a Cognitive Complexity of 26 (exceeds 20 allowed). Consider refactoring.
Open

func (m *BPEModel) mergeWord(w string) (*Word, error) {
    word := NewWordWithCapacity(len(w))

    var unkTokenID int
Severity: Minor
Found in models/bpemodel/bpemodel.go - About 1 hr to fix

Cognitive Complexity

Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.

A method's cognitive complexity is based on a few simple rules:

  • Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
  • Code is considered more complex for each "break in the linear flow of the code"
  • Code is considered more complex when "flow breaking structures are nested"

Further reading

Function New has 8 arguments (exceeds 4 allowed). Consider refactoring.
Open

    vocab *vocabulary.Vocabulary,
    merges *MergeMap,
    cacheCapacity int,
    dropout float64,
    unknownToken string,
Severity: Major
Found in models/bpemodel/bpemodel.go - About 1 hr to fix

    exported var ErrUnknownTokenOutOfVocabulary should have comment or be unexported
    Open

    var ErrUnknownTokenOutOfVocabulary = fmt.Errorf("the provided unk token is out of vocabulary")
    Severity: Minor
    Found in models/bpemodel/bpemodel.go by golint

    exported function NewDefault should have comment or be unexported
    Open

    func NewDefault() *BPEModel {
    Severity: Minor
    Found in models/bpemodel/bpemodel.go by golint

    exported method BPEModel.Tokenize should have comment or be unexported
    Open

    func (m *BPEModel) Tokenize(sequence string) ([]models.Token, error) {
    Severity: Minor
    Found in models/bpemodel/bpemodel.go by golint

    There are no issues that match your filters.

    Category
    Status