Sign upLogin

nlpodyssey/gotokenizers

View on GitHub
Star

Showing 92 of 92 total issues

exported function FromString should have comment or be unexported
Open

func FromString(s string) *StringSplitPattern {
Severity: Minor
Found in splitpattern/string.go by golint

TODO found
Open

        //TODO: (from Rust) char::is_ascii_punctuation(&x) || ...
Severity: Minor
Found in pretokenizers/bertpretokenizer/bertpretokenizer.go by fixme

FIXME found
Open

    // FIXME: it's probably wrong to pass the rune index, instead of byte index
Severity: Minor
Found in normalizedstring/normalizedstring.go by fixme

TODO found
Open

    // TODO: to be implemented...
Severity: Minor
Found in models/bpemodel/wordcache.go by fixme

TODO found
Open

    // TODO: ns.Nfd()
Severity: Minor
Found in normalizers/bertnormalizer/bertnormalizer.go by fixme

FIXME found
Open

// FIXME: Append does nothing if the normalized string is empty
Severity: Minor
Found in normalizedstring/normalizedstring.go by fixme

FIXME found
Open

            // FIXME: is len of runes (and not bytes) correct?
Severity: Minor
Found in splitpattern/string.go by fixme

FIXME found
Open

// FIXME: Prepend does nothing if the normalized string is empty
Severity: Minor
Found in normalizedstring/normalizedstring.go by fixme

FIXME found
Open

// FIXME: Go `unicode` package does not consider Unicode special casing
Severity: Minor
Found in normalizedstring/normalizedstring.go by fixme

FIXME found
Open

// FIXME: see Unicode special casing notes on `NormalizedString.ToUpper()`
Severity: Minor
Found in normalizedstring/normalizedstring.go by fixme

TODO found
Open

    // TODO: to be implemented...
Severity: Minor
Found in models/bpemodel/wordcache.go by fixme

TODO found
Open

    // TODO: to be implemented...
Severity: Minor
Found in models/bpemodel/wordcache.go by fixme
Severity
Category
Status
Source
Language