Sign upLogin

nlpodyssey/gotokenizers

View on GitHub
Star
pretokenizers/bertpretokenizer/bertpretokenizer.go

Summary

Maintainability
A
40 mins
Test Coverage

Method BertPreTokenizer.PreTokenize has 6 return statements (exceeds 4 allowed).
Open

func (b *BertPreTokenizer) PreTokenize(pts *pretokenizedstring.PreTokenizedString) error {
    isWhitespacePattern := splitpattern.FromFunc(func(r rune) bool {
        return unicode.In(r, unicode.White_Space)
    })
    isBertPunctuationPattern := splitpattern.FromFunc(func(r rune) bool {
Severity: Major
Found in pretokenizers/bertpretokenizer/bertpretokenizer.go - About 40 mins to fix

    TODO found
    Open

            //TODO: (from Rust) char::is_ascii_punctuation(&x) || ...
    Severity: Minor
    Found in pretokenizers/bertpretokenizer/bertpretokenizer.go by fixme

    There are no issues that match your filters.

    Category
    Status