codeprep/api/corpus.py from giganticode/codeprep

codeprep/api/corpus.py

6 hrs

def basic(path: str, extensions: Optional[str] = None, split_numbers: bool = False, ronin = False, stem: bool = False,

Found in codeprep/api/corpus.py - About 1 hr to fix

def bpe(path: str, bpe_codes_id: str, extensions: Optional[str] = None, no_spaces: bool = False,

Found in codeprep/api/corpus.py - About 1 hr to fix

def nosplit(path: str, extensions: Optional[str] = None, no_spaces: bool = False, no_unicode: bool = False,

Found in codeprep/api/corpus.py - About 1 hr to fix

def chars(path: str, extensions: Optional[str] = None, no_spaces: bool = False, no_unicode: bool = False,

Found in codeprep/api/corpus.py - About 1 hr to fix

def preprocess_corpus(path: str, prep_config: PrepConfig, bpe_codes_id: Optional[str]=None,

Found in codeprep/api/corpus.py - About 50 mins to fix

giganticode/codeprep