Uses of Interface
opennlp.tools.tokenize.Tokenizer
Packages that use Tokenizer
Package
Description
Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).
Experimental package related to the
MUC
corpus format.Contains classes related to finding token or words in a string.
This package contains classes for generating sequence features.
-
Uses of Tokenizer in opennlp.tools.cmdline.parser
Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer -
Uses of Tokenizer in opennlp.tools.formats.brat
Constructors in opennlp.tools.formats.brat with parameters of type TokenizerModifierConstructorDescriptionBratDocumentParser
(SentenceDetector sentenceDetector, Tokenizer tokenizer) BratDocumentParser
(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes) BratNameSampleStream
(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples) Creates a newBratNameSampleStream
.BratNameSampleStream
(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes) Creates a newBratNameSampleStream
. -
Uses of Tokenizer in opennlp.tools.formats.muc
Constructors in opennlp.tools.formats.muc with parameters of type TokenizerModifierConstructorDescriptionMucNameContentHandler
(Tokenizer tokenizer, List<NameSample> storedSamples) Initializes aMucNameContentHandler
.protected
MucNameSampleStream
(Tokenizer tokenizer, ObjectStream<String> samples) Initializes aMucNameSampleStream
. -
Uses of Tokenizer in opennlp.tools.tokenize
Classes in opennlp.tools.tokenize that implement TokenizerModifier and TypeClassDescriptionclass
A basicTokenizer
implementation which performs tokenization using character classes.class
ATokenizer
for converting raw text into separated tokens.class
A basicTokenizer
implementation which performs tokenization using white spaces.class
ATokenizer
implementation which performs tokenization using word pieces.Constructors in opennlp.tools.tokenize with parameters of type TokenizerModifierConstructorDescriptionTokenizerEvaluator
(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners) Initializes an instance to evaluate aTokenizer
.TokenizerStream
(Tokenizer tokenizer, ObjectStream<String> input) Initializes ainstance
. -
Uses of Tokenizer in opennlp.tools.util.featuregen
Constructors in opennlp.tools.util.featuregen with parameters of type TokenizerModifierConstructorDescriptionTokenPatternFeatureGenerator
(Tokenizer supportTokenizer) Initializes aTokenPatternFeatureGenerator
instance.