Uses of Class
opennlp.tools.tokenize.TokenSample

Packages that use TokenSample

Package	Description
opennlp.tools.cmdline.tokenizer
opennlp.tools.formats	Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.formats.ad	Experimental package related to the Arvores Deitadas corpus format.
opennlp.tools.formats.conllu	Experimental package related to the CoNNL-U format.
opennlp.tools.formats.convert
opennlp.tools.formats.irishsentencebank	Experimental package related to the Irish Sentence Bank format.
opennlp.tools.formats.masc	Experimental package related to the MASC corpus format.
opennlp.tools.tokenize	Contains classes related to finding token or words in a string.
opennlp.tools.tokenize.lang.en

Uses of TokenSample in opennlp.tools.cmdline.tokenizer

Subclasses with type arguments of type TokenSample in opennlp.tools.cmdline.tokenizer

Modifier and Type	Class	Description
class	DetokenEvaluationErrorListener	A default implementation of EvaluationMonitor that prints to an output stream.
class	TokenEvaluationErrorListener	A default implementation of EvaluationMonitor that prints to an output stream.
class	TokenizerConverterTool	Tool to convert multiple data formats into native OpenNLP sentence detector training format.
final class	TokenizerCrossValidatorTool
final class	TokenizerMEEvaluatorTool	A default TokenSample-centric implementation of AbstractEvaluatorTool that prints to an output stream.
final class	TokenizerTrainerTool

Methods in opennlp.tools.cmdline.tokenizer with parameters of type TokenSample

Modifier and Type	Method	Description
void	DetokenEvaluationErrorListener.misclassified(TokenSample reference, TokenSample prediction)
void	TokenEvaluationErrorListener.misclassified(TokenSample reference, TokenSample prediction)

Uses of TokenSample in opennlp.tools.formats

Subclasses with type arguments of type TokenSample in opennlp.tools.formats

Modifier and Type	Class	Description
class	ConllXTokenSampleStreamFactory	Note: Do not use this class, internal use only!
class	TokenSampleStreamFactory	Factory producing OpenNLP TokenSampleStreams.

Methods in opennlp.tools.formats that return types with arguments of type TokenSample

Modifier and Type	Method	Description
ObjectStream<TokenSample>	ConllXTokenSampleStreamFactory.create(String[] args)
ObjectStream<TokenSample>	TokenSampleStreamFactory.create(String[] args)

Uses of TokenSample in opennlp.tools.formats.ad

Subclasses with type arguments of type TokenSample in opennlp.tools.formats.ad

Modifier and Type	Class	Description
class	ADTokenSampleStreamFactory	Note: Do not use this class, internal use only!

Methods in opennlp.tools.formats.ad that return types with arguments of type TokenSample

Modifier and Type	Method	Description
ObjectStream<TokenSample>	ADTokenSampleStreamFactory.create(String[] args)

Uses of TokenSample in opennlp.tools.formats.conllu

Subclasses with type arguments of type TokenSample in opennlp.tools.formats.conllu

Modifier and Type	Class	Description
class	ConlluTokenSampleStream
class	ConlluTokenSampleStreamFactory	Note: Do not use this class, internal use only!

Methods in opennlp.tools.formats.conllu that return TokenSample

Modifier and Type	Method	Description
TokenSample	ConlluTokenSampleStream.read()

Methods in opennlp.tools.formats.conllu that return types with arguments of type TokenSample

Modifier and Type	Method	Description
ObjectStream<TokenSample>	ConlluTokenSampleStreamFactory.create(String[] args)

Uses of TokenSample in opennlp.tools.formats.convert

Subclasses with type arguments of type TokenSample in opennlp.tools.formats.convert

Modifier and Type	Class	Description
class	NameToTokenSampleStream	Note: Do not use this class, internal use only!
class	NameToTokenSampleStreamFactory	Note: Do not use this class, internal use only!
class	ParseToTokenSampleStreamFactory	Note: Do not use this class, internal use only!
class	POSToTokenSampleStream	Note: Do not use this class, internal use only!
class	POSToTokenSampleStreamFactory	Note: Do not use this class, internal use only!

Methods in opennlp.tools.formats.convert that return TokenSample

Modifier and Type	Method	Description
TokenSample	NameToTokenSampleStream.read()
TokenSample	POSToTokenSampleStream.read()

Methods in opennlp.tools.formats.convert that return types with arguments of type TokenSample

Modifier and Type	Method	Description
ObjectStream<TokenSample>	NameToTokenSampleStreamFactory.create(String[] args)
ObjectStream<TokenSample>	ParseToTokenSampleStreamFactory.create(String[] args)
ObjectStream<TokenSample>	POSToTokenSampleStreamFactory.create(String[] args)

Uses of TokenSample in opennlp.tools.formats.irishsentencebank

Subclasses with type arguments of type TokenSample in opennlp.tools.formats.irishsentencebank

Modifier and Type	Class	Description
class	IrishSentenceBankTokenSampleStreamFactory	Note: Do not use this class, internal use only!

Methods in opennlp.tools.formats.irishsentencebank that return TokenSample

Modifier and Type	Method	Description
TokenSample	IrishSentenceBankDocument.IrishSentenceBankSentence.getTokenSample()

Methods in opennlp.tools.formats.irishsentencebank that return types with arguments of type TokenSample

Modifier and Type	Method	Description
ObjectStream<TokenSample>	IrishSentenceBankTokenSampleStreamFactory.create(String[] args)

Uses of TokenSample in opennlp.tools.formats.masc

Subclasses with type arguments of type TokenSample in opennlp.tools.formats.masc

Modifier and Type	Class	Description
class	MascTokenSampleStream
class	MascTokenSampleStreamFactory	Note: Do not use this class, internal use only!

Methods in opennlp.tools.formats.masc that return TokenSample

Modifier and Type	Method	Description
TokenSample	MascTokenSampleStream.read()

Methods in opennlp.tools.formats.masc that return types with arguments of type TokenSample

Modifier and Type	Method	Description
ObjectStream<TokenSample>	MascTokenSampleStreamFactory.create(String[] args)

Uses of TokenSample in opennlp.tools.tokenize

Subclasses with type arguments of type TokenSample in opennlp.tools.tokenize

Modifier and Type	Class	Description
class	DetokenizerEvaluator	The DetokenizerEvaluator measures the performance of the given Detokenizer with the provided reference samples.
class	TokenizerEvaluator	The TokenizerEvaluator measures the performance of the given Tokenizer with the provided reference samples.
class	TokenSampleStream	This class is a stream filter which reads in string encoded samples and creates samples out of them.
class	TokSpanEventStream	This class reads the samples via an Iterator and converts the samples into events which can be used by the maxent library for training.
class	WhitespaceTokenStream	This stream formats ObjectStream of samples into whitespace separated token strings.

Subinterfaces with type arguments of type TokenSample in opennlp.tools.tokenize

Modifier and Type	Interface	Description
interface	TokenizerEvaluationMonitor	A marker interface for evaluating tokenizers.

Classes in opennlp.tools.tokenize that implement interfaces with type arguments of type TokenSample

Modifier and Type	Class	Description
class	TokenizerStream	The TokenizerStream uses a Tokenizer to tokenize the input string and output samples.

Methods in opennlp.tools.tokenize that return TokenSample

Modifier and Type	Method	Description
static TokenSample	TokenSample.parse(String sampleString, String separatorChars)	Parses a string sample.
protected TokenSample	DetokenizerEvaluator.processSample(TokenSample reference)
protected TokenSample	TokenizerEvaluator.processSample(TokenSample reference)
TokenSample	TokenizerStream.read()
TokenSample	TokenSampleStream.read()

Methods in opennlp.tools.tokenize with parameters of type TokenSample

Modifier and Type	Method	Description
protected Iterator<Event>	TokSpanEventStream.createEvents(TokenSample tokenSample)	Adds training events to the event stream for each of the specified sample.
protected TokenSample	DetokenizerEvaluator.processSample(TokenSample reference)
protected TokenSample	TokenizerEvaluator.processSample(TokenSample reference)

Method parameters in opennlp.tools.tokenize with type arguments of type TokenSample

Modifier and Type	Method	Description
void	TokenizerCrossValidator.evaluate(ObjectStream<TokenSample> samples, int nFolds)	Starts the evaluation.
static TokenizerModel	TokenizerME.train(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams)	Trains a model for the TokenizerME.

Constructor parameters in opennlp.tools.tokenize with type arguments of type TokenSample

Modifier	Constructor	Description
	TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics)	Initializes a new event stream based on the data stream using a TokenContextGenerator that relies on a DefaultTokenContextGenerator.
	TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, TokenContextGenerator cg)	Initializes a new event stream based on the data stream using a TokenContextGenerator.
	TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg)	Initializes a new event stream based on the data stream using a TokenContextGenerator.
	WhitespaceTokenStream(ObjectStream<TokenSample> tokens)	Initializes a WhitespaceTokenStream.

Uses of TokenSample in opennlp.tools.tokenize.lang.en

Classes in opennlp.tools.tokenize.lang.en that implement interfaces with type arguments of type TokenSample

Modifier and Type	Class	Description
class	TokenSampleStream	Class which produces an Iterator<TokenSample> from a file of space delimited token.

Methods in opennlp.tools.tokenize.lang.en that return TokenSample

Modifier and Type	Method	Description
TokenSample	TokenSampleStream.next()