edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Interface WordTokenizer

All Known Implementing Classes:

AbstractWordTokenizer, ApostrophesAreNotQuotesWordTokenizer, ContractionTokenizer, DefaultWordTokenizer, EEBOWordTokenizer, ICU4JBreakIteratorWordTokenizer, NoopWordTokenizer, PennTreebankTokenizer, WhitespaceWordTokenizer

public interface WordTokenizer

Interface for tokenizing a string into "words".

Method Summary

Methods

Modifier and Type	Method and Description
void	addWordToSentence(java.util.List<java.lang.String> sentence, java.lang.String word) Add word to list of words in sentence.
void	close() Close down the word tokenizer.
java.util.List<java.lang.String>	extractWords(java.lang.String text) Break text into word tokens.
int[]	findWordOffsets(java.lang.String sentenceText, java.util.List<?> words) Find starting offsets of words in a sentence.
PreTokenizer	getPreTokenizer() Get the preTokenizer.
java.lang.String	preprocessToken(java.lang.String token, java.util.List<java.lang.String> tokenList) Preprocess a word token.
void	setAbbreviations(Abbreviations abbreviations) Set abbreviations.
void	setAposTokens(AposTokens aposTokens) Set apostophe tokens.
void	setPreTokenizer(PreTokenizer preTokenizer) Set the preTokenizer.

Method Detail

getPreTokenizer

PreTokenizer getPreTokenizer()

Get the preTokenizer.

Returns:

The preTokenizer.

setPreTokenizer

void setPreTokenizer(PreTokenizer preTokenizer)

Set the preTokenizer.

Parameters:

preTokenizer - The preTokenizer.

setAbbreviations

void setAbbreviations(Abbreviations abbreviations)

Set abbreviations.

Parameters:

abbreviations - Abbreviations.

setAposTokens

void setAposTokens(AposTokens aposTokens)

Set apostophe tokens.

Parameters:

aposTokens - Apostrophe tokens.

addWordToSentence

void addWordToSentence(java.util.List<java.lang.String> sentence,
                     java.lang.String word)

Add word to list of words in sentence.

Parameters:

sentence - Result sentence.

word - Word to add.

extractWords

java.util.List<java.lang.String> extractWords(java.lang.String text)

Break text into word tokens.

Parameters:

text - Text to break into word tokens.

Returns:

List of word tokens.

Word tokens may be words, numbers, punctuation, etc.

findWordOffsets

int[] findWordOffsets(java.lang.String sentenceText,
                    java.util.List<?> words)

Find starting offsets of words in a sentence.

Parameters:

sentenceText - Text from which tokens were extracted.

words - List of words extracted from sentence text. N.B. If the words aren't from the specified sentence text, the resulting offsets will be meaningless.

Returns:

int array of starting offsets in sentenceText for each word. The first offset starts at 0. There is one more offset than the number of words -- the last offset is where the word after the last word would start.

preprocessToken

java.lang.String preprocessToken(java.lang.String token,
                               java.util.List<java.lang.String> tokenList)

Preprocess a word token.

Parameters:

token - Token to preprocess.

tokenList - List of previous tokens already issued.

Returns:

Preprocessed token. The token list may also have been modified.

close

void close()

Close down the word tokenizer.