edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class DefaultWordTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultWordTokenizer

All Implemented Interfaces:

WordTokenizer, IsCloseable, UsesLogger

Direct Known Subclasses:

ApostrophesAreNotQuotesWordTokenizer, EEBOWordTokenizer

public class DefaultWordTokenizer
extends AbstractWordTokenizer
implements WordTokenizer

Default word tokenizer.

Field Summary

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer

Constructor Summary

Constructors

Constructor and Description
DefaultWordTokenizer() Create a simple word tokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
void	addWordToSentence(java.util.List<java.lang.String> sentence, java.lang.String word) Add word to list of words in sentence.
java.util.List<java.lang.String>	extractWords(java.lang.String text) Break text into word tokens.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, preprocessToken, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

close, findWordOffsets, getPreTokenizer, preprocessToken, setAbbreviations, setAposTokens, setPreTokenizer

Methods inherited from interface edu.northwestern.at.utils.IsCloseable

close

Constructor Detail

DefaultWordTokenizer

public DefaultWordTokenizer()

Create a simple word tokenizer.

Method Detail

extractWords

public java.util.List<java.lang.String> extractWords(java.lang.String text)

Break text into word tokens.

Specified by:

extractWords in interface WordTokenizer

Specified by:

extractWords in class AbstractWordTokenizer

Parameters:

text - Text to break into word tokens.

Returns:

List of word tokens.

Word tokens may be words, numbers, punctuation, etc.

addWordToSentence

public void addWordToSentence(java.util.List<java.lang.String> sentence,
                     java.lang.String word)

Add word to list of words in sentence.

Specified by:

addWordToSentence in interface WordTokenizer

Overrides:

addWordToSentence in class AbstractWordTokenizer

Parameters:

sentence - Result sentence.

word - Word to add.