edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class ICU4JBreakIteratorWordTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.ICU4JBreakIteratorWordTokenizer

All Implemented Interfaces:

CanSplitAroundPeriods, CanTokenizeWhitespace, WordTokenizer, IsCloseable, UsesLogger

public class ICU4JBreakIteratorWordTokenizer
extends AbstractWordTokenizer
implements WordTokenizer, CanTokenizeWhitespace, CanSplitAroundPeriods

Word tokenizer which uses ICU library for tokenization.

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.Locale	locale Locale.
protected boolean	mergeWhitespaceTokens Merge whitespace tokens.
protected boolean	splitAroundPeriods Check for potential splitting of tokens around periods.
protected boolean	storeWhitespaceTokens Store whitespace tokens.
protected java.lang.String	wordBreakRulesFileName Word break rules template file.
protected com.ibm.icu.text.BreakIterator	wordIterator The word based break iterator.

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer

Constructor Summary

Constructors

Constructor and Description
ICU4JBreakIteratorWordTokenizer() Create a word tokenizer that uses the ICU4J word break iterator.
ICU4JBreakIteratorWordTokenizer(java.util.Locale locale) Create a word tokenizer that uses the ICU4J word break iterator.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	createWordIterator() Create word based break iterator.
java.util.List<java.lang.String>	extractWords(java.lang.String text) Break text into word tokens.
boolean	getMergeWhitespaceTokens() Get merge whitespace tokens.
boolean	getSplitAroundPeriods() Get splitting around periods.
boolean	getStoreWhitespaceTokens() Get store whitespace tokens.
void	setMergeWhitespaceTokens(boolean mergeWhitespaceTokens) Set merge whitespace tokens.
void	setSplitAroundPeriods(boolean splitAroundPeriods) Set splitting around periods.
void	setStoreWhitespaceTokens(boolean storeWhitespaceTokens) Set store whitespace tokens.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

addWordToSentence, findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, preprocessToken, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

addWordToSentence, close, findWordOffsets, getPreTokenizer, preprocessToken, setAbbreviations, setAposTokens, setPreTokenizer

Methods inherited from interface edu.northwestern.at.utils.IsCloseable

close

Field Detail

locale

protected java.util.Locale locale

Locale.

storeWhitespaceTokens

protected boolean storeWhitespaceTokens

Store whitespace tokens.

mergeWhitespaceTokens

protected boolean mergeWhitespaceTokens

Merge whitespace tokens.

splitAroundPeriods

protected boolean splitAroundPeriods

Check for potential splitting of tokens around periods.

wordIterator

protected com.ibm.icu.text.BreakIterator wordIterator

The word based break iterator.

wordBreakRulesFileName

protected java.lang.String wordBreakRulesFileName

Word break rules template file.

Constructor Detail

ICU4JBreakIteratorWordTokenizer

public ICU4JBreakIteratorWordTokenizer()

Create a word tokenizer that uses the ICU4J word break iterator.

ICU4JBreakIteratorWordTokenizer

public ICU4JBreakIteratorWordTokenizer(java.util.Locale locale)

Create a word tokenizer that uses the ICU4J word break iterator.

Parameters:

locale - Locale to use for tokenization.

Method Detail

getStoreWhitespaceTokens

public boolean getStoreWhitespaceTokens()

Get store whitespace tokens.

Specified by:

getStoreWhitespaceTokens in interface CanTokenizeWhitespace

setStoreWhitespaceTokens

public void setStoreWhitespaceTokens(boolean storeWhitespaceTokens)

Set store whitespace tokens.

Specified by:

setStoreWhitespaceTokens in interface CanTokenizeWhitespace

getMergeWhitespaceTokens

public boolean getMergeWhitespaceTokens()

Get merge whitespace tokens.

Specified by:

getMergeWhitespaceTokens in interface CanTokenizeWhitespace

setMergeWhitespaceTokens

public void setMergeWhitespaceTokens(boolean mergeWhitespaceTokens)

Set merge whitespace tokens.

Specified by:

setMergeWhitespaceTokens in interface CanTokenizeWhitespace

getSplitAroundPeriods

public boolean getSplitAroundPeriods()

Get splitting around periods.

Specified by:

getSplitAroundPeriods in interface CanSplitAroundPeriods

setSplitAroundPeriods

public void setSplitAroundPeriods(boolean splitAroundPeriods)

Set splitting around periods.

Specified by:

setSplitAroundPeriods in interface CanSplitAroundPeriods

createWordIterator

protected void createWordIterator()

Create word based break iterator.

extractWords

public java.util.List<java.lang.String> extractWords(java.lang.String text)

Break text into word tokens.

Specified by:

extractWords in interface WordTokenizer

Specified by:

extractWords in class AbstractWordTokenizer

Parameters:

text - Text to break into word tokens.

Returns:

Input text broken into list of tokens.