edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

## Class ICU4JBreakIteratorWordTokenizer

- java.lang.Object
- - edu.northwestern.at.utils.IsCloseableObject
  - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer
    - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.ICU4JBreakIteratorWordTokenizer

- All Implemented Interfaces:
    CanSplitAroundPeriods, CanTokenizeWhitespace, WordTokenizer, IsCloseable, UsesLogger

      public class ICU4JBreakIteratorWordTokenizer
      extends AbstractWordTokenizer
      implements WordTokenizer, CanTokenizeWhitespace, CanSplitAroundPeriods

  Word tokenizer which uses ICU library for tokenization.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected java.util.Locale` | `locale` Locale.
    `protected boolean` | `mergeWhitespaceTokens` Merge whitespace tokens.
    `protected boolean` | `splitAroundPeriods` Check for potential splitting of tokens around periods.
    `protected boolean` | `storeWhitespaceTokens` Store whitespace tokens.
    `protected java.lang.String` | `wordBreakRulesFileName` Word break rules template file.
    `protected com.ibm.icu.text.BreakIterator` | `wordIterator` The word based break iterator.

    - ### Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

      `abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `ICU4JBreakIteratorWordTokenizer()` Create a word tokenizer that uses the ICU4J word break iterator.
    `ICU4JBreakIteratorWordTokenizer(java.util.Locale locale)` Create a word tokenizer that uses the ICU4J word break iterator.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `createWordIterator()` Create word based break iterator.
    `java.util.List<java.lang.String>` | `extractWords(java.lang.String text)` Break text into word tokens.
    `boolean` | `getMergeWhitespaceTokens()` Get merge whitespace tokens.
    `boolean` | `getSplitAroundPeriods()` Get splitting around periods.
    `boolean` | `getStoreWhitespaceTokens()` Get store whitespace tokens.
    `void` | `setMergeWhitespaceTokens(boolean mergeWhitespaceTokens)` Set merge whitespace tokens.
    `void` | `setSplitAroundPeriods(boolean splitAroundPeriods)` Set splitting around periods.
    `void` | `setStoreWhitespaceTokens(boolean storeWhitespaceTokens)` Set store whitespace tokens.

    - ### Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

      `addWordToSentence, findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, preprocessToken, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken`

    - ### Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

      `close`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

      `addWordToSentence, close, findWordOffsets, getPreTokenizer, preprocessToken, setAbbreviations, setAposTokens, setPreTokenizer`

    - ### Methods inherited from interface edu.northwestern.at.utils.IsCloseable

      `close`

- - ### Field Detail

    - #### locale

          protected java.util.Locale locale

      Locale.

    - #### storeWhitespaceTokens

          protected boolean storeWhitespaceTokens

      Store whitespace tokens.

    - #### mergeWhitespaceTokens

          protected boolean mergeWhitespaceTokens

      Merge whitespace tokens.

    - #### splitAroundPeriods

          protected boolean splitAroundPeriods

      Check for potential splitting of tokens around periods.

    - #### wordIterator

          protected com.ibm.icu.text.BreakIterator wordIterator

      The word based break iterator.

    - #### wordBreakRulesFileName

          protected java.lang.String wordBreakRulesFileName

      Word break rules template file.

  - ### Constructor Detail

    - #### ICU4JBreakIteratorWordTokenizer

          public ICU4JBreakIteratorWordTokenizer()

      Create a word tokenizer that uses the ICU4J word break iterator.

    - #### ICU4JBreakIteratorWordTokenizer

          public ICU4JBreakIteratorWordTokenizer(java.util.Locale locale)

      Create a word tokenizer that uses the ICU4J word break iterator.
      Parameters:
        `locale` - Locale to use for tokenization.

  - ### Method Detail

    - #### getStoreWhitespaceTokens

          public boolean getStoreWhitespaceTokens()

      Get store whitespace tokens.
      Specified by:
        `getStoreWhitespaceTokens` in interface `CanTokenizeWhitespace`

    - #### setStoreWhitespaceTokens

          public void setStoreWhitespaceTokens(boolean storeWhitespaceTokens)

      Set store whitespace tokens.
      Specified by:
        `setStoreWhitespaceTokens` in interface `CanTokenizeWhitespace`

    - #### getMergeWhitespaceTokens

          public boolean getMergeWhitespaceTokens()

      Get merge whitespace tokens.
      Specified by:
        `getMergeWhitespaceTokens` in interface `CanTokenizeWhitespace`

    - #### setMergeWhitespaceTokens

          public void setMergeWhitespaceTokens(boolean mergeWhitespaceTokens)

      Set merge whitespace tokens.
      Specified by:
        `setMergeWhitespaceTokens` in interface `CanTokenizeWhitespace`

    - #### getSplitAroundPeriods

          public boolean getSplitAroundPeriods()

      Get splitting around periods.
      Specified by:
        `getSplitAroundPeriods` in interface `CanSplitAroundPeriods`

    - #### setSplitAroundPeriods

          public void setSplitAroundPeriods(boolean splitAroundPeriods)

      Set splitting around periods.
      Specified by:
        `setSplitAroundPeriods` in interface `CanSplitAroundPeriods`

    - #### createWordIterator

          protected void createWordIterator()

      Create word based break iterator.

    - #### extractWords

          public java.util.List<java.lang.String> extractWords(java.lang.String text)

      Break text into word tokens.
      Specified by:
        `extractWords` in interface `WordTokenizer`
      Specified by:
        `extractWords` in class `AbstractWordTokenizer`
      Parameters:
        `text` - Text to break into word tokens.
      Returns:
        Input text broken into list of tokens.