edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

## Class EEBOWordTokenizer

- java.lang.Object
- - edu.northwestern.at.utils.IsCloseableObject
  - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer
    - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultWordTokenizer
      - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.EEBOWordTokenizer

- All Implemented Interfaces:
    WordTokenizer, IsCloseable, UsesLogger

      public class EEBOWordTokenizer
      extends DefaultWordTokenizer
      implements WordTokenizer

  Word tokenizer for EEBO texts.

  Do not use this when EEBO texts have been converted to TEIAnalytics format.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected static java.util.regex.Matcher` | `numberDotSpellingMatcher`
    `protected static java.util.regex.Pattern` | `numberDotSpellingPattern` Pattern to match number.word
    `protected static java.util.regex.Matcher` | `underlineCapCapMatcher`
    `protected static java.util.regex.Pattern` | `underlineCapCapPattern` Pattern to match _CapCap

    - ### Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

      `abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `EEBOWordTokenizer()` Create EEBO word tokenizer.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `java.lang.String` | `preprocessToken(java.lang.String token, java.util.List<java.lang.String> tokenList)` Preprocess a word token.

    - ### Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultWordTokenizer

      `addWordToSentence, extractWords`

    - ### Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

      `findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken`

    - ### Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

      `close`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

      `addWordToSentence, close, extractWords, findWordOffsets, getPreTokenizer, setAbbreviations, setAposTokens, setPreTokenizer`

    - ### Methods inherited from interface edu.northwestern.at.utils.IsCloseable

      `close`

- - ### Field Detail

    - #### numberDotSpellingPattern

          protected static final java.util.regex.Pattern numberDotSpellingPattern

      Pattern to match number.word

    - #### numberDotSpellingMatcher

          protected static final java.util.regex.Matcher numberDotSpellingMatcher

    - #### underlineCapCapPattern

          protected static java.util.regex.Pattern underlineCapCapPattern

      Pattern to match _CapCap

    - #### underlineCapCapMatcher

          protected static final java.util.regex.Matcher underlineCapCapMatcher

  - ### Constructor Detail

    - #### EEBOWordTokenizer

          public EEBOWordTokenizer()

      Create EEBO word tokenizer.

  - ### Method Detail

    - #### preprocessToken

          public java.lang.String preprocessToken(java.lang.String token,
                                         java.util.List<java.lang.String> tokenList)

      Preprocess a word token.
      Specified by:
        `preprocessToken` in interface `WordTokenizer`
      Overrides:
        `preprocessToken` in class `AbstractWordTokenizer`
      Parameters:
        `token` - Token to preprocess.
        `tokenList` - List of previous tokens already issued.
      Returns:
        Preprocessed token. The token list may also have been modified.