edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

## Class EEBOPostTokenizer

- java.lang.Object
- - edu.northwestern.at.utils.IsCloseableObject
  - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer
    - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.EEBOPostTokenizer

- All Implemented Interfaces:
    PostTokenizer, UsesLogger

      public class EEBOPostTokenizer
      extends AbstractPostTokenizer
      implements PostTokenizer

  Post tokenizer for EEBO texts.

  This post tokenizer processes tokens extracted from EEBO corpus texts. It removes soft hyphens and regularizes some EEBO specific tagging. This can be used for either original format EEBO texts or EEBO texts in TEIAnalytics format.

- - ### Field Summary

    - ### Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

      `logger`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `EEBOPostTokenizer()` Create an EEBO PostTokenizer.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `java.lang.String[]` | `postTokenize(java.lang.String token)` Process a token after tokenization.

    - ### Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

      `getLogger, setLogger`

    - ### Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

      `close`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PostTokenizer

      `close`

- - ### Constructor Detail

    - #### EEBOPostTokenizer

          public EEBOPostTokenizer()

      Create an EEBO PostTokenizer.

  - ### Method Detail

    - #### postTokenize

          public java.lang.String[] postTokenize(java.lang.String token)

      Process a token after tokenization.
      Specified by:
        `postTokenize` in interface `PostTokenizer`
      Overrides:
        `postTokenize` in class `AbstractPostTokenizer`
      Parameters:
        `token` - The token to process after tokenization.
      Returns:
        Array of two strings. [0] = the token minimally processed. [1] = the token maximally processed.

        The minimally processed token is typically results in an original spelling.

        The maximally processed token typically results in a partially or completely standardized spelling.

        These may be identical.