edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class EEBOPostTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.EEBOPostTokenizer

All Implemented Interfaces:

PostTokenizer, UsesLogger

public class EEBOPostTokenizer
extends AbstractPostTokenizer
implements PostTokenizer

Post tokenizer for EEBO texts.

This post tokenizer processes tokens extracted from EEBO corpus texts. It removes soft hyphens and regularizes some EEBO specific tagging. This can be used for either original format EEBO texts or EEBO texts in TEIAnalytics format.

Field Summary

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

logger

Constructor Summary

Constructors

Constructor and Description
EEBOPostTokenizer() Create an EEBO PostTokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String[]	postTokenize(java.lang.String token) Process a token after tokenization.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

getLogger, setLogger

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PostTokenizer

close

Constructor Detail

EEBOPostTokenizer

public EEBOPostTokenizer()

Create an EEBO PostTokenizer.

Method Detail

postTokenize

public java.lang.String[] postTokenize(java.lang.String token)

Process a token after tokenization.

Specified by:

postTokenize in interface PostTokenizer

Overrides:

postTokenize in class AbstractPostTokenizer

Parameters:

token - The token to process after tokenization.

Returns:

Array of two strings. [0] = the token minimally processed. [1] = the token maximally processed.

The minimally processed token is typically results in an original spelling.

The maximally processed token typically results in a partially or completely standardized spelling.

These may be identical.