edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class AbstractPostTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

All Implemented Interfaces:

PostTokenizer, UsesLogger

Direct Known Subclasses:

DefaultPostTokenizer, EEBOPostTokenizer

public abstract class AbstractPostTokenizer
extends IsCloseableObject
implements PostTokenizer, UsesLogger

Post tokenizer which processes an extracted token.

This post tokenizer leaves the token unchanged.

Field Summary

Fields

Modifier and Type	Field and Description
protected Logger	logger Logger used for output.

Constructor Summary

Constructors

Constructor and Description
AbstractPostTokenizer() Create a postTokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
Logger	getLogger() Get the logger.
java.lang.String[]	postTokenize(java.lang.String token) Process a token after tokenization.
void	setLogger(Logger logger) Set the logger.

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PostTokenizer

close

Field Detail

logger

protected Logger logger

Logger used for output.

Constructor Detail

AbstractPostTokenizer

public AbstractPostTokenizer()

Create a postTokenizer.

Method Detail

postTokenize

public java.lang.String[] postTokenize(java.lang.String token)

Process a token after tokenization.

Specified by:

postTokenize in interface PostTokenizer

Parameters:

token - The token to process after tokenization.

Returns:

Array of two strings. [0] = the token minimally processed. [1] = the token maximally processed.

The minimally processed token is typically results in an original spelling.

The maximally processed token typically results in a partially or completely standardized spelling.

These may be identical.

getLogger

public Logger getLogger()

Get the logger.

Specified by:

getLogger in interface UsesLogger

Returns:

The logger.

setLogger

public void setLogger(Logger logger)

Set the logger.

Specified by:

setLogger in interface UsesLogger

Parameters:

logger - The logger.