edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class AbstractPreTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

All Implemented Interfaces:

PreTokenizer, UsesLogger

Direct Known Subclasses:

DefaultPreTokenizer, EccoPreTokenizer, EEBOPreTokenizer, NoopPreTokenizer

public abstract class AbstractPreTokenizer
extends IsCloseableObject
implements PreTokenizer, UsesLogger

Default pretokenizes which prepares a string for tokenization.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.lang.String	alwaysSeparators Pattern to match characters which are always separators.
protected static PatternReplacer	alwaysSeparatorsReplacer Always Separators replacer pattern.
protected static java.lang.String	asterisks Pattern to match one or more asterisk.
protected static java.lang.String	commaSeparator Pattern to match comma as a separator.
protected static PatternReplacer	commaSeparatorReplacer Comma separator replacer pattern.
protected static java.lang.String	hyphens Pattern to match two or more hyphens in a row.
protected Logger	logger Logger used for output.
protected static java.lang.String	periods Pattern to match three or more periods.

Constructor Summary

Constructors

Constructor and Description
AbstractPreTokenizer() Create a preTokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
Logger	getLogger() Get the logger.
java.lang.String	pretokenize(java.lang.String line) Prepare text for tokenization.
void	setLogger(Logger logger) Set the logger.

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PreTokenizer

close

Field Detail

periods

protected static final java.lang.String periods

Pattern to match three or more periods.

See Also:

Constant Field Values

asterisks

protected static final java.lang.String asterisks

Pattern to match one or more asterisk.

See Also:

Constant Field Values

hyphens

protected static final java.lang.String hyphens

Pattern to match two or more hyphens in a row.

See Also:

Constant Field Values

commaSeparator

protected static final java.lang.String commaSeparator

Pattern to match comma as a separator.

See Also:

Constant Field Values

logger

protected Logger logger

Logger used for output.

alwaysSeparators

protected static final java.lang.String alwaysSeparators

Pattern to match characters which are always separators.

Unicode ? (BLACKCIRCLE) is the dot character which marks character lacunae. This is not a token separator. Neither is Unicode • (SOLIDCIRCLE) which was used in the old EEBO format TCP files to mark character lacunae.

Unicode ?, the non-breaking hyphen, is not treated as a token separator.

Unicode ? (DEGREES_MARK) is degrees quote symbol. Unicode ? (MINUTES_MARK) is minutes quote symbol. Unicode ? (SECONDS_MARK) is seconds quote symbol. These are not token separators.

Unicode ‘ (LSQUOTE) is left single curly quote. Unicode ’ (RSQUOTE) is right single curly quote. These may or may not be token separators. It is up to the word tokenizer to decide.

Unicode “ (LDQUOTE) is left double curly quote. Unicode ” (RDQUOTE) is right double curly quote. These are token separators.

See Also:

Constant Field Values

alwaysSeparatorsReplacer

protected static PatternReplacer alwaysSeparatorsReplacer

Always Separators replacer pattern.

commaSeparatorReplacer

protected static PatternReplacer commaSeparatorReplacer

Comma separator replacer pattern.

Constructor Detail

AbstractPreTokenizer

public AbstractPreTokenizer()

Create a preTokenizer.

Method Detail

getLogger

public Logger getLogger()

Get the logger.

Specified by:

getLogger in interface UsesLogger

Returns:

The logger.

setLogger

public void setLogger(Logger logger)

Set the logger.

Specified by:

setLogger in interface UsesLogger

Parameters:

logger - The logger.

pretokenize

public java.lang.String pretokenize(java.lang.String line)

Prepare text for tokenization.

Specified by:

pretokenize in interface PreTokenizer

Parameters:

line - The text to prepare for tokenization,

Returns:

The pretokenized text.