edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class EEBOPreTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.EEBOPreTokenizer

All Implemented Interfaces:

PreTokenizer, UsesLogger

public class EEBOPreTokenizer
extends AbstractPreTokenizer
implements PreTokenizer

A pretokenizer for original form EEBO texts (not converted to TEIAnalytics).

Field Summary

Fields

Modifier and Type	Field and Description
protected static PatternReplacer	doubleBackTicksReplacer Double back-ticks.
protected static java.lang.String	EEBOAlwaysSeparators EEBO separators do not include the vertical bars.
protected static PatternReplacer	singleBackTicksReplacer Single back-tick followed by a capital letter.
protected static PatternReplacer	wordOrSpanGapReplacer Word or span gap.

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

alwaysSeparators, alwaysSeparatorsReplacer, asterisks, commaSeparator, commaSeparatorReplacer, hyphens, logger, periods

Constructor Summary

Constructors

Constructor and Description
EEBOPreTokenizer() Create an EEBO pretokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	pretokenize(java.lang.String line) Prepare text for tokenization.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

getLogger, setLogger

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PreTokenizer

close

Field Detail

EEBOAlwaysSeparators

protected static final java.lang.String EEBOAlwaysSeparators

EEBO separators do not include the vertical bars.

See Also:

Constant Field Values

wordOrSpanGapReplacer

protected static final PatternReplacer wordOrSpanGapReplacer

Word or span gap.

doubleBackTicksReplacer

protected static final PatternReplacer doubleBackTicksReplacer

Double back-ticks.

singleBackTicksReplacer

protected static final PatternReplacer singleBackTicksReplacer

Single back-tick followed by a capital letter.

Constructor Detail

EEBOPreTokenizer

public EEBOPreTokenizer()

Create an EEBO pretokenizer.

Method Detail

pretokenize

public java.lang.String pretokenize(java.lang.String line)

Prepare text for tokenization.

Specified by:

pretokenize in interface PreTokenizer

Overrides:

pretokenize in class AbstractPreTokenizer

Parameters:

line - The text to prepare for tokenization,

Returns:

The pretokenized text.