src.preprocessing package

Submodules

src.preprocessing.preprocess module

Includes all functions necessary to preprocess files.

class src.preprocessing.preprocess.PreprocessConfig(ABBREVIATION_FILE: str = None, ABBREVIATION_LIST: defaultdict = <factory>, KONJ: list = <factory>, PUNC: str = '!"\\#\\$%\\&\'\\(\\)\\*\\+, \\-\\./: ;< = >\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~«»„—¦¬', IN_WORD_SPLITTERS: str = '/', SENTENCE_ENDING: list = <factory>)

Bases: object

Class for the preprocess configuration

ABBREVIATION_FILE: str = None
ABBREVIATION_LIST: defaultdict
IN_WORD_SPLITTERS: str = '/'
KONJ: list
PUNC: str = '!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~«»„—¦¬'
SENTENCE_ENDING: list
load_abbrevs() → defaultdict

Load abbreviations from a file and build a context dictionary.

Reads abbreviations from the file and creates a dictionary where each word (including abbreviations) is mapped to its surrounding context. This allows for quick lookup of whether a word appears as an abbreviation and what words typically appear before and after it.

Returns:

A dictionary mapping each word to a list of context dictionaries. Each context dictionary contains ‘before’ and ‘after’ keys with lists of surrounding words.

Return type:

defaultdict

Example:

If the input file contains:

"Dr. John Smith"
"Prof. Jane Doe"

The returned dictionary will be:

{"dr.": [{"before": [], "after": ["john", "smith"]}],
"john": [{"before": ["dr."], "after": ["smith"]}],
"smith": [{"before": ["dr.", "john"], "after": []}],
"prof.": [{"before": [], "after": ["jane", "doe"]}],
"jane": [{"before": ["prof."], "after": ["doe"]}],
"doe": [{"before": ["prof.", "jane"], "after": []}]}

Each entry follows this structure:

{"word": [{"before": [...], "after": [...]}]}

This structure also helps prevent duplicate entries.

src.preprocessing.preprocess.check_for_abbrev(pos: int, text, preprocess_data: PreprocessConfig) → bool

Determine whether the token at a given position is an abbreviation.

Checks if the token at the specified index matches a known abbreviation and verifies that the surrounding words match the expected context patterns (words before and after) defined for that abbreviation.

Parameters:

  • pos (int) – The index position of the token to check within the text list

  • text (list of tuples) – Tokenized text where each element is a tuple containing the word and possibly other metadata

  • preprocess_data (PreprocessConfig) – Configuration object containing preprocessing settings, including sentence-ending punctuation

Returns:

True if the token is a recognized abbreviation with matching context; else False

Return type:

bool

src.preprocessing.preprocess.check_roman_numeral(word: str) → bool

Check if a word matches the format for lowercase Roman numerals.

Determines whether the input string ends with a period and contains only the lowercase Roman numeral characters ‘i’, ‘v’, or ‘x’ before the period.

Parameters:

word (str) – The input string to check

Returns:

True if the word ends with a period and is preceded only by lowercase ‘i’, ‘v’, or ‘x’ characters; else False

Return type:

bool

src.preprocessing.preprocess.execute_preprocessing() → list

Execute preprocessing on files specified in the configuration.

Processes all files defined in the configuration dictionary and yields preprocessed results for each year as they complete.

Returns:

Generator yielding tuples for each year:

  • Year (str): The year being processed

  • Data (dict): Dictionary of preprocessed data for that year

Return type:

Iterator[tuple]

src.preprocessing.preprocess.fuse_hyphens(content: str, preprocess_data: PreprocessConfig) → list

Merge words that were split across lines by hyphens in OCR output.

Processes OCR text to reunite words that were broken by line breaks and hyphenation. Each word’s coordinate metadata is preserved and combined during the fusion process.

Parameters:

  • content (str) – Raw text containing words and their coordinate metadata

  • preprocess_data (PreprocessConfig) – Configuration object containing preprocessing settings, including sentence-ending punctuation

Returns:

A list of dictionaries, each containing a “word” key (the fused text) and a “coord” key (list of associated coordinates)

Return type:

list

Example:

>>> sample_text = "Hel¬ 001122\nlo 12345\nWorld 67890"
>>> fuse_hyphens(sample_text)
# output: [{'word': 'Hello', 'coord': ['001122', '12345']},
#          {'word': 'World', 'coord': ['67890']}]

Processing logic:

  1. Lines with fewer than two tokens are skipped

  2. Words ending with special markers (like trailing hyphens) are fused with the following token

  3. Coordinate metadata from both parts is combined and preserved

  4. Returns fully assembled words with their merged coordinates

src.preprocessing.preprocess.get_year_chunk_paths(year: str) → list

Retrieve and group page paths for a given year into chunks.

Takes a year folder path and organizes its page files into chunks, returning them as grouped lists.

Parameters:

year (str) – Path to the year folder

Returns:

List of chunks, where each chunk is a list of file paths

Return type:

list

src.preprocessing.preprocess.main()
src.preprocessing.preprocess.prep_year_data_for_tagging(data: tuple) → tuple

Prepare year data for the tagging process. TODO this is the fourth function that just “starts” the preprocessing. Surely we can do better.

Processes input files for a specific year and structures the data in a format ready for tagging operations.

Parameters:

data (tuple) –

Tuple containing three elements:

  • Year (str): The year being processed

  • Paths (list): List of input file paths for that year

  • Config (dict): Configuration dictionary

Returns:

Tuple containing two elements:

  • Data dictionary (dict): Keys are years, values are prepared tagging data

  • Year (str): The year that was processed

Return type:

tuple

src.preprocessing.preprocess.preprocess_file(infile: str) → list

Preprocess OCR output file and return structured sentences.

Reads and processes an OCR-generated file, applying tokenization and sentence segmentation to produce a structured list of sentences.

Parameters:

infile (str) – Path to the input file to preprocess

Returns:

List of sentences, where each sentence is a list of token dictionaries

Return type:

list

Raises:

FileNotFoundError: If the input file does not exist

Raises:

KeyError: If the configuration dictionary is missing required keys

Example:

>>> sentences = preprocess_file("/path/to/input/file")
src.preprocessing.preprocess.split_sentences(content: list, preprocess_data: PreprocessConfig) → list

Split a list of tokens into sentences based on ending punctuation.

Segments tokens into separate sentences by identifying sentence-ending punctuation marks as defined in the preprocessing configuration.

Parameters:

  • content (list) – List of token dictionaries, each containing token information

  • preprocess_data (PreprocessConfig) – Configuration object containing preprocessing settings, including sentence-ending punctuation

Returns:

List of sentences, where each sentence is a list of token dictionaries

Return type:

list

src.preprocessing.preprocess.start_preprocessing(year_directories: List[str])

Preprocess files from multiple year directories in chunks. TODO this is the third function that just “starts” the preprocessing. Surely we can do better.

Iterates through year directories and processes their files in manageable chunks, yielding results for each year as they complete.

Parameters:

year_directories (List[str]) – List of year directory paths to process

Returns:

Generator yielding tuples for each year:

  • Year (str): The year directory being processed

  • Data (dict): Dictionary of preprocessed data for that year

Return type:

Iterator[tuple]

src.preprocessing.preprocess.timed_execute_preprocessing() → dict

Runs execute preprocessing but also logs the time it took to run.

src.preprocessing.preprocess.tokenize(content: list, preprocess_data: PreprocessConfig) → list

Tokenize words and split punctuation from a list of word-coordinate pairs.

Processes a list of dictionaries containing words and their coordinates, separating punctuation and normalizing text while preserving coordinate metadata.

Input format:

Each dictionary must contain:

{"word": "ExampleWord", "coord": "ExampleCoordinate"}

Tokenization process:

  1. Separates leading and trailing punctuation from words, except periods in abbreviations

  2. Splits certain mid-word punctuation (e.g., semicolons, dashes) into separate tokens

  3. Converts fully uppercase tokens to title case (e.g., “HANS” → “Hans”)

  4. Handles periods at word endings as either abbreviations or separate tokens

Example:

Input:

[{"word": "Hello", "coord": "1209745"},
 {"word": "v.a.", "coord": "1908234"}]

Output:

[{"token": "Hello", "coord": "1;2;0;9;7;4;5:main",
  "normalized": "Hello"},
 {"token": "v.", "coord": "1;9;0;8;2;3;4:main"},
 {"token": "a.", "coord": "1;9;0;8;2;3;4:main"}]
Parameters:

  • content (list of dict) –

    List of dictionaries, each containing:

    • ”word” (str): The raw text token

    • ”coord” (str): The coordinate or reference string for the token

  • preprocess_data (PreprocessConfig) – Configuration object containing preprocessing settings, including sentence-ending punctuation

Returns:

List of dictionaries, each containing:

  • ”token” (str): The processed token string

  • ”coord” (str): The coordinate string with type suffix (e.g., ‘:main’, ‘:lpunc’, ‘:rpunc’)

  • ”normalized” (str, optional): Title-cased version if originally uppercase

Return type:

list of dict

Module contents