src package

Subpackages

• src.preprocessing package
  • Submodules
  • src.preprocessing.preprocess module
    • PreprocessConfig
      • PreprocessConfig.ABBREVIATION_FILE
      • PreprocessConfig.ABBREVIATION_LIST
      • PreprocessConfig.IN_WORD_SPLITTERS
      • PreprocessConfig.KONJ
      • PreprocessConfig.PUNC
      • PreprocessConfig.SENTENCE_ENDING
      • PreprocessConfig.load_abbrevs()
    • check_for_abbrev()
    • check_roman_numeral()
    • execute_preprocessing()
    • fuse_hyphens()
    • get_year_chunk_paths()
    • main()
    • prep_year_data_for_tagging()
    • preprocess_file()
    • split_sentences()
    • start_preprocessing()
    • timed_execute_preprocessing()
    • tokenize()
  • Module contents

Submodules

src.aggregation module

Aggregation module

src.aggregation.abbrev_firstname_match(reference: dict, aggregated_names: list) → list

Find matches by abbreviated first name and last name.

Searches the aggregated names for entries where both the abbreviated first name and last name match the reference.

Parameters:

• reference (dict) – Dictionary containing the reference person entity information

• aggregated_names (list) – List of existing aggregated units to search

Returns:

List of matching aggregated units; empty list if no matches found

Return type:

list

src.aggregation.aggregate_names(input_triplet) → list

Aggregate person entity mentions into unified entities.

Groups multiple mentions of the same person into single aggregated units based on name matching and contextual information from the journal.

Parameters:

input_triplet (tuple) – Triplet consisting of (in this order): * Tuple (mag, year) * List of person entity dictionaries to aggregate * List of paths for the tagging output files of this mag-year tuple

Returns:

Triplet consisting of (in this order): * List of aggregated person entities with consolidated mentions * Tuple (mag, year) * List of paths for the tagging output files of this mag-year tuple

Return type:

list

src.aggregation.aggregate_place(found: dict, place: dict) → None

Merge place reference information into an existing aggregated unit.

Updates an existing found dictionary in place by adding the information from the place reference.

Parameters:

• found (dict) – The existing aggregated place unit to update

• place (dict) – Dictionary containing place entity information and its location references in the journal

src.aggregation.aggregate_places(all_places: list, aggregated_places: list)

Aggregate place entity references into unified units.

Processes all place entity references and either merges them into existing aggregated units or creates new units as appropriate, updating the aggregated places list in place.

Parameters:

• all_places (list) – List of place entity dictionaries containing place information and their location references in the journal

• aggregated_places (list) – List of existing aggregated place units, modified in place by merging or adding new units

src.aggregation.aggregate_with(namepart_dict: dict, aggregated_names: list, namepart: str) → None

Aggregate person entity references by a specified name part.

Processes references grouped by a name part (e.g., last name) and either merges them into existing aggregated units or creates new units where appropriate.

Parameters:

• namepart_dict (dict) – Dictionary mapping name parts to lists of person entity references. Keys are name parts (e.g., last names), values are lists of reference dictionaries

• aggregated_names (list) – List of existing aggregated units, modified in place by merging or adding new units

• namepart (str) – The type of name part to use for aggregation (e.g., ‘lastname’, ‘firstname’)

Raises:

Exception – If the provided namepart is not recognized

src.aggregation.clean_lastname(word: str) → str

Remove prefix and suffix patterns from a last name.

Applies global PRE and POSTPATTERN regex patterns to strip common prefixes and suffixes from the given word.

Parameters:

word (str) – The last name string to clean

Returns:

The cleaned last name with patterns removed

Return type:

str

src.aggregation.clean_up_aggregation(aggregated_names: list) → list

Clean and restructure the aggregated person list.

Sorts the aggregated person entities and reformats the internal data structure of their references for consistency.

Parameters:

aggregated_names (list) – List of aggregated person entity dictionaries

Returns:

Cleaned and restructured list of aggregated person entities

Return type:

list

src.aggregation.clean_up_aggregation_places(aggregated_places: list, last_index: int) → list

Clean and restructure the aggregated places list.

Sorts the aggregated place entities and reformats the internal data structure of their references for consistency.

Parameters:

• aggregated_places (list) – List of aggregated place entity dictionaries

• last_index (int) – Index of the last place entity processed

Returns:

Cleaned and restructured list of aggregated place entities

Return type:

list

src.aggregation.create_new_aggregated_place(reference: dict) → dict

Create a new aggregated unit for a place entity.

Transforms a place entity reference into an aggregated format where values are converted to sets of tuples for efficient tracking of where the place appears in the journal.

Parameters:

reference (dict) – Dictionary containing place entity information and its location references in the journal

Returns:

Dictionary with the same structure but values converted to sets of tuples for aggregation

Return type:

dict

src.aggregation.create_new_aggregated_unit(ref: dict) → dict

Create a new aggregated unit for a person entity.

Transforms a person entity reference into an aggregated format where values are converted to sets of tuples for efficient tracking of where the person appears in the journal.

Parameters:

ref (dict) – Dictionary containing person entity information and where the person appeared in the journal

Returns:

Dictionary with the same structure but values converted to sets of tuples for aggregation

Return type:

dict

src.aggregation.decide_candidates(ref: dict, candidates: list, aggregated_names: list, verbose=False) → None

Select the best matching candidate and merge, or create a new unit.

Evaluates potential candidate matches for a person entity reference. If a suitable match is found, merges the reference into that aggregated unit in-place. Otherwise, creates a new aggregated unit and adds it to the list.

Parameters:

• ref (dict) – A dictionary containing information about the person entity and where the person appeared in the journal

• candidates (list) – A list of dictionaries representing existing aggregated units that are potential matches for the reference

• aggregated_names (list) – List of all aggregated units, modified in place by adding new units or updating existing ones

• verbose (bool, optional) – If True, prints detailed information about the reference and the selected candidate, defaults to False

src.aggregation.execute_aggregation(postprocessed_data, tasks: list, timed=True)

Aggregate postprocessed data and optionally log execution time.

Parameters:

• postprocessed_data (list of triplets (year, data, paths)) – Postprocessed entity data ready for aggregation

• tasks (list) – List of tasks to perform during aggregation

• timed (bool, optional) – If True, logs execution time. Defaults to True

Returns:

Generator yielding dictionaries of aggregated data, each containing a dictionary with keys “agg_data” containing tagged people and places and a key “paths” with a list of all the tagged files that belong to this year

Return type:

Iterator[dict]

Raises:

Exception – If ‘post’ is not included in the tasks list, an exception is raised indicating that ‘post,agg,link’ must be called together.

src.aggregation.find_place_match(place_name: str, place_type: str, aggregated_places: list) → dict

Find an exact match for a place name.

Searches the aggregated place names for an entry that exactly matches both the place name and type.

Parameters:

• place_name (str) – The place name string to match

• place_type (str) – The entity type of the place (e.g., ‘GPE’, ‘LOC’)

• aggregated_places (list) – List of existing aggregated place units to search

Returns:

The matching aggregated place unit if found, else None

Return type:

dict or None

src.aggregation.full_firstname_match(ref: dict, aggregated_names: list) → dict

Find an exact match by full first name and last name.

Searches the aggregated names for an entry that matches both the complete first name and last name of the reference.

Parameters:

• ref (dict) – Dictionary containing the reference person entity information

• aggregated_names (list) – List of existing aggregated units to search

Returns:

The matching aggregated unit if found, else None

Return type:

dict

src.aggregation.map_genitive_places(all_names: list, place_list: list) → None

Map genitive place name forms to their base forms.

Converts genitive versions of place names (ending in ‘s’) to their non-genitive base forms by removing the trailing ‘s’ where appropriate, updating place_list in place.

Parameters:

• all_names (list) – List of place names mentioned in the journal

• place_list (list) – List of places, modified in place to map genitive forms to base forms

src.aggregation.map_genitive_versions(all_names: list, lastname_dict: dict, key: str) → None

Map genitive name forms to their base forms.

Converts genitive versions of names (ending in ‘s’) to their non-genitive base forms by removing the trailing ‘s’ where appropriate, updating the lastname_dict in place.

Parameters:

• all_names (list) – List of person names mentioned in the journal

• lastname_dict (dict) – Dictionary of last names, modified in place to map genitive forms to base forms

• key (str) – The name part to process (e.g., ‘lastname’, ‘firstname’)

src.aggregation.merge_to_existing_aggregated_unit(match_in: dict, ref: dict) → None

Merge reference information into an existing aggregated unit.

Updates the match_in dictionary in place by adding entity information from the reference.

Parameters:

• match_in (dict) – The existing aggregated unit to update

• ref (dict) – Dictionary containing person entity information and where the person appeared in the journal

src.aggregation.only_abbrev_firstname_match(ref: dict, aggregated_names: list) → list

Find matches by abbreviated first name only.

Searches the aggregated names for entries where the abbreviated first name matches the reference, regardless of last name.

Parameters:

• ref (dict) – Dictionary containing the reference person entity information

• aggregated_names (list) – List of existing aggregated units to search

Returns:

List of matching aggregated units. Empty list if no matches found

Return type:

list

src.aggregation.only_firstname_match(ref: dict, aggregated_names: list) → list

Find matches by first name only.

Searches the aggregated names for entries where the first name matches the reference, regardless of last name.

Parameters:

• ref (dict) – Dictionary containing the reference person entity information

• aggregated_names (list) – List of existing aggregated units to search

Returns:

List of matching aggregated units. Empty list if no matches found

Return type:

list

src.aggregation.only_lastname_match(reference: dict, aggregated_names: list) → list

Find matches by last name only.

Searches the aggregated names for entries where the last name matches the reference, regardless of first name.

Parameters:

• reference (dict) – Dictionary containing the reference person entity information

• aggregated_names (list) – ist of existing aggregated units to search

Returns:

List of matching aggregated units. Empty list if no matches found

Return type:

list

src.aggregation.others_match(reference: dict, aggregated_names: list) → list

Find matches by the ‘others’ field only.

Searches the aggregated names for entries where the ‘others’ field matches the reference, regardless of first or last name.

Parameters:

• reference (dict) – Dictionary containing the reference person entity information

• aggregated_names (list) – List of existing aggregated units to search

Returns:

List of matching aggregated units; empty list if no matches found

Return type:

list

src.evaluation module

Evaluate performance on entity and reference level

src.evaluation.execute_evaluation(eval_level: str, top_k=3, timed=True) → None

Evaluates the F1-score of the data based on the given configuration and evaluation level

At eval_level “ent”, this is done on entity level. At eval_level “ref”, this is done on reference level, which weighs the entities more that occur more, thus should be linked correctly, since we have more context on them.

Evaluation is done for all magazines in our ground-truth folder to make the evaluations straightforward to compare. If you would like to evaluate only one magazine, you need to create a new directory with only that magazine’s ground-truth in it, change that path in the config, and then run eval.

Parameters:

• eval_level (str) – Evaluation level to be used (“ent” or “ref”).

• top_k (int) – How many candidates to truncate the results to, defaults to 3.

• timed (bool) – Boolean indicating whether to time the execution and log it, defaults to True.

Raises:

NotImplementedError – If no path to GT was passed.

It creates several directories and json files in the output directory specified in the config.

src.linking module

Linking module

src.linking.backend_api_call(content, model, model_name, collection_name, backend_url)

We call our API backend. If OIDC_TOKEN_URL is set, it performs the Paramanera handshake and attaches the Bearer token.

src.linking.compare_to_target_ids_multiplexed(queryids: list[int], text: list[str], target_text_ids: list[list[str]], backend_url: str, collection_name: str, model: str, model_name: str)

Compare query texts to target texts using embeddings and return distances.

Sends text queries to an embedding backend to compute similarity distances between the query texts and a specified list of target text IDs.

Parameters:

• queryids (list[int]) – List of query identifiers

• text (list[str]) – List of text strings to embed and compare

• target_text_ids (list[list[str]]) – List of target text ID lists to compare against

• backend_url (str) – URL of the embedding backend service

• collection_name (str, optional) – Name of the collection in the backend

• model (str) – Model backend to use for generating embeddings

• model_name (str) – Specific model name to use

Returns:

List of results, else logs error

Return type:

list

src.linking.compare_vector_to_text_ids_multiplexed(input: list[dict], vectors: list[list[float]], collection_name: str, distance_metric: str = 'cosine') → list[dict]

Compare query embeddings to candidate embeddings stored in Milvus, restricted by reference_text_ids per query.

input: list of dicts with keys “query_id”: int, “query_text”: str, “reference_text_ids”: list[str] vectors: list of embeddings (aligned with input order) collection_name: Milvus collection ditance_metric: “cosine” | “l2” | “ip”

src.linking.execute_linking(data: dict, tasks: list, timed=True) → None

Links the aggregated data based on the given configuration and tasks.

Parameters:

• data (dict) – The data that has been aggregated and is ready for linking. The keys are the magazine-year shortnames, the values are the dictionaries of the aggregated entities.

• tasks (list) – List of tasks to be performed during linking.

• timed (bool) – Boolean indicating whether to time the execution and log it, defaults to True.

Raises:

Exception – If the tasks list doesn’t include ‘finish’ or if ‘agg’ and ‘post’ is not included in the tasks list, an exception is raised indicating that ‘post,agg,link’ or ‘finish’ must be called together.

src.linking.find_links(data_in) → list

Links all the people in the given data.

Parameters:

data_in (3-tuple) –

• mag_year (tuple): A tuple contaning the magazine shortname and year of the journal we’re currently processing in that order.

• data (list): The list of dictionaries created through aggregating PER entities in our magazine-year tuple.

• tagging_paths (list): The list of paths to the tagging output of this

  magazine_year tuple.

Returns:

The same 4-tuple, but the data list is now an ordered list of person entity dictionaries containing a “gnd_ids” key with the candidates list if there are no candidates, or exactly one candidate, else “context” and “same_score_cand”.

Return type:

tuple

src.linking.get_candidates(person: dict, year: str, gnd_limit: int, wikidata_limit: int) → dict

Searches the GND and Wikidata index for candidates of a given person.

Parameters:

• person (dict) – A dictionary with various information on the given person entity.

• year (str) – The year in which this magazine was published.

• gnd_limit (int) – The number of candidates to truncate our GND candidate list to.

• wikidata_limit (int) – The number of candidates to truncate out wikidata candidate list to.

Returns:

Dictionary of candidates for the given person, containing information on the candidates themselves.

Return type:

dict

src.linking.get_paramanera_token() → str | None

Fetches an OAuth token for the Paramanera API if configured.

src.linking.get_person_context(per: dict, tagging_output_paths: list) → str

Retrieve and concatenate context surrounding all person mentions.

Extracts the text surrounding each reference to the given person and concatenates all context windows into a single string.

Parameters:

• per (dict) – Person entity dictionary containing a “references” key with mention locations

• tagging_output_paths (list) – List of the paths to the tagging output files

Returns:

Concatenated context text from all references to the person

Return type:

str

Raises:

Exception – If the tagging output path is not a valid path

src.linking.get_person_context_reflevel(per: dict, tagging_output_path: str) → list

Retrieve context surrounding all person mentions.

Extracts the text surrounding each reference to the given person and returns them as a list.

Parameters:

• per (dict) – Person entity dictionary containing a “references” key with mention locations

• tagging_output_path (str) – Path to the tagging output file

Returns:

List of context window strings, one per reference to the person

Return type:

list

Raises:

Exception – If the tagging output path is not a valid path

src.linking.link_person(data_in) → dict

Searches GND and Wikidata for candidates, sets a field “gnd_ids” in the person dictionary with at most linked_persons_limit candidates.

Parameters:

data_in (4-tuple) –

• mag_year (tuple): Shortname of the magazine this person is mentioned in. Of the form (“obl”, “2004_000”).

• year (str): The year this magazine was published in.

• person (dict): A dictionary with keys “lastname”, “firstname”, “abbr_firstname”, “address”, “titles”, “profession”, “other”, “references”, “type” and “id”.

  • fields up until “references” are lists, except for lastname which is a string

  • references is a defaultdict(list), a list of page_names where each page_name has coords and sentences associated with it where the candidate was mentioned

  • type is a string denoting the entity type

  • id is an internal id

• tagging_paths (list): List of paths to the tagging output files for

  this magazine-year.

Returns:

The changed person dictionary, now containing the gnd_ids if there are no candidates or exactly one candidate, else “context” and “same_score_cand”.

Return type:

dict

If the list of candidates is longer than linked_persons_limit, we check if the first letters of the firstnames match the abbreviated firstnames. If not, those candidates are deleted first, before we truncate to linked_persons_limit.

src.linking.prep_person_entry(person: dict, mag_year: str) → None

Normalizes and cleans up proper names in the given person dictionary.

Parameters:

• person (dict) – A person dictionary containing at least the keys “firstname”, “abbr_firstname”, “lastname”, and “profession”.

• mag_year (tuple) – Shortname of the magazine this person is mentioned in. Of the form (“obl”, “2004_000”).

src.linking.prep_person_out(person: dict) → None

For a given dictionary, joins the fields “firstname”, “abbr_firstname” and “lastname” repectively to make them strings. Maps the candidate scores to a “confidence score” on a scale from 1 to 5 for the frontend.

Parameters:

person (dict) – A person dictionary containing at least the keys “firstname”, “abbr_firstname”, and “lastname”.

src.linking.prep_word(word: str) → str

Normalizes and cleans up given string.

Parameters:

word (str) – String to normalize.

Returns:

Normalized string.

Return type:

str

Example:

‘OTMAR Mäder’ -> ‘Otmar Mäder’

src.linking.remove_obsolete_abbrevs(fnames: list, abbr_firstnames: list) → list

Removes abbreviated firstnames that are already covered by full firstnames.

Parameters:

• fnames (list) – List of firstnames

• abbr_firstnames (list) – List of abbreviated firstnames

Returns:

List of firstnames where the obsolete abbreviated firstnames have been removed.

Return type:

list

Example:

fnames = [“R.”, “Richard”] => fnames = [“Richard”]

src.linking.update_per_dict_score(dict_in: dict, dict_to_add: dict, strategy='max') → dict

src.postprocess module

Read entities, then aggregate them.

Update 09.03.: We do not append full articles anymore, because we’d need to attach too much information which bloats the file. Instead, we only append the article-ID, so in the other steps like aggregation, linking and prep_import we can simply use that id to get the article information from the xml.

src.postprocess.add_info_to_entity(entity: dict, tag: str, token: dict, pageNo: str, sentNo: str, pageName: str, articles: list, pid: int) → None

Adds information to a person entity based on the provided token dictionary and metadata about the page.

Parameters:

• entity (dict) – The person entity dictionary to update.

• tag (str) – The type of the entity information (e.g., “LN” for last name, “FN” for first name).

• token (dict) –

  A dictionary containing token information, including

  • ”token” (str): The token text.

  • ”coord” (tuple): The coordinates of the token.

• pageNo (str) – The page number where the entity is located.

• sentNo (str) – The sentence number where the entity is located.

• pageName (str) – The name of the page where the entity is located.

• articles (list) – A list of articles associated with the page.

• pid (int) – The physical ID of the page.

Notes:

• Updates the info dictionary in the entity with the token text based on the tag.

• Appends metadata such as pid, pageNames, positions, pageNo, and sentenceNo to the entity.

• Sets the type of the entity to “PER” (person).

• Processes and adds article information to the entity if not already present.

• Logs a warning if an unknown tag is encountered.

src.postprocess.add_info_to_place_entity(entity: dict, tag: str, token: dict, pageNo: str, sentNo: str, pageName: str, articles: list, pid: int) → None

Adds information to a place entity based on the provided token dict and metadata about the page.

Parameters:

• entity (dict) – The place entity dictionary to update.

• tag (str) – The type of the place entity (e.g., “LOC”).

• token (dict) –

  A dictionary containing token information, including

  • ”token” (str): The token text.

  • ”coord” (tuple): The coordinates of the token.

• pageNo (str) – The page number where the entity is located.

• sentNo (str) – The sentence number where the entity is located.

• pageName (str) – The name of the page where the entity is located.

• articles (list) – A list of articles associated with the page.

• pid (int) – The physical ID of the page.

Notes:

• Updates the tokens list in the entity with the token text.

• Sets the type of the entity to the provided tag.

• Appends metadata such as pid, pageNames, positions, pageNo, and sentenceNo to the entity.

• Processes and adds article information to the entity if not already present.

src.postprocess.adjust_information(entitylist: list) → None

Adjusts page information in the given entity list. Only keeps first value in “pageNames”, “pageNo”, “sentenceNo”, and “pid”. Logs a warning if there are several values for these entries. If an entity has no “pid”, this entry is removed.

Parameters:

entitylist (list) – List of entity dictionaries, where each dict contains page information on where this entity appears.

src.postprocess.decide_articles(articles: list) → list

Given a list of lists, only keeps the entries that are exactly length 2. Or, if the input is only length one, keeps it unchanged.

Parameters:

articles (list) – List of articles

Returns:

Cleaned list of pages where we only keep the articles with exactly two pages.

Return type:

list

src.postprocess.execute_postprocessing(magazines, tasks: list, timed=True, places=False) → None

Postprocesses data based on the given configuration and tasks.

Parameters:

• magazines (iterator) – Generator object containing the magazines. The object is a dictionary where keys are tuples of (magazine shortname, year) and the values are file paths or lists of file paths.

• tasks (list) – List of tasks to be performed during postprocessing.

• timed (bool) – Boolean indicating whether to time the execution and log it, defaults to True.

• places (bool) – Boolean indicating whether to process place entities as well, defaults to False.

Returns:

a list of tuples. The first value of the tuple is another tuple, consisting of the year of the magazine and the shortname of the magazine. The second value of the tuple is a dictionary describing the data.

Return type:

list

Notes:

• Logs the start and end time of the postprocessing.

• If “CUSTOM_PATHS” is not in the settings, sets the input folder path to the output folder path with “tag” appended.

• Loads data iteratively based on the configuration.

• Executes postprocessing on the loaded data.

• Saves intermediate data if “agg” is not in the tasks list.

src.postprocess.get_article_info(article)

src.postprocess.get_data_paths_iterative()

Generates year-wise data paths for processing based on the settings.

Raises:

• Exception – If no valid data paths are found in the configuration.

• Exception – If an input path generated with the configuration is neither a valid directory nor a valid file. This means the data paths are valid, but nothing useful is in there.

Returns:

Yields: A dictionary where keys are tuples of (magazine shortname, year) and values are file paths or lists of file paths.

Return type:

Iterator[dict]

src.postprocess.get_found_names(items: tuple) → list

Extracts and processes entity information (person and place names) from tagged files.

Parameters:

items (tuple) –

• year (tuple): A tuple of journal shortname and year as strings (e.g., (“abc”, “2025”)).

• pages (str or list): Either a single string for the tagging output file or a list of paths to the tagging output files.

• places (bool): If places should be considered as well.

Returns:

The first entry is a list of all found entities (person and place names) with their associated metadata. The second is the year information (journal shortname and year). The third is a list of all the tagged files that belong to this year.

Return type:

tuple (list, tuple)

Note

• Person names and place names are written in the same file but are sorted before printing (person names first).

• When looking up the structure information for the pages, we use the information given by the raw data folder structure at the moment. This means that in rare cases two actually different volumes might have the same short-year combination, causing problems when linking pages to structure elements. This will change only once structure files become the initial pipeline input, which requires a larger rework — for now some information will simply be missing at the start.

• Handles cases where tagged files are split into multiple lines for efficiency.

• Adjusts entity information to remove duplicates and ensure consistency.

• Missing structure information may result in incomplete metadata for some entities.

src.postprocess.get_structure_info(year: tuple, custom_path=None) → dict

Retrieves structural information for a given year from an XML file.

Parameters:

• year (tuple) –

  • short (str): The shortname of the journal (e.g., “bse”).

  • year (str): The year of the journal (e.g., “2025”).

• custom_path (_type_, optional) – Path to a custom XML file for debugging, defaults to None

Returns:

A dictionary where keys are page filenames (e.g., “page1.txt”) and values are tuples

• pid (str): The physical ID of the page (e.g., “doc123:page1”).

• articles (list): A list of article IDs associated with the page.

• pagenum (str): The physical page number.

Return type:

dict

Notes:

• If custom_path is provided, it is used to parse the XML file.

• If custom_path is not provided, the function constructs the path to the XML file based on the year tuple.

• Handles cases where the XML file is missing or inaccessible by returning an empty dictionary.

• Skips journal-level connections and focuses on article-level connections.

• Extracts the filename for each page by replacing the .jpg extension with .txt.

src.postprocess.initialize_found_entry() → dict

Returns an empty person entity dictionary.

src.postprocess.initialize_found_place_entry() → dict

Returns an empty place entity dictionary.

src.postprocess.populate_year_dict(year_dict: dict, file_list: list) → None

Populates a dictionary with year-wise data paths for processing.

Parameters:

• year_dict (dict) – A dictionary to be populated. Keys are tuples of (magazine shortname, year), and values are file paths or lists of file paths.

• file_list (list) – A list of file paths to be processed. The files can be in .json or .jsonl format.

Notes:

• For .json files, the file path is directly added to the dictionary.

• For .jsonl files, all matching files are globbed and added as a list.

• Unsupported file types are ignored.

src.postprocess.process_page(page: str, sentences: list, entitylist: list, placeEntitylist: list, structure_info: dict, i: int) → None

Processes a single page of tagged sentences to extract entity information.

Parameters:

• page (str) – The name of the page being processed.

• sentences (list) – A list of sentences, where each sentence is a list of tokens. Each token is a dictionary containing information such as “tag”, “token”, and “coord”.

• entitylist (list) – A list to store extracted person entities.

• placeEntitylist (list) – A list to store extracted place entities.

• structure_info (dict) – A dictionary containing structural information for the page. Keys are page names, and values are tuples of (pid, articles, pagenum).

• i (int) – A fallback page number to use if no structural information is available.

Notes:

• Extracts person entities (tagged with “PER”) and place entities (tagged with other tags).

• Uses BIO tagging format to identify the beginning (“B-”) and continuation (“I-”) of entities.

• Updates the entitylist and placeEntitylist with extracted entities.

• Handles cases where structural information is missing by using the fallback page number.

• Logs warnings for unknown tags encountered during processing.

src.tag_flair module

Includes functions to tag files using flairNLP. Version 31.08.2020: Updated to use new tagging system which uses 2 models at once.

class src.tag_flair.CustomSentence(filename=None, text=None)

Bases: Sentence

class src.tag_flair.CustomToken(text, coords, orig)

Bases: Token

src.tag_flair.add_sentences(new_data: dict, collected_sentences: list) → None

Given the sentences tagged with both models, combines their tags and updates the new_data dictionary with the new sentences.

Parameters:

• new_data (dict) – A dictionary where the keys are the filenames and the values are the tagged sentences in said file.

• collected_sentences (list) – A list of sentences tagged by both models.

src.tag_flair.decide_tag_no_tag_lower_prio(labels: list) → Label

Combining the tags of the two tagging models. If there is disagreement between the two models, “O” always loses.

Parameters:

labels (list) – A list of tags for an entity with at most 2 entries. If there are two, the first entry corresponds to the bio label model and the second to the det label model. If there is only one entry, the model is detected based on the labeling scheme.

Raises:

Exception – If the labels list is empty, an exception is raised.

Returns:

The Label object is the new combined label for this entity, the dictionary

Return type:

Label

src.tag_flair.execute_tagging(preprocessed_data, tasks: list, gpu_num: int) → None

Tags the preprocessed data using the provided flair tagger and configuration.

Parameters:

• preprocessed_data – The data to be tagged, usually loaded or passed from preprocessing.

• tasks (list) – List of tasks to be performed.

• gpu_num (int) – GPU number to use. If it’s 0, CPU is used.

Raises:

Exception – If ‘prep’ is not included in the tasks list, an exception is raised indicating that ‘prep,tag’ must be called together.

src.tag_flair.package_generator_output_paths(generator, batch_size)

Packages the output from a generator into batches of a specified size.

Parameters:

• generator (iterable) – An iterable that yields tuples, where the first element is a year and the second element is a list of files.

• batch_size (int) – The number of items to include in each batch.

Returns:

Yields: A dictionary where the keys are years and the values are lists of files, with the number of items in the dictionary not exceeding the batch size.

Return type:

Iterator[dict]

Example:

>>> generator = [(2020, ['file1', 'file2']), (2021, ['file3', 'file4'])]
>>> batch_size = 1
>>> for batch in packageGeneratorOutput(generator, batch_size):
>>>     print(batch)
    # Output:
    # {2020: ['file1', 'file2']}
    # {2021: ['file3', 'file4']}

src.tag_flair.setup_flair_tagger(gpu_num: int) → MultitaskModel

Sets up and returns a Flair MultitaskModel for two NER models.

Parameters:

gpu_num (int) – The GPU number to use for the Flair models.

Returns:

An instance of Flair’s MultitaskModel loaded with the specified NER models.

Return type:

MultitaskModel

src.tag_flair.tag_year_data_and_save(collection: dict, tagger: MultitaskModel, outfile_path: str, sentence_batch_size: int) → None

Runs tagging on the collection and saves the result into the outfile_path.

Parameters:

• collection (dict) – A dictionary where the keys are the filenames and the values are the sentences in said file.

• tagger (MultitaskModel) – The MultitaskModel containing both tagging models (ner-det and ner-bio).

• outfile_path (str) – String of the outfile path where the tagged file will be saved.

• sentence_batch_size (int) – Number of sentences, after which we start writing the intermediate results into the outfile.

src.tag_flair.write_sentences_to_outfile(outfile, data: dict) → None

For each SENTENCE_BATCH_SIZE (set in the config file) batch of sentences, we write out the sentences into the outfile. This helps with our memory restrictions.

This has the side-effect that the data dictionary is cleared with each time we call this function.

Parameters:

• outfile (TextIOWrapper) – Text stream we can write our intermediate results into.

• data (dict) – Dictionary of filenames, tagged sentences.

Module contents