utility package

Subpackages

Submodules

utility.compare module

Comparison functions for Person entities

utility.compare.compare_gnd_info(entity_pre: dict, entity_post: dict) → bool

Checks if the gnd ids are the same after some code changes occurred.

Parameters:

  • entity_pre (dict) – The entity before any changes were made to the code.

  • entity_post (dict) – The entity after code changes.

Raises:

Exception – Throws an exception if the gnds do not match.

Returns:

True if nothing changed, throws an exception in all other cases.

Return type:

bool

utility.compare.compare_linking(output_path_pre: str, output_path_post: str) → bool

Compares linking output of two jsons at the given paths.

Parameters:

  • output_path_pre (str) – Path to the linking output json before any changes were made.

  • output_path_post (str) – Path to the linking output json after changes were made.

Raises:

Exception – If any of the fields do not match, throws an exception indicating where the mismatch was. The order of the found entities matters.

Returns:

True if nothing changed, throws an exception in all other cases.

Return type:

bool

utility.compare.compare_linking_person(list_pre: list, list_post: list) → bool

Compares the list of entities before and after code changes.

Parameters:

  • list_pre (list) – List of entities before any code changes.

  • list_post (list) – List of entities after code changes.

Raises:

Exception – If any of the fields do not match, throws an exception indicating where the mismatch was. The order of the found entities matters.

Returns:

True if nothing changed, throws an exception in all other cases.

Return type:

bool

utility.compare.compare_linking_places(list_pre: list, list_post: list) → bool

Compares the places lists before and after code changes.

Parameters:

  • list_pre (list) – List of entities before any code changes.

  • list_post (list) – List of entities after code changes.

Raises:

Exception – If any of the fields do not match, throws an exception indicating where the mismatch was. The order of the found entities matters.

Returns:

True if nothing changed, throws an exception in all other cases.

Return type:

bool

utility.compare.compare_references(entity_pre: dict, entity_post: dict) → bool

Checks if the references are the same after some code changes occurred.

Parameters:

  • entity_pre (dict) – The entity before any changes were made to the code.

  • entity_post (dict) – The entity after code changes.

Raises:

Exception – If any of the fields do not match, throws an exception indicating where the mismatch was.

Returns:

True if nothing changed, throws an exception in all other cases.

Return type:

bool

utility.compare.compare_tagging(output_path_pre: str, output_path_post: str) → bool

Compares tagging output of two jsons at the given paths.

Parameters:

  • output_path_pre (str) – Path to the tagging output json before any changes were made.

  • output_path_post (str) – Path to the tagging output json after changes were made.

Raises:

Exception – If any of the fields do not match, throws an exception indicating where the mismatch was. The order of the found entities matters.

Returns:

True if nothing changed, throws an exception in all other cases.

Return type:

bool

utility.compare.main()

utility.evaluation_utils module

Evaluation functions for linking output.

class utility.evaluation_utils.Paths

Bases: object

Defines a “Paths” class to build the various paths to ground-truth, linking output, evaluation output and input files.

check_and_create(type_, key, ref_level_name)
get(type_, key, ref_level_name='')
get_jsonl(type_)
save_json(type_, key, doc, ref_level_name)
update(key, value)
class utility.evaluation_utils.Scores(counts_dict={'fn': 0, 'fp': 0, 'tn': 0, 'tp': 0})

Bases: object

Scores class for the recall, precision, f1 scores.

compute_scores()
get_score(round_to=3)
update_counter(counts_dict)
utility.evaluation_utils.clean_raw(raw: list, top_k, is_gt=False) → list

Given a list of entity dictionaries cleans up said dictionaries by unifying the references to make comparisons easier.

Parameters:

  • raw (list) – List of entity dictionaries.

  • top_k (_type_) – How many gnd candidates to take into account.

  • is_gt (bool, optional) – Whether the entity dictionaries are from a GT file. Defaults to False., defaults to False

Returns:

List of entity dictionaries, cleaned up to make comparisons easier, especially between the coordinates.

Return type:

list

utility.evaluation_utils.eval_entity(entity: dict) → dict

Get evaluation dictionary for given entity and its candidates.

Parameters:

entity (dict) – Person entity dictionary.

Returns:

Dictionary of “tp”, “fp”, “tn”, “fn” counts.

Return type:

dict

utility.evaluation_utils.eval_entity_inkb(entity: dict) → dict

Get evaluation dictionary for given entity and its candidates. Only evaluates disambiguation, so if a person does not have a GT gnd-id they are not considered for this score.

Parameters:

entity (dict) – Person entity dictionary.

Returns:

Dictionary of “tp”, “fp”, “tn”, “fn” counts.

Return type:

dict

utility.evaluation_utils.eval_references(entity: dict) → dict

Get evaluation dictionary for given entity and its candidates. Count not just the entities but the references, in order to give more weight to entities which occur often.

Parameters:

entity (dict) – Person entity dictionary.

Returns:

Dictionary of “tp”, “fp”, “tn”, “fn” counts.

Return type:

dict

utility.evaluation_utils.eval_references_inkb(entity: dict) → dict

Get evaluation dictionary for given entity and its candidates. Count not just the entities but the references, in order to give more weight to entities which occur often. Only evaluates disambiguation, so if a person does not have a GT gnd-id they are not considered for this score.

Parameters:

entity (dict) – Person entity dictionary.

Returns:

Dictionary of “tp”, “fp”, “tn”, “fn” counts.

Return type:

dict

utility.evaluation_utils.evaluate_person(gt: list, linked: list, ref_level=True, top_k=3, inkb_score=False) → dict

This function returns the true and false positives, as well as the true and false negatives of the linked file based on the ground-truth file.

The evaluation can be done on reference level (every time a person is mentioned) and on entity level (every person counts only once).

Special care has to be taken for entities with several ground-truth ids, which can happen when aggregation changed and now what used to be two person entities became one.

Parameters:

  • gt (list) – List of ground-truth entities.

  • linked (list) – List of linked entities.

  • ref_level (bool, optional) – If the evaluation should be done on reference-level. If False, it is done on entity level, defaults to True

  • top_k (int, optional) – How many of the gnd candidates to take into account, defaults to 3

  • inkb_score (bool, optional) – Whether to only evaluate disambiguation, so only count if a person does have a GND entry, defaults to False

Returns:

Dictionary describing the true and false positives, and true and false negatives of this particular magazine-year.

Return type:

dict

utility.evaluation_utils.get_main_name(per_dict: dict) → str

Given a person dictionary returns the persons name as a string.

Parameters:

per_dict (dict) – Person entity dictionary.

Returns:

Name of the person.

Return type:

str

Example:
>>> {"lastname":"Müller", "firstname": "Otto"} => "Otto Müller"
>>> {"lastname":"Müller", "abbr_firstname": "O."} => "O. Müller"
utility.evaluation_utils.label_entity(ent: dict, gt: list) → str

Given an entity and a list of ground-truth entities, checks if they refer to the same person and returns what the ground truth gndid would be.

Parameters:

  • ent (dict) – Person dictionary.

  • gt (list) – List of person dictionaries in the ground-truth files.

Returns:

Ground-truth gnd_id, or “” if none exists.

Return type:

str

utility.linking_utils module

Utility functions for finding candidates via ElasticSearch

utility.linking_utils.clean_namestring(name: str) → str
utility.linking_utils.convert_dates_wikidata(wikidata_date: str) → str

Throws away time part of wikidata date format.

Parameters:

wikidata_date (str) – Date in wikidata format.

Returns:

Date in YYYY-MM-DD format.

Return type:

str

Example :: “+1796-10-16T00:00:00Z” => “1796-10-16”

utility.linking_utils.convert_gnd_format_kibana(person_dict: dict) → dict

Convert the output dictionary we get from the gnd ES index into the dictionary we use for the rest of the pipeline.

Parameters:

person_dict (dict) – Dictionary containing various information on a person entity.

Returns:

Dictionary containing various information on a person entity in our own format.

Return type:

dict

utility.linking_utils.convert_wikidata_format_kibana(person_dict: dict) → dict

Convert the output dictionary we get from wikidata into the dictionary we use for the rest of the pipeline.

Parameters:

person_dict (dict) – Dictionary containing various information on a person entity.

Returns:

Dictionary containing various information on a person entity in our own format.

Return type:

dict

utility.linking_utils.prep_name_for_elasticsearch_query(name: str) → str

Specify allowed edit distance for each word of a name for the elasticsearch fuzzy search functionality.

This is based on their fuzziness:auto implementation, and depends on the length of the word. We do not allow more than 2 edits per word.

Parameters:

name (str) – Name to be searched.

Returns:

Name to be searched including allowed edit distances.

Return type:

str

Example:
>>> "D. Birchall" => "D* Birchall~2"
>>> "J.P. Wittbach => J*P* Wittbach~2"
utility.linking_utils.search_person_gnd(fnames: list, lastname: str, year: str, gnd_limit=15, fuzzy=True) → dict

We search for this firstnames lastname in our elasticsearch GND index. We return at most gnd_limit results.

Parameters:

  • fnames (list) – List of firstnames of the person to search

  • lastname (str) – Lastname of the person to search

  • year (str) – Year this magazine was published in

  • gnd_limit (int, optional) – Number of results, defaults to 15

  • fuzzy (bool, optional) – Whether to search for the names including some edits, defaults to True

Returns:

Dictionary of each viable candidate where the keys are the gnd ids.

Return type:

dict

utility.linking_utils.search_person_gnd_variantName(fullname: str, year: str, gnd_limit=15, fuzzy=True) → dict

We search for this fullname in our elasticsearch GND index. We return at most gnd_limit results.

Parameters:

  • fullname (str) – Full namestring of the person to search

  • year (str) – Year this magazine was published in

  • gnd_limit (int, optional) – Number of results, defaults to 15

  • fuzzy (bool, optional) – Whether to search for the names including some edits, defaults to True

Returns:

Dictionary of each viable candidate where the keys are the gnd ids.

Return type:

dict

utility.linking_utils.search_person_wikidata(search_term: str, year: str, wikidata_limit=5, fuzzy=True) → dict

We search for this firstnames lastname in our elasticsearch Wikidata index. We return at most wikidata_limit results.

Parameters:

  • search_term (str) – first- and lastname of the person to search.

  • year (str) – year this magazine was published in.

  • wikidata_limit (int, optional) – Number of results, defaults to 5

  • fuzzy (bool, optional) – Whether to search for the names including some edits, defaults to True

Returns:

Dictionary of each viable candidate where the keys are the gnd ids.

Return type:

dict

utility.settings module

class utility.settings.ElasticSettings(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: EnvPrefixTarget | None = None, _env_file: DotenvType | None = PosixPath('.'), _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: Mapping[str, str | list[str]] | None = None, _secrets_dir: PathType | None = None, _build_sources: tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, *, username: str = 'elastic', password: str = '', base_url: str = 'https://localhost:9200', index_name_gnd: str = 'gnd_lobid', index_name_wikidata: str = 'wikidata')

Bases: BaseSettings

base_url: str
index_name_gnd: str
index_name_wikidata: str
model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_show_env_vars': False, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': ('.env_template', '.env'), 'env_file_encoding': 'utf-8', 'env_ignore_empty': False, 'env_nested_delimiter': None, 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': 'ELASTIC_', 'env_prefix_target': 'variable', 'extra': 'ignore', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

password: str
username: str
class utility.settings.JsonConfigSettingsSource(settings_cls: type[BaseSettings], _init_state: InitState | None = None)

Bases: PydanticBaseSettingsSource

A custom settings source that loads variables from a JSON file.

get_field_value(field: Any, field_name: str) → Tuple[Any, str, bool]

Gets the value, the key for model creation, and a flag to determine whether value is complex.

This is an abstract method that should be overridden in every settings source classes.

Args:

field: The field. field_name: The field name.

Returns:

A tuple that contains the value, key and a flag to determine whether value is complex.

prepare_field_value(field_name: str, field: Any, value: Any, value_is_complex: bool) → Any

Prepares the value of a field.

Args:

field_name: The field name. field: The field. value: The value of the field that has to be prepared. value_is_complex: A flag to determine whether value is complex.

Returns:

The prepared value.

class utility.settings.Settings(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: EnvPrefixTarget | None = None, _env_file: DotenvType | None = PosixPath('.'), _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: Mapping[str, str | list[str]] | None = None, _secrets_dir: PathType | None = None, _build_sources: tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, *, JOB_ID: str = 'chnobli', CUSTOM_PATHS: list[str] | None = None, CUSTOM_TAG_PATH: str | None = None, EVAL_TOPK: int | None = None, INKB_SCORE: str | None = None, USE_RICH_LOGGING: bool = True, PATH_TO_INPUT_FOLDERS: str = './data/input/', PATH_TO_NER_MODEL_1: str = './models/ner-bio.pt', PATH_TO_NER_MODEL_2: str = './models/ner-det.pt', PATH_TO_OUTFILE_FOLDER: str = './data/output/', PATH_TO_ABBREVIATION_FILE: str = './src/preprocessing/abbrevs.txt', PATH_TO_GROUND_TRUTH: str = './data/ground_truth/ground_truth_linked/with_fuzzy_matching/', DATA2_MNT: str = './adl/', VD_API_TOKEN: str = './src/.env.api.token', VD_MAX_RETRIES: int = 100, VD_QUERY_CHUNK_LEN: int = 5000, VD_MAX_DIST: float = 0.6, VD_CONTEXT_WINDOW_LEN: int = 30, VD_TIMEOUT: int = 300, VD_TIMEOUT_RETRY: int = 600, SENTENCE_BATCH_SIZE: int = 128, GND_LIMIT: int = 15, WIKIDATA_LIMIT: int = 5, LINKED_PERSONS_LIMIT: int = 10, BATCH_SIZE: int = 4, ADD_FUZZY_SEARCH: str = 'True', PATH_TO_CA_CERT: str = '../secrets/certs/ca/ca.crt', CLIENT_ID: str | None = None, CLIENT_SECRET: str | None = None, OIDC_TOKEN_URL: str | None = None, EMBEDDINGS_ENDPOINT: str | None = None, MILVUS_HOST: str = 'localhost', MILVUS_PORT: str = '19530', es: ElasticSettings = ElasticSettings(username='elastic', password='', base_url='https://localhost:9200', index_name_gnd='gnd_lobid', index_name_wikidata='wikidata'))

Bases: BaseSettings

BATCH_SIZE: int
CLIENT_ID: str | None
CLIENT_SECRET: str | None
CUSTOM_PATHS: list[str] | None
CUSTOM_TAG_PATH: str | None
DATA2_MNT: str
EMBEDDINGS_ENDPOINT: str | None
EVAL_TOPK: int | None
GND_LIMIT: int
INKB_SCORE: str | None
JOB_ID: str
LINKED_PERSONS_LIMIT: int
MILVUS_HOST: str
MILVUS_PORT: str
OIDC_TOKEN_URL: str | None
PATH_TO_ABBREVIATION_FILE: str
PATH_TO_CA_CERT: str
PATH_TO_GROUND_TRUTH: str
PATH_TO_INPUT_FOLDERS: str
PATH_TO_NER_MODEL_1: str
PATH_TO_NER_MODEL_2: str
PATH_TO_OUTFILE_FOLDER: str
SENTENCE_BATCH_SIZE: int
USE_RICH_LOGGING: bool
VD_API_TOKEN: str
VD_CONTEXT_WINDOW_LEN: int
VD_MAX_DIST: float
VD_MAX_RETRIES: int
VD_QUERY_CHUNK_LEN: int
VD_TIMEOUT: int
VD_TIMEOUT_RETRY: int
WIKIDATA_LIMIT: int
es: ElasticSettings
model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_show_env_vars': False, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': ('.env_template', '.env'), 'env_file_encoding': 'utf-8', 'env_ignore_empty': False, 'env_nested_delimiter': None, 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': '', 'env_prefix_target': 'variable', 'extra': 'ignore', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_assignment': True, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod settings_customise_sources(settings_cls: Type[BaseSettings], init_settings: PydanticBaseSettingsSource, env_settings: PydanticBaseSettingsSource, dotenv_settings: PydanticBaseSettingsSource, file_secret_settings: PydanticBaseSettingsSource) → Tuple[PydanticBaseSettingsSource, ...]

Define the sources and their order for loading the settings values.

Args:

settings_cls: The Settings class. init_settings: The InitSettingsSource instance. env_settings: The EnvSettingsSource instance. dotenv_settings: The DotEnvSettingsSource instance. file_secret_settings: The SecretsSettingsSource instance.

Returns:

A tuple containing the sources and their order for loading the settings values.

utility.split_year module

Splits a year folder in an intelligent way and names the parts after the issues they’re representing.

NOTE: At the moment, only issues are processed, this means that content outside of issues, like table of contents or appendices, are not processed. Discuss if this is wished for behavior.

utility.split_year.check_for_missing_pages(pagenos: dict) → tuple

Find duplicates in the pagenos dictionary.

Parameters:

pagenos (dict) – Keys are issues, values are lists of pages.

Returns:

The first value is -1 if there were duplicates and 0 else. The second value is the number of pages in all the issues.

Return type:

tuple (int, int)

utility.split_year.compare_pagenames(pagenos: dict, year_pages: list, page_count: int, directory: str) → int

Checks if files are missing and which are missing.

Parameters:

  • pagenos (dict) – Keys are ssues, the values are lists of pages.

  • year_pages (list) – List of paths to pages.

  • page_count (int) – How many pages there should be.

  • directory (str) – Path to the directory where the pages should be.

Returns:

-1 if there were files missing, and we write the missing filepaths into a file, else 0.

Return type:

int

Note:

Ismail: Common missing files are the ones at the end of a file that are not part of any issue. I’ve decided that these files don’t provide any value, so it’s fine to ignore them. check again if the split went correctly

utility.split_year.cut_pagenumbers(pagenos: dict, max_len=500, max_len_warning=1000) → list

Cut the issues into chunks small enough for the processing pipeline.

Parameters:

  • pagenos (dict) – The keys are issues and the values are lists of pages.

  • max_len (int, optional) – Max number of pages in one chunk (regardless of whether it’s from several issues or not), defaults to 500

  • max_len_warning (int, optional) – Max number of pages per issue. If it’s more than that, we log a warning and split it within an issue, defaults to 1000

Returns:

List of Dictionaries of Lists of Pagenumbers

Return type:

list

utility.split_year.get_pagenumbers(xml) → dict

Given an XML tree find all the issues in the journal and their pages.

Parameters:

xml (etree) – XML-Tree with issue-information

Returns:

Dictionary with information about all issues and the filenumbers that refer to the respective pagenumbers

Return type:

dict

utility.split_year.split_directory(directory: str, custom_xml_path=None)

Splits the given directory into chunks of pages.

Parameters:

  • directory (str) – Path to the directory

  • custom_xml_path (str, optional) – If the xml path is not in the “usual” directory xml.cache.prod01 a custom path can be provided here, defaults to None

Returns:

Yields. Dictionary with chunknames as key and list of pages as pathfiles as values

Return type:

Iterator[dict]

utility.utils module

Utility functions

utility.utils.check_gpu(args: Namespace) → int

Checks if an Nvidia GPU is available and sets the GPU number accordingly.

Parameters:

args (argparse.Namespace) – An object containing the GPU argument. It should have an attribute ‘gpu’ which is a string representing the GPU number.

Returns:

The GPU number to be used. If no Nvidia GPU is detected, it returns 0 indicating that the code will run on CPU.

Return type:

int

utility.utils.erara_xml_to_word_coord(xml_path, schema='{http://www.loc.gov/standards/alto/ns-v3#}') → str

Transforms E-Rara xml output into pipeline-compliant input.

Parameters:
Returns:

The string of similar format to ABBYY FineReader output.

Return type:

str

utility.utils.offset_len_to_linking_input(mention_list: list[dict])

Transforms tagging output that contains the mention, offset, length and the document name into a tagging output that can be used by the CHNOBLi system.

Parameters:

mention_list (list[dict]) – List of mention dictionaries.

Returns:

A list of dictionaries of the format used by the CHNOBli system.

Return type:

list[dict]

utility.utils.parse_arguments() → Namespace

Parses command-line arguments for the script.

Returns:

An object containing the parsed command-line arguments.

Return type:

argparse.Namespace

Command-line Arguments:

  • tasks (str): Comma-separated list of tasks to perform. Default is “prep,tag,finish”.

  • gpu (str): GPU identifier to use. Default is “0”.

  • magazine_year_paths (str): Paths to magazine year data.

  • eval_level (str): Evaluation level. Default is “ref”.

utility.utils.positive_int(value: str) → int

Custom type function for argparse that ensures a positive integer.

utility.utils.save_data(data: dict, taskname: str)

Saves the data given in a “taskname” folder in the oufile folder specified in the configurations. If the “taskname” folder doesn’t exist, one is created.

Parameters:

  • data (dict) – The keys are magazine-year tuples, the values are given by the task.

  • taskname (str) – The task at hand, for example “link” or “tag”.

utility.utils.save_data_intermediate(year: list, files: dict, taskname: str)

Saves the data given in a “taskname” folder in the oufile folder specified in the configurations. If the “taskname” folder doesn’t exist, one is created. As opposed to “save_data”, this function combines all the years for a magazine instead of saving each year individually into a file.

Parameters:

  • year (list) – A list [mag, year, ..-] where the first entry is the magazine shortname and the following entry (or entries) is the year (or the years) we processed.

  • files (dict) – A dictionary containing the intermediate results of the given taskname.

  • taskname (str) – The task at hand, for example “link” or “tag”.

utility.utils.set_default(obj)

Helper function to translate all sets in the aggregated dictionaries into lists when dumping them to files.

utility.utils.str2bool(value: str) → bool

Casts a string to a boolean value

Parameters:

value (str) – String input to be cast.

Raises:

argparse.ArgumentTypeError – If the string value cannot be cast to a boolean, this error is raised.

Returns:

Boolean value of the given string.

Return type:

bool

utility.utils.transkribus_xml_to_approx_word_coord(xml_path, schema='{http://schema.primaresearch.org/PAGE/gts/pagecontent/2013-07-15}') → str

Transforms Transkribus xml output which only has baseline level coordinates into word-level coordinates by naively splitting the length by number of words.

Parameters:
Returns:

The string of similar format to ABBYY FineReader output.

Return type:

str

utility.utils.txt_file_to_word_coord(txt_path: str) → str

Transforms .txt OCR output (like for instance from Tesseract) into pipeline-compliant input. The “coordinates” are unique 4-tuples so they can be used for the pipeline but are not coordinates on the page.

Parameters:

txt_path (str) – Path to the txt file.

Returns:

The string of similar format to ABBYY FineReader output.

Return type:

str

Module contents