Using Your Own Data

This page describes how to run CHNOBLi on your own documents, whether they come from OCR software or from an existing entity extraction pipeline.

Input Format: OCR Data

The tagging component expects word coordinates, as produced by ABBYY FineReader. If your OCR comes from another source, CHNOBLi provides transformation utilities for several common formats.

Transkribus

from utility.utils import transkribus_xml_to_approx_word_coord

E-Rara

from utility.utils import erara_xml_to_word_coord

Tesseract or Plain Text

from utility.utils import txt_file_to_word_coord

Note

Contributing: Have a transformation function for another format? Please submit a pull request!

Once your data has been transformed, you can run the pipeline just as you did with the example data in Quick Start.

Custom Tagging Output

If you already have entity extractions (e.g., from SpaCy), you can transform them into CHNOBLi’s expected format instead of running the tagging step yourself.

Transformation

Input example:

{
   "mention": "Kamal Kharrazi",
   "offset": 237,
   "length": 14,
   "docName": "APW19981109_0464.htm"
}

Transform using:

from utility.utils import offset_len_to_linking_input

This produces output like:

{
   "info": {
      "lastnames": ["Kharrazi"],
      "firstnames": ["Kamal"],
      "abbr_firstnames": [],
      "address": [],
      "titles": [],
      "occupations": [],
      "others": [],
      "type": "PER",
      "id": 0,
      "gt_wikipedia": "Kamal_Kharazi",
      "gt_wikidata": "Q435799",
      "gt_gnd": "1222390949"
   },
   "pageNo": 0,
   "pageNames": "APW19981109_0464.htm",
   "pid": "APW19981109_0464.htm",
   "sentenceNo": 0,
   "positions": "237:14",
   "articles": "",
   "context": "al bodies about the U.S.-funded Radio Free Europe, the Iran Daily reported Monday. It quoted Foreign Minister Kamal Kharrazi as saying the radio \"was set up to interfere in Iran's internal affairs.\" It did not say when the complaints wil"
}

Linking

1. Configure your data path. Edit configs/configurations_customtag.json and set CUSTOM_TAGGING_OUTPUT to your data path.

2. Run aggregation (with linking):

   python main.py --tasks finish --config_file configs/configurations_customtag.json
   

   Or skip aggregation and only link:

   python main.py --tasks link --config_file configs/configurations_customtag.json
   

Note

On context: The pipeline reads context from ABBYY FineReader format, which is why for custom data you must include the "context" key with the context string explicitly. If your data doesn’t include context, simply omit the "context" key and disambiguation via the vector database will be skipped.

Note

On dates: The publication year is used for sanity checks (e.g., not considering people born after that year). For custom data, this defaults to year 3000 — adjust as needed in main.py.