How to Evaluate

To evaluate the output of this system, you need:

1. Ground-Truth Data

The ground-truth data, consisting of select years and pages of the magazines [cmt, edu, fsi, gfr, hvg, obl, rep, tjb, woh, zut] was created “with fuzzy matching”, i.e., minor misspellings of the entity name are tolerated when looking for the corresponding GND entry.

Your configuration file then specifies where the ground-truth data for the fuzzy level is located:

nla/configs/eval_config.json

2. Evaluation Level

Evaluation can be done at an entity or reference level. To select which level of evaluation you would like to see, set the command-line argument –eval_level ref or –eval_level ent.

3. Run Evaluation

You can simply run the script:

sh scripts/eval.sh

The magazine_year_paths argument describes the path to the directory where the data to be evaluated is located. Don’t worry if that directory contains even more linked files, only the ones for which a corresponding ground-truth file exists are compared.

Note, however, that for each ground-truth file, there must exist a corresponding linked file. If you would like to exclude some ground-truth files, you must remove them from the ground-truth data directory. This is to ensure that your model is tested against a variety of magazines and can be fairly compared to previous runs.

4. Analyze Evaluation

In the PATH_TO_OUTFILE_FOLDER directory specified in your eval_config.json, you can find the evaluations for each magazine in the corresponding file, for each year in a separate file under the magazine directory, and aggregated for all magazines and years under, for example, eval_ent_with_fuzzy.json.