Setup¶

This page describes how to install and configure CHNOBLi, a named entity linking pipeline for retro-digitized documents.

Quick Setup (Recommended)¶

The easiest way to set up the project is using the interactive setup wizard:

git clone git@github.com:eth-library/CHNOBLi.git
cd CHNOBLi
make setup

The wizard will guide you through:

Creating your .env configuration file.
Choosing between:
- Minimal Setup (using remote APIs), or
- Full Local Setup (cloning and setting up local databases). Once the databases are running, you can import the required data using:
```
make import-data
```

Manual Setup¶

If you prefer to set everything up manually, follow the steps below.

Step 1: Clone the Repository¶

git clone git@github.com:eth-library/CHNOBLi.git
cd CHNOBLi

Step 2: Create an Environment¶

Option A: Using Conda

conda create -n env_chnobli python=3.12 ipython
conda activate env_chnobli

Option B: Using venv

python3.12 -m venv .env_chnobli   # Windows: py -3.12 -m venv .env_chnobli
source .env_chnobli/bin/activate  # Windows: .env_chnobli\\Scripts\\activate

Step 3: Install Dependencies¶

pip install -r requirements.txt

Step 4: Download Models¶

Download the tagging models from the ETH Research Collection (DOI: 10.3929/ethz-c-000799811) and save them to the models/ directory.

Alternative to Steps 2-4: Docker¶

Instead of setting up the environment yourself, you can run:

docker compose --file docker-compose-dev.yml up

This automatically sets up the environment for you, although you still need to set up your own vector database and ElasticSearch index. To run linking, call:

docker exec -it linking sh scripts/link_example.sh

Step 5: Configure ElasticSearch¶

A public API endpoint is coming soon. To set up your own:

Follow the setup guide in CHNOBLi-elasticsearch.
Update CHNOBLi/.env_template with your endpoints and index names.
Rename the file to .env.
Copy the certificate hierarchy secrets/certs/ca/ca.crt from the CHNOBLi-elasticsearch directory into this repository.

Step 6: Configure Milvus¶

A public API endpoint is coming soon. To set up your own:

Set up Milvus following the setup guide in CHNOBLi-vectordb.
Update CHNOBLi/.env_template with your host and port.
Rename the file to .env.

Managing the Project with the Makefile¶

The project includes a Makefile to simplify common tasks and ensure consistent environments (especially regarding file permissions).

Setup & Data¶

make setup: Run the interactive configuration wizard.
make import-data: Interactive tool to download and import Wikidata, GND, and Milvus data.

Container Management¶

make build: Build the Docker images from source.
make up: Start the linking pipeline in the background.
make down: Stop and remove all containers.
make logs: Tail the logs from all running services.

Interactive Access¶

make shell: Drop into a bash shell inside the linking container.
make shell-root: Same as above, but with root privileges.

Next Steps¶

Once setup is complete, see Quick Start to try the pipeline on the provided example data.