ARC — Archival Research Corpus (working title, draft 0.1)

A lightweight, IIIF-compatible standard that lets researchers bundle material from different archives into one corpus — context-preserving, citable, shareable — and that is at the same time openable in a viewer and loadable in a notebook (images, texts, and data).

⚠️ Feasibility study / pilot. ARC is an exploratory proof of concept, not a ratified standard. The model, vocabularies, and reference code here are provisional and subject to change.

Documentation

Phase	File
Landscape analysis	docs/01-landscape-analysis.md
Concept	docs/02-concept.md
Technical design	docs/03-technical-design.md

Prototype (0.2)

arc/__init__.py           thin Python reference client
arc/iiif.py               projector ARC → IIIF Presentation 3.0 Collection
arc/dts.py                export adapter ARC → DTS 1.0 (text side)
arc/linked_art.py         crosswalk ARC → Linked Art (CIDOC-CRM profile)
arc/ric.py                crosswalk ARC → RiC-O 1.1 (archive-native ontology)
arc/ead.py                converter EAD finding aid → ARC corpus
arc/mets.py               converter METS package → ARC corpus (reads MODS + Dublin Core)
arc/dc.py                 converter OAI-PMH / Dublin Core → ARC corpus (breadth path)
schema/arc.schema.json    JSON Schema (Draft 2020-12)
context/v0.1.json         JSON-LD @context (draft)
examples/korrespondenz-x/ runnable demo corpus (2 archives, local contents)
  corpus.json             the ARC corpus
  texts/*.txt             full texts (text instantiations)
  data/briefregesten.csv  dataset (dataset instantiation)
  iiif-collection.json    generated IIIF Collection (projector output)
  dts/                    generated DTS 1.0 documents (Entry/Collection/Navigation/Document)
examples/uv-demo/         corpus with real IIIF manifests (for the Universal Viewer)
examples/ead-demo/        EAD 2002 finding aid + the ARC corpus generated from it
examples/mets-demo/       METS package (MODS + DC) + the ARC corpus generated from it
examples/oai-demo/        OAI-PMH ListRecords (oai_dc) + the ARC corpus generated from it
notebooks/demo.ipynb      usage example: corpus → notebook analysis
viewer/index.html         custom browser viewer (context · data · IIIF · DTS)
viewer/uv/                Universal Viewer 4.4.2 (vendored) + serve.sh
viewer/collector/         bookmarklet + demo pages: assemble a corpus across archives
tests/smoke.py            schema validation + client end-to-end test
tests/iiif_test.py        IIIF projector: structure checks + Collection generation
tests/dts_test.py         DTS adapter: spec conformance + DTS document generation
tests/crosswalk_test.py   ontology crosswalks: Linked Art + RiC-O
tests/ead_test.py         EAD converter: convert + validate + EAD→ARC→IIIF pipeline
tests/mets_test.py        METS converter: convert + validate + METS→ARC→IIIF pipeline
tests/dc_test.py          OAI-DC converter: convert + validate + DC→ARC→IIIF pipeline

Ontology crosswalks (CIDOC vs. RiC — both)

ARC deliberately stays ontology-neutral; the heavy ontology is a projection — just like IIIF/DTS. This means the same corpus can be carried over into both worlds:

.venv/bin/python -m arc.linked_art examples/korrespondenz-x/corpus.json out.json  # CIDOC/Linked Art
.venv/bin/python -m arc.ric        examples/korrespondenz-x/corpus.json out.json  # RiC-O 1.1

Both carry all 6 records (including the non-digitized one and the dataset); the IIIF projection carries the four that have a manifest — each projection expresses what its target world supports. Linked Art gives reach (the CIDOC world), RiC-O archival depth. Generated examples: examples/korrespondenz-x/linked-art.json and …/ric-o.json.

EAD → ARC converter (the adoption path)

Turns an existing EAD finding aid (EAD 2002 or EAD3) into an ARC corpus — so an archive that already has a finding aid gets an ARC corpus for free. The component hierarchy becomes partOf chains, <dao> links become IIIF instantiations, and — importantly — undigitized components become description-only nodes automatically, first-class records in the corpus.

.venv/bin/python -m arc.ead examples/ead-demo/finding-aid.xml out.json \
    --ead-url=https://.../finding-aid.xml

The demo finding aid yields 7 records (1 fonds, 2 series, 3 files, 1 item; 3 digitized, 4 description-only) and feeds straight into the IIIF/DTS/crosswalk projectors — a full EAD→ARC→IIIF pipeline. See examples/ead-demo/.

METS → ARC converter (structure + files — and MODS & DC for free)

arc/mets.py turns a METS package into an ARC corpus. METS carries the digital files (fileSec) and the object structure (structMap): described divs (DMDID) become Records, purely structural divs (pages) roll their files up as instantiations of the nearest described ancestor, and the descriptive dmdSec is read whether it holds MODS or Dublin Core — so both formats come along for free. <fptr> files become typed instantiations (IIIF, image, text, pdf, audio…); USE="THUMBS" is skipped.

.venv/bin/python -m arc.mets examples/mets-demo/album.mets.xml out.json \
    --mets-url=https://.../album.mets.xml

EAD and METS complement each other: EAD gives the archival hierarchy, METS the digitized object (files + structure). See examples/mets-demo/.

OAI-PMH / Dublin Core → ARC (the breadth path)

Every OAI-PMH endpoint must expose oai_dc (simple Dublin Core), so arc/dc.py can ingest a whole ListRecords response in one pass — broad coverage across a repository. DC is item-level and compact, so records carry the essentials (title→label, description→summary, date, creator→agents, identifier→ sourceReference or instantiation, setSpec→a shallow partOf). Deleted records are skipped; records without a media URL become description-only nodes.

.venv/bin/python -m arc.dc examples/oai-demo/listrecords.xml out.json \
    --label="Archive A — OAI-DC harvest"

Breadth (DC/OAI) vs. depth (EAD/METS): the same thin ARC target, different on-ramps. See examples/oai-demo/.

View in the browser

viewer/serve.sh          # local server at http://127.0.0.1:8000

• Custom viewer: http://127.0.0.1:8000/viewer/index.html — the corpus with its three faces.
• Universal Viewer (real images): http://127.0.0.1:8000/viewer/uv/arc.html
• Mirador + ARC panel ("IIIF shows 4, ARC knows 6"): http://127.0.0.1:8000/viewer/mirador/arc.html
• Collector (bookmarklet): http://127.0.0.1:8000/viewer/collector/index.html — add archive objects to your own corpus, then export corpus.json

Details in viewer/README.md.

Set up & run

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt          # pandas, jsonschema

# End-to-end test (validates the schema + runs the client through)
.venv/bin/python tests/smoke.py

# Notebook (register the kernel once, then run or open in Jupyter)
.venv/bin/pip install ipykernel nbconvert nbformat
.venv/bin/python -m ipykernel install --prefix "$PWD/.venv" --name python3
.venv/bin/jupyter nbconvert --to notebook --execute --inplace notebooks/demo.ipynb
# or interactively:  .venv/bin/jupyter lab notebooks/demo.ipynb

What the notebook shows

A letter corpus (1897) merged from two archives, immediately processed further:

1. Overview — corpus.to_dataframe() (level, context path, instantiations)
2. Archival bond — the ancestral chain travels along with every item
3. Full-text analysis — concordance/KWIC across archive boundaries
4. Dataset — rec.table() → pandas.DataFrame including aggregation
5. Relationships — cito:repliesTo chain (horizontal bond)
6. Annotations — W3C TextQuoteSelector, the same vocabulary as IIIF image regions
7. Filter — where(...) returns a sub-corpus

IIIF Collection projector

Demonstrates the compatibility promise: an ARC corpus is projected onto a valid IIIF Presentation 3.0 Collection (grouped hierarchically by fonds), so that any IIIF viewer can open it. The projector reports what the IIIF layer carries and what stays with ARC (relations, text selectors, the data layer), along with which records have a manifest to project.

# generates/updates examples/korrespondenz-x/iiif-collection.json + report
.venv/bin/python tests/iiif_test.py
# or as CLI:
.venv/bin/python -m arc.iiif examples/korrespondenz-x/corpus.json out.json

from arc import ArchiveCorpus
from arc.iiif import to_iiif_collection, format_report
corpus = ArchiveCorpus.load("examples/korrespondenz-x/corpus.json")
collection, report = to_iiif_collection(corpus)   # (dict, report)
print(format_report(report))

DTS export adapter (text side)

The counterpart to the IIIF projector: it exports the text instantiations as DTS 1.0 responses (Entry / Collection / Navigation / Document), faithful to the specification (@context https://dtsapi.org/context/v1.0.json). Over the unstructured letter texts the adapter lays a simple paragraph citation tree (citeType: "paragraph"), so that Navigation and Document work. DTS carries the text side; images, datasets, relations, and provenance stay with the IIIF projection or with ARC (the adapter reports the split).

# generates examples/korrespondenz-x/dts/ (entry, collection, navigation, document)
.venv/bin/python tests/dts_test.py
# or as CLI:
.venv/bin/python -m arc.dts examples/korrespondenz-x/corpus.json out-dts/

from arc import ArchiveCorpus
from arc.dts import to_dts_collection, to_dts_navigation, dts_document
corpus = ArchiveCorpus.load("examples/korrespondenz-x/corpus.json")
collection, report = to_dts_collection(corpus)            # DTS Collection
rec = corpus.get("https://archiv-a.example/item/1897-03-04")
nav = to_dts_navigation(rec)                              # CitableUnits (paragraphs)
para = dts_document(rec, ref=1)                           # one paragraph as text
tei  = dts_document(rec, ref=1, media_type="application/tei+xml")  # in the dts:wrapper

Client quick reference

from arc import ArchiveCorpus
corpus = ArchiveCorpus.load("examples/korrespondenz-x/corpus.json")

corpus.to_dataframe()                 # an overview, one row per record
for rec in corpus:
    rec.label, rec.level, rec.date, rec.context   # metadata + bond
    rec.text()                        # full text (text instantiation)
    rec.table()                       # DataFrame (dataset instantiation)
    rec.related("cito:repliesTo")     # horizontal relations
corpus.where(level="item", type="text")
corpus.annotations(target=rec)

Design principles (short version)

1. Link instead of overload · 2. Level ⟂ medium · 3. Record ↔ instantiation ·
2. thin JSON-LD profile over IIIF (+ a small RiC/CiTO subset) ·
3. Change Discovery as federation · 6. Machine actionability (viewer and dataset).

Compatibility promise: IIIF tools see a valid corpus; ARC tools additionally see context and data — backward-compatible, no lock-in.

Reviewing & license

This is a pilot circulated for review — see CONTRIBUTING.md for what we'd most like feedback on.

• Code (arc/, tests/, notebooks/, viewer/) — Apache-2.0, see LICENSE.
• Documentation & specification (docs/, schema/, context/, this README) — CC BY 4.0, see LICENSE-docs.md; this matches the licensing of the IIIF and DTS specifications.
• Vendored viewers (Universal Viewer · MIT, Mirador · Apache-2.0): see NOTICE.