overthelex/crossjur-citation-coupling

Cross-Jurisdictional Legal Citation Coupling — Ukrainian data & tools

Reproducibility artifacts for the paper "Bibliographic Coupling as a Cross-Jurisdictional Evaluation Framework for Legal Case Retrieval: Civil-Law and Common-Law Citation Graphs Compared."

The paper compares a civil-law citation graph (Ukrainian, from the EDRSR state court-decision registry) with a common-law one (Canadian, from the COLIEE 2026 Task 1 corpus) under a single coupling / co-citation / temporal methodology. This repository holds the tools and the Ukrainian-side derived data so the Ukrainian results can be reproduced and the numbers cited in the paper can be traced to their source.

The Canadian (COLIEE Task 1) texts are not redistributed here — they are governed by the COLIEE data memorandum. The same tools reproduce the Canadian figures from the COLIEE corpus once its licence terms are met.

Update, 25 July 2026

Two changes, both prompted by a re-check of the coupling metrics after the ICTIR '26 framework paper appeared (doi:10.1145/3805713.3820412).

Correction. Earlier versions of the paper stated that Cite(d) was available over the whole Ukrainian pool while the COLIEE side had it only for the labelled query cases, and that the Canadian figures were therefore the only lower bound. That is true of the extraction but not of the outcome: of the 2,217 pool documents with a non-empty citation list, 2,000 are the query cases, and only 3 of the 3,650 pure distractors carry one. Both coupling layers rest on query-case citations, so both sets of figures are lower bounds. Section 3.4, the RQ3 discussion and the Limitations are corrected accordingly. The paper also now reports a corpus property it had omitted: Ukrainian queries carry a mean of 1.31 gold cases (84.5% have exactly one) against 4.12 for the Canadian queries.

New tools.

Tool	Purpose
bc_diagnostics.py	Popularity floor, Coverage decomposed by how often each gold case is cited, and growth of the coupling layer with citation knowledge
reachability_gain.py	Reachability Gain, a nugget/diversity formulation whose corner cases reproduce Extended Precision and Coverage exactly (--verify asserts it)
dump_rankings.py	Dump dense-encoder top-k lists, so rank-aware measures can be scored beyond the BM25 baseline

Outputs are in data/bc_diagnostics_{ua,ca}.json and data/rg_{ua,ca}.json.

Contents

tools/

Standard-library / sentence-transformers Python (and R for figures); no third-party services beyond a PostgreSQL source and, for retrieval, a GPU.

Tool	Purpose
build_canada_citation_graph.py	Build the case→case graph from a COLIEE-format zip; coupling, co-citation, citation-age + Mann-Kendall
build_ua_coupling_sample.py	UA article-level coupling / co-citation on a fixed-seed decision sample (runs on the EDRSR Postgres)
build_ua_temporal.py	Full UA case→case citation-age trend + Mann-Kendall over all resolved precedent edges
build_ua_case_retrieval_package.py	Assemble a COLIEE-format UA case-retrieval task (gold = resolved precedent edges)
probe_ua_case_retrieval.py	Read-only scoping probe used to size the UA task
retrieval_experiment.py	Dense first-stage retrieval (E5, BGE-M3) + exponential recency-weighting sweep; --years-json for non-English dates
figures/	prepare_fig_data.py (JSON→CSV), R + tikzDevice figure scripts, build_claims_source.py (flatten stats for fact-checking)

data/ (Ukrainian, derived from EDRSR)

File	What
ua_graph_stats.json	UA article-level coupling / co-citation sample statistics
ua_temporal_stats.json	UA full case→case citation-age-by-year series + Mann-Kendall (6.59M precedent edges)
ua_retrieval_results.json	UA case-retrieval macro P/R/F1 @k, baseline vs recency-weighting sweep (E5, BGE-M3)
ua_case_retrieval.zip	The UA case-retrieval benchmark in COLIEE Task 1 format: ua_cases/cases/<doc_id>.txt (2,000 queries + ~7,700-decision pool, texts truncated) + clean_ua_case_labels.json (query → gold precedent)
ua_years.json	{<doc_id>.txt: decision_year} for recency weighting

Reproduce the Ukrainian retrieval result

python tools/retrieval_experiment.py \
  --zip data/ua_case_retrieval.zip \
  --years-json data/ua_years.json \
  --models e5 bge-m3 --max-chars 4000 --num-gpus 1 \
  --out ua_retrieval_results.json

Headline: exponential recency weighting degrades case retrieval in both Ukraine (−37% to −41% at λ=0.2) and Canada (−64% to −67%), while it improves Ukrainian statute retrieval — so the effect tracks the target type (superseded statute vs. cumulative precedent), not the legal tradition. Coupling density, by contrast, is tradition-dependent (Ukrainian decisions couple ~18× more densely than Canadian cases).

Source & ethics

Ukrainian data derives from the public EDRSR (Єдиний державний реєстр судових рішень). Decision texts are truncated and included only as a research benchmark. See also overthelex/ukrainian-court-decisions and overthelex/ua-court-citation-graph.

Citation

Bibliographic Coupling as a Cross-Jurisdictional Evaluation Framework for Legal Case Retrieval. Working draft, 2026 -- UAlberta / COLIEE collaboration; authorship to be finalized.