Mailroom-Corpus-EDA: technical report

For the versioned corpus reference and source-license context, see Cite the dataset.

Data engineering · dataset build, audit & publishing

Summary

Mailroom-Corpus-EDA is the audit and publishing repository for the mailroom dataset, a corpus of 3,302 legal and insurance documents in five classes and 55 strata. The repository downloads the published dataset, checks its structure and provenance, measures its composition, draws 30 static and 18 interactive figures, audits correspondence intent coverage, and holds the helpers that build and publish new revisions to the Hugging Face Hub. This report describes the Hub tag v9.2. The current published revision is v9.3, released in November 2026: it adds corporate presence-layer labels over the same 3,302 documents and routes merger agreements to their own specialist. A v9.4 revision is staged and not yet published.

It is an evaluation and ingress-simulation corpus for the mailroom projects. It is not a training set, and the report says where that matters.

System and data

Sources

I excluded three candidate insurance samples because they carry a non-commercial licence.

Configurations

Pipeline

run_all.py runs all seven phases, or any subset with --phases. The summary JSON is written only by a full run, because a subset run once overwrote the full-corpus summary with partial numbers.

The src/mailroom_eda/ package is about 21,100 lines of Python at the 6 November 2026 snapshot (revision 32d34a2). Its modules cover identity, the evaluation contract, matter grouping, the integrity audit, ground-truth quality rules, the intent backfill, token budgets, export and the Hub client.

Method

Label separation

The blind configuration is the surface a model sees, so nothing derived from a label may appear on it. An early revision still carried two integers in the blind metadata, a contract clause count and a merger annotation count. Both are functions of the hidden labels and leak a weak signal: a non-zero count says the document has labelled clauses. The v9.1 revision moved them into the ground-truth configuration. The hardened release code never restages the blind configuration, so labels cannot ride along with it.

Identity and provenance

A document id is derived from the source corpus and source filename. It does not come from a row index, the content, the split or the configuration, so it survives rebuilds and reordering. Two rows may share content, since duplicates are a legitimate mailroom scenario, but each row stays a distinct incoming artefact. Content and normalised-text SHA-256 hashes support duplicate detection and rebuild verification.

The published annotation_method records six regimes for how a label was produced:

The six counts add up to 3,302, one regime per row.

Correspondence intent

The canonical vocabulary is eight classes: analysis, meeting_invite, notice, other, payment_demand, press_communication, request and update. other is the explicit fallback, never null. Rows reach a label by one of four disjoint paths, recorded in intent_source:

All 1,000 correspondence rows have an intent, and 25 are flagged for review. The model pass is checkpointed, so a rerun skips rows that already have a label.

Split rule and determinism

A document goes to the test set when the MD5 of its filename modulo 10 is zero. That gives 2,979 train and 323 test rows, stable across rebuilds. Random state is fixed at 42, rows are sorted before export, and a rebuild of the JSONL and parquet files is byte-identical. The interactive Plotly figures are the one exception, because each embeds a random identifier, so the committed files are treated as canonical.

Export hardening

Metadata keys are uniform and string-typed, and absence is an empty string. JSON Lines output escapes the Unicode line separators (U+2028, U+2029 and NEL), because some readers split records on them. A verification script compares a local export byte for byte against the Hub.

Results

Integrity

• The blind and ground-truth configurations agree on all 3,302 filenames.
• The split rule has zero mismatches.
• All 13,753 CUAD span annotations match the document text at their stated character offsets.
• The MAUD annotation count matches the count in the labels for all 152 merger agreements.

Composition

The largest class is 7.2 times the smallest, and the largest stratum is 557 times the smallest. One merger-agreement stratum, a mixed cash and stock election, has a single document. Ten strata have no test document and five have fewer than ten documents in total.

Text and token geometry

Token counts use a characters-divided-by-four heuristic.

Return to chart

Merger agreements are the long tail, with a mean near 22,000 tokens and a maximum near 63,000. Insurance claims are short, about 690 characters on average.

Revision history

None of v9.1, v9.2 or v9.3 added or removed a document, or changed an identifier or any document text. The changelog records zero drift. The tables and counts in this report and its appendix are for v9.2.

An audit that was wrong

After the v9.1 change moved the MAUD count to the ground-truth side, the integrity audit kept reading it from the blind metadata. It regenerated its report without error and printed a count-consistency result of 0 of 152. Nothing was wrong with the data. The audit was looking in the old place. The v9.2 revision reads the ground-truth count, restores the correct 152 of 152, and pins the case with a regression test.

Tests

I ran the test suite in a fresh environment, with the latest versions of the dependencies and not the pinned ones. That older run had 200 tests, of which 185 passed, 14 skipped and 1 failed. The current revision declares 594 def test_ functions in 26 files; that is a count, not a passing run. The 14 skipped tests need the full dataset snapshot, which run_all.py --phases P0 downloads. The 14 skipped tests need the full dataset snapshot, which run_all.py --phases P0 downloads. The failure is a contract test that reads a taxonomy file from a sibling checkout of another repository, which my environment did not have.

Limitations

• Evaluation corpus, not a training set. The 90/10 split is for reproducible sampling and is not stratified, so ten strata have no test documents.
• Imbalance. The largest stratum is 557 times the smallest, so any per-stratum score is noisy for the small ones.
• No clause annotations on 91 contracts. The SEC EDGAR exhibits were never annotated.
• Provenance mix. 637 intents come from a zero-shot model pass and 1,100 insurance labels are synthetic, so results on those rows describe the labelling method as much as the documents.
• Personal data. The Enron emails are real messages from named people.
• Token counts are a heuristic. Characters divided by four will differ from a real tokenizer.
• Continuous integration is a placeholder. The repository’s own workflow prints a message and does not run the tests, so a regression would not be caught automatically.

Reproduce

git clone https://github.com/Exios66/Mailroom-Corpus-EDA.git
cd Mailroom-Corpus-EDA
python -m venv .venv && .venv/bin/pip install -r requirements.txt pytest
.venv/bin/python run_all.py            # all seven phases
.venv/bin/python run_all.py --phases P3 P4   # figures only
.venv/bin/python -m pytest tests

Reading the public snapshot needs no token. Publishing a revision needs a Hub token.