Experiment Log

This page is generated from reports/experiment_log.jsonl on every render — the same renderer as reports/experiment_log.md (src/experiment_log.py::render_full_log), with per-document dumps omitted. Each run deep-links to its interactive detail page in the explorer.

Index

#	Experiment	Task	Model	Prompt(s)	Headline score	Rows	Total tokens
1	qwen_qwen3.7-flash_sorter_docclass_v6_docclass_merged_ab	docclass_classification	qwen/qwen3.7-flash	sorter_docclass_v6	exact_match 0.6333	—	621037
2	qwen_qwen3.7-flash_sorter_docclass_v7_docclass_merged_ab	docclass_classification	qwen/qwen3.7-flash	sorter_docclass_v7	exact_match 0.6667	—	652282
3	qwen_qwen3.7-flash_sorter_docclass_v6_docclass_ab120_s42	docclass_classification	qwen/qwen3.7-flash	sorter_docclass_v6	exact_match 0.5833	—	1873843
4	qwen_qwen3.7-flash_sorter_docclass_v7_docclass_ab120_s42	docclass_classification	qwen/qwen3.7-flash	sorter_docclass_v7	exact_match 0.6833	—	1999237
5	qwen_qwen3.7-flash_contracts_specialist_docclass_v0_docclass_ab120_s42	docclass_specialist_extraction	qwen/qwen3.7-flash	contracts_specialist_docclass_v0	extraction 0.6884	—	346327
6	qwen_qwen3.7-flash_contracts_specialist_docclass_v1_docclass_ab120_s42	docclass_specialist_extraction	qwen/qwen3.7-flash	contracts_specialist_docclass_v1	extraction 0.8444	—	655412
7	qwen_qwen3.7-flash_insurance_claims_specialist_docclass_v0_docclass_ab120_s42	docclass_specialist_extraction	qwen/qwen3.7-flash	insurance_claims_specialist_docclass_v0	extraction 0.6957	—	44376
8	qwen_qwen3.7-flash_insurance_claims_specialist_docclass_v1_docclass_ab120_s42	docclass_specialist_extraction	qwen/qwen3.7-flash	insurance_claims_specialist_docclass_v1	extraction 0.6904	—	48609

qwen_qwen3.7-flash_sorter_docclass_v6_docclass_merged_ab (docclass_classification)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T17:58:16.272567+00:00
Model	qwen/qwen3.7-flash
Prompt version	sorter: sorter_docclass_v6
Git commit	`64420c6` (dirty tree)

Data source

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
ground_truth	doc_type + doc_subclass
ground_truth_mode	maud_consideration_gt / s1_record_type / cuad_subtype
dataset_fingerprint	fa16e25a91631122b3e1d2111dadeca05194bcf162dd22474a2890fc83fcb7cb
n_samples	30
sample_requested	0
stratified	30
seed	42

Parameters

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
sample	0
stratified	30
seed	42
max_input_chars	100000
max_tokens	4096
reasoning_effort	medium
temperature	0.1
input_mode	text
vision_pages	all
max_concurrency	10
rate_limit_retries	0
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_sorter_docclass_v6_docclass_merged_ab · trace_name: docclass_classification · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
doc_type_accuracy	0.9667
subclass_accuracy	0.6667
subclass_accuracy_equiv	0.6333
exact_match	0.6333
confidence	0.951
n_rows	30
n_errors	0
equiv_recovered	—

Scores — doc_type_accuracy_ci

Field	Score
half	0.05
hi	1
lo	0.9
method	percentile-bootstrap
n	30
n_boot	2000
seed	42

Scores — subclass_accuracy_ci

Field	Score
half	0.1667
hi	0.8333
lo	0.5
method	percentile-bootstrap
n	30
n_boot	2000
seed	42

Scores — exact_match_ci

Field	Score
half	0.1667
hi	0.8
lo	0.4667
method	percentile-bootstrap
n	30
n_boot	2000
seed	42

Scores — per_class_accuracy

Field	Score
contract	1
corporate_record	0.9
merger_agreement	1

Scores — per_subclass_accuracy

Field	Score
Development	1
Franchise	1
Hosting	1
IP	1
Manufacturing	1
Outsourcing	1
Supply	1
Transportation	1
all_cash	1
all_stock	0.6667
articles_of_incorporation	0.2857
mixed_cash_stock	0.3333
other	0.0
powers_of_attorney	0.0
rights_instrument	1

Scores — per_subclass_support

Field	Score
Development	2
Franchise	1
Hosting	1
IP	1
Manufacturing	2
Outsourcing	1
Supply	1
Transportation	1
all_cash	3
all_stock	3
articles_of_incorporation	7
mixed_cash_stock	3
other	1
powers_of_attorney	1
rights_instrument	2

Scores — subclass_confusion

Field	Score
Development	development: 2
Franchise	franchise: 1
Hosting	hosting: 1
IP	ip: 1
Manufacturing	manufacturing: 2
Outsourcing	outsourcing: 1
Supply	supply: 1
Transportation	transportation: 1
all_cash	all_cash: 3
all_stock	all_stock: 2 · all_cash: 1
articles_of_incorporation	articles_of_incorporation: 2 · other: 2 · bylaws: 3
mixed_cash_stock	all_stock: 2 · mixed_cash_stock: 1
other	all_stock: 1
powers_of_attorney	bylaws: 1
rights_instrument	rights_instrument: 1 · other: 1

Scores — input_mode_counts

Field	Score
text	30

Scores — sorter

Field	Score
failure_insights	mode_counts: {‘subclass_miss’: 10, ‘doc_type_miss’: 1} · n_failed: 11 · failures: [{‘filename’: ‘0001547903-13-000020_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate for Class A Common Stock of NMI Holdings, Inc., presented as Exhibit 4.1. Per rule 3, the SEC exhibit wrapper is disregarded in favor of the substantive form, which is a corporate record evidencing share ownership. It does not match any of the specifically enumerated record types in rule 33, so doc_subclass is set to ‘other’. Note: contract_subtype must be null when doc_type is not contract; corrected to null.”}, {‘filename’: ‘0001078782-12-001347_s1_ex3z2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SIMPLE PRODUCTS CORPORATION’ and contains standard internal governance provisions organized into Articles covering offices, stockholder/director meetings, officers, stock certificates, and indemnification. Per Rule 32, corporate records filed as SEC exhibits (e.g., EX-3.2) retain their substantive classification as corporate records rather than compliance filings. The title directly matches the ‘bylaws’ subclass per Rule 33.”}, {‘filename’: ‘0001615774-14-000417_s100510_ex4-6.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘predicted’: {‘doc_type’: ‘contract’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘doc_type_miss’, ‘reasoning’: “The document is a formal ‘WARRANT AGREEMENT’ executed between the issuing company and a warrant agent, detailing the issuance, exercise, redemption, adjustment, and transfer mechanics of the warrants. Its substantive form is a bilateral contract governing financial instrument terms. It does not fit any specific commercial family (e.g., license, distribution, service, or agency) and is not a corporate governance record or regulatory filing, thus classifying it as ‘other’. The SEC exhibit wrapper does not change its substantive contractual nature per rule 3.”}, {‘filename’: ‘0001144204-19-043489_tv528080_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is labeled EXHIBIT 3.2 and contains the Organizational Minutes of the Initial Board of Directors along with the full text of the BY-LAWS OF INCUMAKER, INC. Per Rule 32, corporate governance records like bylaws attached to SEC filings as exhibits retain their substantive classification as corporate_record rather than compliance_filing. Per Rule 33, the doc_subclass is correctly identified as ‘bylaws’ based on the document’s own title. No competing signals are present.”}, {‘filename’: ‘0001469709-14-000003_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘EXHIBIT 3.2 BYLAWS’ and contains standard internal governance provisions for a Nevada corporation, including sections on shareholder meetings, board of directors, officers, and amendment procedures. Per rule 32, exhibits filed as corporate governance records are classified as corporate_record, and per rule 33, the subclass is bylaws.”}, {‘filename’: ‘0001469709-14-000529_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SWEETS & TREATS, INC.’ and contains standard corporate governance provisions regarding stockholders, directors, officers, meetings, voting, and amendment procedures. Per Rule 32, corporate records such as bylaws filed as SEC exhibits (here, Exhibit 3.2) retain their substantive classification as corporate_record rather than compliance_filing. The content and exhibit wrapper confirm this classification.”}, {‘filename’: ‘0001547903-13-000006_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate for Class A Common Stock of NMI Holdings, Inc. It certifies share ownership, outlines transfer and assignment procedures, and features corporate seal and officer signature blocks. Its substantive form is a corporate securities/governance instrument rather than a commercial contract or regulatory filing. Since it does not match the specific record-type subcategories listed (e.g., bylaws, articles of incorporation), it is classified as corporate_record with doc_subclass ‘other’.”}, {‘filename’: ‘contract_105_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document title is explicitly ‘AGREEMENT AND PLAN OF MERGER’. It contains classic M&A deal structure provisions, including parties labeled as Parent, US Holdco, Merger Sub, and Company; dedicated articles for ‘THE MERGER’, ‘EFFECT OF THE MERGER ON CAPITAL STOCK’, representations and warranties, covenants, conditions precedent, and termination rights. The consideration mechanics reference ‘Parent Share Issuance’ and a ‘Parent Equity Conversion Ratio’ based on a ‘Cash Equivalent Consideration’ (standard for cross-border stock conversions), confirming an all-stock transaction. Under Rule 31, this is classified as a merger_agreement.”}, {‘filename’: ‘contract_145_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including ‘THE MERGER’, ‘Representations and Warranties’, ‘Conditions to the Merger’, ‘Termination’, and an ‘Exchange Ratio’ converting East Common Stock into Central Common Stock. Per Rule 31, this title and structure mandate classification as merger_agreement. The consideration section (Section 1.6) specifies that shares will be converted solely into shares of Central Common Stock (with cash provided only for fractional share interests), which aligns with the all_stock subclass.”}, {‘filename’: ‘contract_5_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A machinery: parties structured as Parent, Merger Subs, and Company; articles covering The Mergers, Reps & Warranties, Conditions to Closing, No-Solicitation covenants, and Termination Payments. The consideration structure centers on an ‘Exchange Ratio’ based on the Parent’s ADS price, indicating a stock-based acquisition. Per Rule 31, this is classified as merger_agreement rather than contract.”}, {‘filename’: ‘contract_100_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including ‘THE MERGER,’ representations and warranties of the Company and Parent/Merger Sub, closing conditions, and termination provisions. The consideration section defines the Merger Consideration as ‘$56.00 in cash,’ confirming an all-cash deal. Per rules 31 and 33, this is classified as a merger_agreement with doc_subclass all_cash.”}]

failure_insights

mode_counts: {‘subclass_miss’: 10, ‘doc_type_miss’: 1} · n_failed: 11 · failures: [{‘filename’: ‘0001547903-13-000020_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate for Class A Common Stock of NMI Holdings, Inc., presented as Exhibit 4.1. Per rule 3, the SEC exhibit wrapper is disregarded in favor of the substantive form, which is a corporate record evidencing share ownership. It does not match any of the specifically enumerated record types in rule 33, so doc_subclass is set to ‘other’. Note: contract_subtype must be null when doc_type is not contract; corrected to null.”}, {‘filename’: ‘0001078782-12-001347_s1_ex3z2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SIMPLE PRODUCTS CORPORATION’ and contains standard internal governance provisions organized into Articles covering offices, stockholder/director meetings, officers, stock certificates, and indemnification. Per Rule 32, corporate records filed as SEC exhibits (e.g., EX-3.2) retain their substantive classification as corporate records rather than compliance filings. The title directly matches the ‘bylaws’ subclass per Rule 33.”}, {‘filename’: ‘0001615774-14-000417_s100510_ex4-6.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘predicted’: {‘doc_type’: ‘contract’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘doc_type_miss’, ‘reasoning’: “The document is a formal ‘WARRANT AGREEMENT’ executed between the issuing company and a warrant agent, detailing the issuance, exercise, redemption, adjustment, and transfer mechanics of the warrants. Its substantive form is a bilateral contract governing financial instrument terms. It does not fit any specific commercial family (e.g., license, distribution, service, or agency) and is not a corporate governance record or regulatory filing, thus classifying it as ‘other’. The SEC exhibit wrapper does not change its substantive contractual nature per rule 3.”}, {‘filename’: ‘0001144204-19-043489_tv528080_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is labeled EXHIBIT 3.2 and contains the Organizational Minutes of the Initial Board of Directors along with the full text of the BY-LAWS OF INCUMAKER, INC. Per Rule 32, corporate governance records like bylaws attached to SEC filings as exhibits retain their substantive classification as corporate_record rather than compliance_filing. Per Rule 33, the doc_subclass is correctly identified as ‘bylaws’ based on the document’s own title. No competing signals are present.”}, {‘filename’: ‘0001469709-14-000003_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘EXHIBIT 3.2 BYLAWS’ and contains standard internal governance provisions for a Nevada corporation, including sections on shareholder meetings, board of directors, officers, and amendment procedures. Per rule 32, exhibits filed as corporate governance records are classified as corporate_record, and per rule 33, the subclass is bylaws.”}, {‘filename’: ‘0001469709-14-000529_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SWEETS & TREATS, INC.’ and contains standard corporate governance provisions regarding stockholders, directors, officers, meetings, voting, and amendment procedures. Per Rule 32, corporate records such as bylaws filed as SEC exhibits (here, Exhibit 3.2) retain their substantive classification as corporate_record rather than compliance_filing. The content and exhibit wrapper confirm this classification.”}, {‘filename’: ‘0001547903-13-000006_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate for Class A Common Stock of NMI Holdings, Inc. It certifies share ownership, outlines transfer and assignment procedures, and features corporate seal and officer signature blocks. Its substantive form is a corporate securities/governance instrument rather than a commercial contract or regulatory filing. Since it does not match the specific record-type subcategories listed (e.g., bylaws, articles of incorporation), it is classified as corporate_record with doc_subclass ‘other’.”}, {‘filename’: ‘contract_105_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document title is explicitly ‘AGREEMENT AND PLAN OF MERGER’. It contains classic M&A deal structure provisions, including parties labeled as Parent, US Holdco, Merger Sub, and Company; dedicated articles for ‘THE MERGER’, ‘EFFECT OF THE MERGER ON CAPITAL STOCK’, representations and warranties, covenants, conditions precedent, and termination rights. The consideration mechanics reference ‘Parent Share Issuance’ and a ‘Parent Equity Conversion Ratio’ based on a ‘Cash Equivalent Consideration’ (standard for cross-border stock conversions), confirming an all-stock transaction. Under Rule 31, this is classified as a merger_agreement.”}, {‘filename’: ‘contract_145_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including ‘THE MERGER’, ‘Representations and Warranties’, ‘Conditions to the Merger’, ‘Termination’, and an ‘Exchange Ratio’ converting East Common Stock into Central Common Stock. Per Rule 31, this title and structure mandate classification as merger_agreement. The consideration section (Section 1.6) specifies that shares will be converted solely into shares of Central Common Stock (with cash provided only for fractional share interests), which aligns with the all_stock subclass.”}, {‘filename’: ‘contract_5_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A machinery: parties structured as Parent, Merger Subs, and Company; articles covering The Mergers, Reps & Warranties, Conditions to Closing, No-Solicitation covenants, and Termination Payments. The consideration structure centers on an ‘Exchange Ratio’ based on the Parent’s ADS price, indicating a stock-based acquisition. Per Rule 31, this is classified as merger_agreement rather than contract.”}, {‘filename’: ‘contract_100_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including ‘THE MERGER,’ representations and warranties of the Company and Parent/Merger Sub, closing conditions, and termination provisions. The consideration section defines the Merger Consideration as ‘$56.00 in cash,’ confirming an all-cash deal. Per rules 31 and 33, this is classified as a merger_agreement with doc_subclass all_cash.”}]

Per-subclass accuracy (second-level dimension)

subclass	accuracy	support (rows with GT)
Development	1	2
Franchise	1	1
Hosting	1	1
IP	1	1
Manufacturing	1	2
Outsourcing	1	1
Supply	1	1
Transportation	1	1
all_cash	1	3
all_stock	0.6667	3
articles_of_incorporation	0.2857	7
mixed_cash_stock	0.3333	3
other	0.0	1
powers_of_attorney	0.0	1
rights_instrument	1	2

subclass_accuracy_equiv: 0.6333 — strict subclass OR a defensible equivalent family (mixed_cash_stock ↔︎ mixed_cash_stock_election); equiv-recovered rows: none
input-mode split: text 30

qwen_qwen3.7-flash_sorter_docclass_v7_docclass_merged_ab (docclass_classification)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T18:01:30.740514+00:00
Model	qwen/qwen3.7-flash
Prompt version	sorter: sorter_docclass_v7
Git commit	`64420c6` (dirty tree)

Data source

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
ground_truth	doc_type + doc_subclass
ground_truth_mode	maud_consideration_gt / s1_record_type / cuad_subtype
dataset_fingerprint	fa16e25a91631122b3e1d2111dadeca05194bcf162dd22474a2890fc83fcb7cb
n_samples	30
sample_requested	0
stratified	30
seed	42

Parameters

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
sample	0
stratified	30
seed	42
max_input_chars	100000
max_tokens	4096
reasoning_effort	medium
temperature	0.1
input_mode	text
vision_pages	all
max_concurrency	10
rate_limit_retries	0
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_sorter_docclass_v7_docclass_merged_ab · trace_name: docclass_classification · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
doc_type_accuracy	0.9667
subclass_accuracy	0.7
subclass_accuracy_equiv	0.6667
exact_match	0.6667
confidence	0.957
n_rows	30
n_errors	0
equiv_recovered	—

Scores — doc_type_accuracy_ci

Field	Score
half	0.05
hi	1
lo	0.9
method	percentile-bootstrap
n	30
n_boot	2000
seed	42

Scores — subclass_accuracy_ci

Field	Score
half	0.1667
hi	0.8667
lo	0.5333
method	percentile-bootstrap
n	30
n_boot	2000
seed	42

Scores — exact_match_ci

Field	Score
half	0.1667
hi	0.8333
lo	0.5
method	percentile-bootstrap
n	30
n_boot	2000
seed	42

Scores — per_class_accuracy

Field	Score
contract	1
corporate_record	0.9
merger_agreement	1

Scores — per_subclass_accuracy

Field	Score
Development	1
Franchise	1
Hosting	1
IP	1
Manufacturing	1
Outsourcing	1
Supply	1
Transportation	1
all_cash	1
all_stock	0.6667
articles_of_incorporation	0.2857
mixed_cash_stock	0.6667
other	0.0
powers_of_attorney	0.0
rights_instrument	1

Scores — per_subclass_support

Field	Score
Development	2
Franchise	1
Hosting	1
IP	1
Manufacturing	2
Outsourcing	1
Supply	1
Transportation	1
all_cash	3
all_stock	3
articles_of_incorporation	7
mixed_cash_stock	3
other	1
powers_of_attorney	1
rights_instrument	2

Scores — subclass_confusion

Field	Score
Development	development: 2
Franchise	franchise: 1
Hosting	hosting: 1
IP	ip: 1
Manufacturing	manufacturing: 2
Outsourcing	outsourcing: 1
Supply	supply: 1
Transportation	transportation: 1
all_cash	all_cash: 3
all_stock	all_stock: 2 · all_cash: 1
articles_of_incorporation	articles_of_incorporation: 2 · other: 2 · bylaws: 3
mixed_cash_stock	mixed_cash_stock: 2 · all_stock: 1
other	all_stock: 1
powers_of_attorney	bylaws: 1
rights_instrument	rights_instrument: 1 · other: 1

Scores — input_mode_counts

Field	Score
text	30

Scores — sorter

Field	Score
failure_insights	mode_counts: {‘subclass_miss’: 9, ‘doc_type_miss’: 1} · n_failed: 10 · failures: [{‘filename’: ‘0001547903-13-000020_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate (Exhibit 4.1) for NMI Holdings, Inc., evidencing ownership of Class A Common Stock. It contains standard corporate governance and transfer language. Per SEC exhibit conventions, such instruments filed as exhibits are classified as corporate records. ‘Stock certificate’ is not a predefined subclass, so it defaults to ‘other’. Note: contract_subtype is set to ‘other’ per schema constraints when doc_type is not explicitly ‘contract’, but since the prompt says contract_subtype must be null when doc_type is NOT contract, I will correct that to null.”}, {‘filename’: ‘0001078782-12-001347_s1_ex3z2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SIMPLE PRODUCTS CORPORATION’ and contains standard internal governance provisions regarding offices, stockholder meetings, board composition, officers, and amendments. Filed as Exhibit 3.2, it squarely fits the corporate_record class with the bylaws subclass per rules 32 and 33.”}, {‘filename’: ‘0001615774-14-000417_s100510_ex4-6.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘predicted’: {‘doc_type’: ‘contract’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘doc_type_miss’, ‘reasoning’: “The document is explicitly titled ‘WARRANT AGREEMENT’ and contains standard bilateral contractual provisions between a company and a warrant agent governing the issuance, exercise, redemption, and adjustment of warrants. It does not match any of the specific named contract families, placing it in the ‘other’ subgroup. Although filed as an SEC exhibit (EX-4.6), its substantive form is a private contract rather than a corporate governance record or regulatory filing.”}, {‘filename’: ‘0001144204-19-043489_tv528080_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is filed as Exhibit 3.2 and contains organizational minutes alongside the ‘BY-LAWS OF INCUMAKER, INC.’. Per Rule 32, corporate governance records attached to SEC filings retain their substantive classification as corporate_record rather than compliance_filing. The explicit heading ‘BY-LAWS’ identifies the primary record type, triggering the bylaws subclass under Rule 33.”}, {‘filename’: ‘0001469709-14-000003_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS FOR THE REGULATION OF NIGHTFOOD HOLDINGS, INC.’ and contains standard internal governance provisions covering offices, shareholder and board meetings, officer appointments, indemnification, and amendment procedures. Filed as Exhibit 3.2, it is an internal corporate governance record per rule 32, and its subclass is determined directly from its own heading as ‘bylaws’ per rule 33.”}, {‘filename’: ‘0001469709-14-000529_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SWEETS & TREATS, INC.’ and contains standard corporate governance provisions regarding stockholders, directors, officers, meetings, and amendments. Filed as Exhibit 3.2, it squarely fits the corporate_record class and bylaws subclass per rules 32 and 33.”}, {‘filename’: ‘0001547903-13-000006_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate for NMI Holdings, Inc., filed as Exhibit 4.1. Per rule 32, corporate records attached to SEC filings retain their substantive classification as corporate records rather than compliance filings. It certifies equity ownership and outlines transfer/assignment mechanics, aligning with internal governance/capitalization records. Subclass is ‘other’ as it is a standard stock certificate not explicitly enumerated in the primary record-type list.”}, {‘filename’: ‘contract_145_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative clauses including a Merger Sub, Articles detailing the Merger mechanics, Representations & Warranties, Covenants, Conditions to Closing, and Termination. The consideration section (Section 1.6) specifies that East Common Stock will be converted solely into shares of Central Common Stock (with cash only for fractional shares), indicating an all-stock transaction. Embedded exhibits (Officer’s Certificates) do not change the parent class per rule 37.”}, {‘filename’: ‘contract_5_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains the hallmark structure of a public-company acquisition agreement, including parties designated as Parent, Merger Subs, and Company, alongside detailed articles on merger mechanics, representations, covenants, conditions to closing, and termination payments. Per rule 31, this definitively falls under merger_agreement. References to Exchange Ratios, Parent ADS prices, and stock conversion mechanics indicate a stock-based consideration, supporting the all_stock subclass.”}, {‘filename’: ‘contract_100_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard M&A structure with Parent, Merger Sub, and Company parties, representations, covenants, and closing conditions. Section 1.1(iii) defines the Merger Consideration as ‘$56.00 in cash’, confirming an all-cash transaction per Rules 31 and 33.”}]

failure_insights

mode_counts: {‘subclass_miss’: 9, ‘doc_type_miss’: 1} · n_failed: 10 · failures: [{‘filename’: ‘0001547903-13-000020_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate (Exhibit 4.1) for NMI Holdings, Inc., evidencing ownership of Class A Common Stock. It contains standard corporate governance and transfer language. Per SEC exhibit conventions, such instruments filed as exhibits are classified as corporate records. ‘Stock certificate’ is not a predefined subclass, so it defaults to ‘other’. Note: contract_subtype is set to ‘other’ per schema constraints when doc_type is not explicitly ‘contract’, but since the prompt says contract_subtype must be null when doc_type is NOT contract, I will correct that to null.”}, {‘filename’: ‘0001078782-12-001347_s1_ex3z2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SIMPLE PRODUCTS CORPORATION’ and contains standard internal governance provisions regarding offices, stockholder meetings, board composition, officers, and amendments. Filed as Exhibit 3.2, it squarely fits the corporate_record class with the bylaws subclass per rules 32 and 33.”}, {‘filename’: ‘0001615774-14-000417_s100510_ex4-6.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘predicted’: {‘doc_type’: ‘contract’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘doc_type_miss’, ‘reasoning’: “The document is explicitly titled ‘WARRANT AGREEMENT’ and contains standard bilateral contractual provisions between a company and a warrant agent governing the issuance, exercise, redemption, and adjustment of warrants. It does not match any of the specific named contract families, placing it in the ‘other’ subgroup. Although filed as an SEC exhibit (EX-4.6), its substantive form is a private contract rather than a corporate governance record or regulatory filing.”}, {‘filename’: ‘0001144204-19-043489_tv528080_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is filed as Exhibit 3.2 and contains organizational minutes alongside the ‘BY-LAWS OF INCUMAKER, INC.’. Per Rule 32, corporate governance records attached to SEC filings retain their substantive classification as corporate_record rather than compliance_filing. The explicit heading ‘BY-LAWS’ identifies the primary record type, triggering the bylaws subclass under Rule 33.”}, {‘filename’: ‘0001469709-14-000003_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS FOR THE REGULATION OF NIGHTFOOD HOLDINGS, INC.’ and contains standard internal governance provisions covering offices, shareholder and board meetings, officer appointments, indemnification, and amendment procedures. Filed as Exhibit 3.2, it is an internal corporate governance record per rule 32, and its subclass is determined directly from its own heading as ‘bylaws’ per rule 33.”}, {‘filename’: ‘0001469709-14-000529_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SWEETS & TREATS, INC.’ and contains standard corporate governance provisions regarding stockholders, directors, officers, meetings, and amendments. Filed as Exhibit 3.2, it squarely fits the corporate_record class and bylaws subclass per rules 32 and 33.”}, {‘filename’: ‘0001547903-13-000006_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate for NMI Holdings, Inc., filed as Exhibit 4.1. Per rule 32, corporate records attached to SEC filings retain their substantive classification as corporate records rather than compliance filings. It certifies equity ownership and outlines transfer/assignment mechanics, aligning with internal governance/capitalization records. Subclass is ‘other’ as it is a standard stock certificate not explicitly enumerated in the primary record-type list.”}, {‘filename’: ‘contract_145_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative clauses including a Merger Sub, Articles detailing the Merger mechanics, Representations & Warranties, Covenants, Conditions to Closing, and Termination. The consideration section (Section 1.6) specifies that East Common Stock will be converted solely into shares of Central Common Stock (with cash only for fractional shares), indicating an all-stock transaction. Embedded exhibits (Officer’s Certificates) do not change the parent class per rule 37.”}, {‘filename’: ‘contract_5_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains the hallmark structure of a public-company acquisition agreement, including parties designated as Parent, Merger Subs, and Company, alongside detailed articles on merger mechanics, representations, covenants, conditions to closing, and termination payments. Per rule 31, this definitively falls under merger_agreement. References to Exchange Ratios, Parent ADS prices, and stock conversion mechanics indicate a stock-based consideration, supporting the all_stock subclass.”}, {‘filename’: ‘contract_100_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard M&A structure with Parent, Merger Sub, and Company parties, representations, covenants, and closing conditions. Section 1.1(iii) defines the Merger Consideration as ‘$56.00 in cash’, confirming an all-cash transaction per Rules 31 and 33.”}]

Per-subclass accuracy (second-level dimension)

subclass	accuracy	support (rows with GT)
Development	1	2
Franchise	1	1
Hosting	1	1
IP	1	1
Manufacturing	1	2
Outsourcing	1	1
Supply	1	1
Transportation	1	1
all_cash	1	3
all_stock	0.6667	3
articles_of_incorporation	0.2857	7
mixed_cash_stock	0.6667	3
other	0.0	1
powers_of_attorney	0.0	1
rights_instrument	1	2

subclass_accuracy_equiv: 0.6667 — strict subclass OR a defensible equivalent family (mixed_cash_stock ↔︎ mixed_cash_stock_election); equiv-recovered rows: none
input-mode split: text 30

qwen_qwen3.7-flash_sorter_docclass_v6_docclass_ab120_s42 (docclass_classification)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T18:55:24.849191+00:00
Model	qwen/qwen3.7-flash
Prompt version	sorter: sorter_docclass_v6
Git commit	`388c466` (dirty tree)

Data source

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
ground_truth	doc_type + doc_subclass
ground_truth_mode	maud_consideration_gt / s1_record_type / cuad_subtype
dataset_fingerprint	05f05e9f45ce67c9880884f0378652369fda1782ebcd3141cfeea0259c1c490b
n_samples	120
sample_requested	0
stratified	0
seed	42

Parameters

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
sample	0
stratified	0
seed	42
max_input_chars	100000
max_tokens	4096
reasoning_effort	medium
temperature	0.1
input_mode	text
vision_pages	all
max_concurrency	13
rate_limit_retries	0
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_sorter_docclass_v6_docclass_ab120_s42 · trace_name: docclass_classification · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
doc_type_accuracy	0.9917
subclass_accuracy	0.5917
subclass_accuracy_equiv	0.5833
exact_match	0.5833
confidence	0.9445
n_rows	120
n_errors	0
equiv_recovered	—

Scores — doc_type_accuracy_ci

Field	Score
half	0.0125
hi	1
lo	0.975
method	percentile-bootstrap
n	120
n_boot	2000
seed	42

Scores — subclass_accuracy_ci

Field	Score
half	0.0875
hi	0.675
lo	0.5
method	percentile-bootstrap
n	120
n_boot	2000
seed	42

Scores — exact_match_ci

Field	Score
half	0.0917
hi	0.675
lo	0.4917
method	percentile-bootstrap
n	120
n_boot	2000
seed	42

Scores — per_class_accuracy

Field	Score
contract	1
corporate_record	0.9583
correspondence	1
insurance_claim	1
merger_agreement	1

Scores — per_subclass_accuracy

Field	Score
Agency Agreements	1
Co_Branding	1
Collaboration	1
Endorsement	1
Franchise	1
Hosting	1
License_Agreements	1
Outsourcing	1
Reseller	1
Sponsorship	1
Strategic Alliance	1
Supply	1
Transportation	1
all_cash	0.8
all_stock	1
articles_of_incorporation	0.5833
attorney_demand	0.0
bylaws	1
carrier	0.25
demand	0.0
email	1
inpatient	1
letter	0.4
meeting_request	0.0
memo	0.0
mixed_cash_stock	1
notice	0.4
other	0.0
outpatient	1
pde	0.0
powers_of_attorney	0.0
press_release	0.5
rights_instrument	1

Scores — per_subclass_support

Field	Score
Agency Agreements	2
Co_Branding	1
Collaboration	4
Endorsement	1
Franchise	2
Hosting	4
License_Agreements	2
Outsourcing	1
Reseller	1
Sponsorship	2
Strategic Alliance	2
Supply	1
Transportation	1
all_cash	5
all_stock	2
articles_of_incorporation	12
attorney_demand	1
bylaws	1
carrier	4
demand	2
email	4
inpatient	8
letter	5
meeting_request	3
memo	2
mixed_cash_stock	1
notice	5
other	17
outpatient	6
pde	6
powers_of_attorney	2
press_release	2
rights_instrument	8

Scores — subclass_confusion

Field	Score
Agency Agreements	agency: 2
Co_Branding	co_branding: 1
Collaboration	collaboration: 4
Endorsement	endorsement: 1
Franchise	franchise: 2
Hosting	hosting: 4
License_Agreements	license: 2
Outsourcing	outsourcing: 1
Reseller	reseller: 1
Sponsorship	sponsorship: 2
Strategic Alliance	strategic_alliance: 2
Supply	supply: 1
Transportation	transportation: 1
all_cash	all_cash: 4 · other: 1
all_stock	all_stock: 2
articles_of_incorporation	articles_of_incorporation: 7 · other: 1 · bylaws: 4
attorney_demand	email: 1
bylaws	bylaws: 1
carrier	carrier: 1 · other: 2 · outpatient: 1
demand	memo: 1 · email: 1
email	email: 4
inpatient	inpatient: 8
letter	letter: 2 · email: 3
meeting_request	notice: 1 · email: 2
memo	notice: 1 · email: 1
mixed_cash_stock	mixed_cash_stock: 1
notice	email: 3 · notice: 2
other	articles_of_incorporation: 1 · all_cash: 12 · all_stock: 3 · mixed_cash_stock: 1
outpatient	outpatient: 6
pde	other: 6
powers_of_attorney	bylaws: 2
press_release	email: 1 · press_release: 1
rights_instrument	rights_instrument: 7 · other: 1

Scores — input_mode_counts

Field	Score
text	120

Scores — sorter

Field	Score
failure_insights	mode_counts: {‘subclass_miss’: 49, ‘doc_type_miss’: 1} · n_failed: 50 · failures: [{‘filename’: ‘0001615774-14-000417_s100510_ex3-1.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is titled ‘Amended and Restated Memorandum and Articles of Association’ and filed as SEC Exhibit 3.1. Per rule 32, corporate governance documents filed as exhibits retain their substantive classification as corporate records. The content establishes the company’s name, registered office, share capital, and internal governance structure, functioning as the jurisdictional equivalent of articles of incorporation.”}, {‘filename’: ‘0001547903-13-000020_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate (Exhibit 4.1) issued by NMI Holdings, Inc. It certifies ownership of Class A Common Stock, outlines transfer procedures, and references the company’s governing documents. This format and content are characteristic of corporate records documenting equity issuance and securityholder rights, fitting the corporate_record class rather than contract, merger_agreement, correspondence, or insurance_claim.”}, {‘filename’: ‘0001469709-14-000003_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘EXHIBIT 3.2 BYLAWS’ and contains standard corporate governance provisions (offices, shareholder meetings, board of directors, officers, amendments). Per rule 32, corporate records filed as SEC exhibits retain their substantive form, making this a corporate_record with doc_subclass ‘bylaws’.”}, {‘filename’: ‘0001062993-15-000198_s1011515_ex3z2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF REVENUE.COM CORPORATION’ and contains standard internal governance provisions for stockholders, directors, officers, capital stock, and indemnification. Although filed as SEC Exhibit 3.2, Rule 32 dictates that such exhibits retain their substantive classification as corporate records rather than compliance filings. The content directly matches the ‘bylaws’ subclass.”}, {‘filename’: ‘0001547903-13-000006_a45warrantno1fbrcapitalmar.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘predicted’: {‘doc_type’: ‘contract’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘doc_type_miss’, ‘reasoning’: “The document is a standalone Warrant Agreement filed as SEC Exhibit 4.5. It establishes a formal agreement between NMI Holdings, Inc. and FBR Capital Markets & Co., detailing the right to purchase common stock, exercise mechanics, anti-dilution adjustments, and transfer restrictions. As it does not align with any of the specified commercial contract families, it is classified as ‘other’ under the contract category.”}, {‘filename’: ‘0001144204-19-043489_tv528080_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document contains the Organizational Minutes and the full text of the ‘By-Laws of Incumaker, Inc.’, filed as SEC Exhibit 3.2. Per Rule 32, corporate governance records like bylaws attached as exhibits are classified as corporate_record. Per Rule 33, the doc_subclass is bylaws based on the explicit title within the document. No contract or other class signals are present.”}, {‘filename’: ‘0001079974-08-000839_artdimensionss1x32_9252008.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF ARTDIMENSIONS, INC.’ and filed as SEC Exhibit 3.2. Its substantive content covers standard corporate governance provisions including principal offices, registered agent, stockholder and director meetings, officer roles, capital stock, dividends, and indemnification. Per rules 32 and 33, the SEC exhibit wrapper does not change the substantive form, which is a corporate record classified as bylaws.”}, {‘filename’: ‘0001144204-19-030811_tv521467_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is filed as SEC Exhibit 3.2 and contains the Organizational Minutes of the Initial Board of Directors followed by the complete ‘BY-LAWS OF INCUMAKER, INC.’ Per Rule 32, corporate records like bylaws attached as SEC exhibits are classified as corporate_record. Per Rule 33, the subclass is bylaws based on the document’s own title and content.”}, {‘filename’: ‘0001469709-14-000529_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SWEETS & TREATS, INC.’ and filed as SEC Exhibit 3.2. It contains standard corporate governance provisions covering stockholders, directors, officers, meetings, voting, and amendment procedures. Per rules 32 and 33, this is classified as a corporate_record with the subclass bylaws.”}, {‘filename’: ‘shackleton-s/all_documents/3774.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is an internal business email coordinating the execution and routing of attached notices and letters. It features standard email phrasing (‘To catch everyone up’, ‘I will be out of the office’), direct routing instructions to multiple parties, and a professional signature block with contact details, firmly placing it in the correspondence class.”}, {‘filename’: ‘hyatt-k/to_do/51.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document consists solely of a task assignment header with administrative fields (status, priority, dates). It lacks substantive legal language, contractual obligations, or corporate governance content. Given the limited options, it best fits correspondence as an internal administrative memo/note. Confidence is low due to the minimal content and absence of clear legal context.’}, {‘filename’: ‘ermis-f/all_documents/303.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The text is a standalone corporate disclaimer clarifying the operational and regulatory separation between PG&E National Energy Group and the regulated utility. It functions as a business notice or email accompanying an invitation, lacking any contractual, corporate governance, M&A, or insurance claim characteristics.’}, {‘filename’: ‘taylor-m/all_documents/7663.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘press_release’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is an email forwarding a press release regarding bankruptcy legislation. It contains standard email routing headers (‘Forwarded by…’, ‘To:’, ‘Subject:’) and includes the full text of a press release, matching the correspondence class.”}, {‘filename’: ‘sanders-r/inbox/38.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a business email discussing litigation strategy, due diligence, and scheduling meetings with the plaintiff. It explicitly refers to itself as ‘this email’ and includes a standard email confidentiality notice at the end, firmly placing it in the correspondence category.”}, {‘filename’: ‘dasovich-j/all_documents/3005.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an internal business email with standard headers (To, cc, Subject) requesting meeting availability for a specific date. It clearly matches the correspondence class and specifically functions as an email/meeting request.’}, {‘filename’: ‘campbell-l/inbox/618.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an automated commercial email/notification from 1-800 CONTACTS to a customer regarding re-ordering contact lenses. It contains standard email formatting, a direct greeting, a secure link for ordering, customer support contact information, and an unsubscribe instruction, clearly fitting the correspondence class with an email subclass.’}, {‘filename’: ‘sanders-r/px/17.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘attorney_demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an internal email circulating revisions to a draft arbitration demand letter among committee members. Its format, salutation, and transmittal language clearly identify it as email correspondence rather than the underlying demand letter itself.’}, {‘filename’: ‘shackleton-s/inbox/675.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly formatted as a formal notice (labeled ‘Notice # 02-46’) with standard TO/FROM/RE headers, communicating procedural instructions to exchange members. It squarely fits the correspondence class.”}, {‘filename’: ‘kean-s/all_documents/1990.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The text is a brief internal communication/email discussing the distribution of a regulatory analysis memo to a stakeholder committee and government officials. It lacks any contractual, corporate, M&A, or insurance claim characteristics, fitting squarely into the correspondence category.’}, {‘filename’: ‘schoolcraft-d/inbox/junk/351.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a weekly industry newsletter/email digest from Rigzone featuring oil & gas headlines, market prices, event listings, and subscription management instructions. It contains no operative legal terms, agreements, or claims, squarely fitting the correspondence class as an email.’}, {‘filename’: ‘storey-g/sent_items/76.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a chain of internal corporate emails discussing terms for commodity trading confirmations (gas delivery points and pipeline nomination deadlines). It lacks operative contract language and clearly fits the correspondence class.’}, {‘filename’: ‘campbell-l/all_documents/1428.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a direct email notification from a customer relations representative to a discussion list regarding scheduled system maintenance. It lacks contractual, corporate, or M&A elements, fitting squarely into the correspondence class as an operational email.’}, {‘filename’: ‘kaminski-v/inbox/81.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a direct email exchange between two academics discussing conference travel and a seminar invitation. It features standard email formatting, conversational tone, and lacks any operative legal clauses, corporate governance records, or M&A/insurance structures.’}, {‘filename’: ‘staab-t/deleted_items/418.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a direct email from an executive to a specific recipient, containing standard email headers/footers (Message-Id, subscribe/unsubscribe instructions, ‘This email has been sent to…’). The content is a customer/stakeholder update regarding an aviation incident and precautionary inspections, fitting the correspondence class perfectly. No contractual or corporate record elements are present.”}, {‘filename’: ‘pde:233634491375303.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement containing a coverage determination (‘APPROVED’), prescription details, and cost-share breakdown. This directly aligns with the insurance_claim class, which encompasses coverage determinations and EOB-style claim documents.”}, {‘filename’: ‘carrier:887993387407016.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘carrier’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘MEDICARE SUMMARY NOTICE – PHYSICIAN/SUPPLIER CLAIM’ and contains standard claim adjudication details including beneficiary information, service dates, HCPCS codes, allowed/paid amounts, and a coverage determination. This matches the insurance_claim class (EOB/adjudication notice). Doc_subclass is set to other as it does not fit the specific carrier/pde/outpatient/inpatient categories.”}, {‘filename’: ‘pde:233804493732348.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a pharmacy statement and coverage determination for a Medicare Part D prescription drug claim, containing beneficiary details, prescription information, cost-sharing breakdown, and an explicit ‘APPROVED’ coverage outcome. This matches the format and content of an insurance claim document.”}, {‘filename’: ‘carrier:887433385493589.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘carrier’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Summary Notice detailing a physician/supplier claim, including adjudication results, allowed/paid amounts, and a coverage determination (‘APPROVED’). This squarely matches the insurance_claim category. Subclass is set to ‘other’ as it functions as a standard EOB/summary notice rather than fitting the specific inpatient, outpatient, PDE, or carrier sub-types.”}, {‘filename’: ‘pde:233934491832935.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement detailing a prescription fill, cost share, and explicitly includes a ‘COVERAGE DETERMINATION: APPROVED’ section. This format and content directly match insurance claim documentation, specifically an EOB or coverage determination record.”}, {‘filename’: ‘pde:233714489398465.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement containing beneficiary details, prescription information, cost-sharing breakdown, and a coverage determination (‘APPROVED’). This directly matches the insurance_claim class, specifically functioning as an EOB or coverage determination record.”}, {‘filename’: ‘carrier:887413386039256.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘carrier’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘outpatient’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Summary Notice for a physician/supplier claim, containing claim adjudication details, itemized service lines, payment amounts, and a coverage determination (‘APPROVED’). This aligns directly with the insurance_claim category, which encompasses claim documentation and coverage determinations.”}, {‘filename’: ‘pde:233464489735061.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement detailing a prescription fill, cost share, and explicitly includes a ‘COVERAGE DETERMINATION: APPROVED’. This directly matches the insurance_claim category, which covers claim documentation, EOB statements, and coverage determinations.”}, {‘filename’: ‘pde:233114489887590.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement/EOB detailing an insured beneficiary, prescription details, cost share, and a coverage determination (‘APPROVED’). This directly matches the insurance_claim class, which encompasses EOB statements and coverage determinations.”}, {‘filename’: ‘contract_123_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and details a tender offer followed by a statutory merger between Safari Parent, Inc., Safari Merger Subsidiary, Inc., and Seacor Holdings Inc. It contains standard M&A operative machinery including representations and warranties, conditions to closing, termination covenants, and specifies that shares will be converted into a fixed cash amount ($41.50 per share). Per Rule 31, this title and structure mandate the merger_agreement class. The consideration is exclusively cash, triggering the all_cash subclass under Rule 33. Embedded corporate records (e.g., Exhibit A Certificate of Incorporation) are governed by Rule 34 and do not change the parent classification.”}, {‘filename’: ‘contract_50_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery, including a Parent/Merger Sub structure, Effective Time mechanics, merger consideration, representations and warranties, and closing conditions. Per rule 31, this squarely fits the merger_agreement class. Sections 2.1(d) and 2.2(d) specify that eligible shares will be converted into the right to receive a fixed amount in cash (amended in the attached schedule to $12.15 and $8.35 per share respectively), with no stock component, satisfying the all_cash subclass definition under rule 33.”}, {‘filename’: ‘contract_119_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard M&A structure with defined parties (Company, Parent, Merger Subsidiary), Articles covering the Merger mechanics, Conversion of Shares, extensive Representations and Warranties, Conditions to Closing, and Termination provisions. Section 2.02 specifies that each share of Company Stock will be converted into the right to receive ‘$58.00 in cash’, establishing an all-cash consideration. Under Rules 31 and 33, this squarely fits the merger_agreement class with an all_cash subclass.”}, {‘filename’: ‘contract_26_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains classic M&A machinery: a ‘Parent’ and ‘Merger Sub’ counterparty, detailed representations and warranties, conditions to closing, termination fees, and no-shop/solicitation covenants. This directly triggers Rule 31 for merger_agreement. The transaction is funded through referenced Debt and Equity Commitment Letters and third-party Guarantors, which is characteristic of a private equity-led cash acquisition rather than a stock-based exchange, supporting the all_cash subclass.”}, {‘filename’: ‘contract_113_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure provisions, including a tender offer, merger mechanics, capital stock conversion, extensive representations and warranties, covenants, and closing conditions. The consideration is explicitly stated as ‘$22.50 per Share… in cash’, aligning with the all_cash subclass.”}, {‘filename’: ‘contract_69_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard structure of an M&A transaction agreement, including articles for the Merger, Delivery of Merger Consideration, Representations and Warranties, Covenants, Conditions to Closing, and Termination. The consideration section defines the ‘Per Share Merger Consideration’ as ‘$156.00 per Share in cash,’ confirming an all-cash deal. This squarely fits the merger_agreement class with an all_cash subclass.”}, {‘filename’: ‘contract_120_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains classic M&A operative machinery: parties designated as ‘Parent’ and ‘Merger Sub’, sections on ‘The Merger’ and ‘Effective Time’, extensive representations and warranties, no-shop/fiduciary out covenants, and termination fees. Per Rule 31, this title and structure strictly dictate the merger_agreement class. The deal structure reflects a private-equity-led acquisition (Mirasol Parent/Thoma Bravo acquiring RealPage), which utilizes cash consideration, justifying the all_cash subclass.”}, {‘filename’: ‘contract_22_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A machinery including a Parent, Merger Sub, and Company, with sections detailing the merger effective time, capital stock conversion, extensive representations and warranties, covenants, conditions to closing, and termination rights. Section 4.1(a) and 4.3 specify the merger consideration as ‘$25.75 per Share in cash’, making it an all-cash transaction. Embedded corporate documents like Annex B do not alter the parent classification per rule 34.”}, {‘filename’: ‘contract_16_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure machinery: a Parent/Merger Sub counterparty setup, Effective Time/Closing mechanics, extensive Representations and Warranties, no-shop/termination covenants, and disclosure schedules. Per Rule 31, this squarely falls under merger_agreement. Section 3.1(a)(i) specifies that each share of Common Stock shall convert into the right to receive ‘$56.00 in cash’, indicating an all-cash consideration structure, which triggers doc_subclass ‘all_cash’ per Rule 33.”}, {‘filename’: ‘contract_110_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure provisions, including parties designated as ‘Parent’ and ‘Merger Sub’, articles detailing ‘THE MERGER’, ‘MERGER CONSIDERATION’, extensive representations and warranties, covenants, closing conditions, and termination fees. This directly triggers rule 31 for the merger_agreement class. The specific form of merger consideration (cash vs. stock) is not detailed in the provided excerpt, so ‘other’ is assigned for doc_subclass per rule 33.”}, {‘filename’: ‘contract_93_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure provisions, including merger mechanics, stock exchange ratios, representations and warranties, covenants, and termination clauses. Under rule 31, this squarely falls under merger_agreement. The consideration section specifies that Company Common Stock will be converted into an exchange ratio of Buyer Common Stock (0.275 shares), indicating an all-stock transaction, which sets doc_subclass to all_stock.”}, {‘filename’: ‘contract_68_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal machinery, including a Parent and Merger Sub structure, Effective Time mechanics, exchange ratios for stock consideration, extensive representations and warranties, and closing conditions. Per Rule 31, this triggers the merger_agreement class. The consideration section specifies conversion of Company Common Stock into shares of Parent Common Stock, indicating an all-stock consideration type per Rule 33. Ancillary registration rights and adoption agreements are treated as embedded annexes and do not alter the primary classification.”}, {‘filename’: ‘contract_141_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative provisions, including a ‘Merger Sub’, ‘Effective Time’, share conversion mechanics, extensive representations and warranties, conditions to closing, and termination fees. Section 2.1(a) specifies that each share of Company Common Stock will be converted into ‘$177.50 in cash’, indicating an all-cash consideration structure. Under Rule 31 and Rule 33, this clearly classifies as a merger_agreement with an all_cash subclass.”}, {‘filename’: ‘contract_31_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard M&A structure with a Parent, Merger Sub, and Target Company. Article II outlines the merger mechanics, while Article III specifies that each share of Company Common Stock will be converted into a fixed number of shares of Parent Common Stock (Exchange Ratio of 1.46), with only standard cash-in-lieu-of-fractions provisions. This confirms an all-stock consideration structure, satisfying rule 31 for merger_agreement and rule 33 for doc_subclass.”}, {‘filename’: ‘contract_9_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AMENDED AND RESTATED AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery, including a cash tender offer, merger mechanics, representations and warranties, covenants, conditions precedent, and termination provisions. The consideration is strictly cash ($37.00 per share), satisfying the all_cash subclass criteria under Rule 33.”}, {‘filename’: ‘contract_76_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including the mergers, conversion of shares, representations and warranties, covenants, conditions, and termination. Section 2.1 specifies that shareholders will receive both Parent Common Shares and $90.00 in cash per share, indicating a mixed cash and stock consideration. Thus, it fits the merger_agreement class with doc_subclass mixed_cash_stock.”}, {‘filename’: ‘contract_59_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure machinery, including a Parent/Merger Sub counterparty setup, Effective Time mechanics, extensive representations and warranties, conditions precedent, and termination covenants. Section 2.1 specifies that each share of common stock will be converted into the right to receive $19.25 in cash, confirming an all-cash consideration type. Per Rule 31 and Rule 33, this is classified as a merger_agreement with doc_subclass all_cash.”}]

failure_insights

mode_counts: {‘subclass_miss’: 49, ‘doc_type_miss’: 1} · n_failed: 50 · failures: [{‘filename’: ‘0001615774-14-000417_s100510_ex3-1.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is titled ‘Amended and Restated Memorandum and Articles of Association’ and filed as SEC Exhibit 3.1. Per rule 32, corporate governance documents filed as exhibits retain their substantive classification as corporate records. The content establishes the company’s name, registered office, share capital, and internal governance structure, functioning as the jurisdictional equivalent of articles of incorporation.”}, {‘filename’: ‘0001547903-13-000020_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a specimen stock certificate (Exhibit 4.1) issued by NMI Holdings, Inc. It certifies ownership of Class A Common Stock, outlines transfer procedures, and references the company’s governing documents. This format and content are characteristic of corporate records documenting equity issuance and securityholder rights, fitting the corporate_record class rather than contract, merger_agreement, correspondence, or insurance_claim.”}, {‘filename’: ‘0001469709-14-000003_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘EXHIBIT 3.2 BYLAWS’ and contains standard corporate governance provisions (offices, shareholder meetings, board of directors, officers, amendments). Per rule 32, corporate records filed as SEC exhibits retain their substantive form, making this a corporate_record with doc_subclass ‘bylaws’.”}, {‘filename’: ‘0001062993-15-000198_s1011515_ex3z2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF REVENUE.COM CORPORATION’ and contains standard internal governance provisions for stockholders, directors, officers, capital stock, and indemnification. Although filed as SEC Exhibit 3.2, Rule 32 dictates that such exhibits retain their substantive classification as corporate records rather than compliance filings. The content directly matches the ‘bylaws’ subclass.”}, {‘filename’: ‘0001547903-13-000006_a45warrantno1fbrcapitalmar.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘predicted’: {‘doc_type’: ‘contract’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘doc_type_miss’, ‘reasoning’: “The document is a standalone Warrant Agreement filed as SEC Exhibit 4.5. It establishes a formal agreement between NMI Holdings, Inc. and FBR Capital Markets & Co., detailing the right to purchase common stock, exercise mechanics, anti-dilution adjustments, and transfer restrictions. As it does not align with any of the specified commercial contract families, it is classified as ‘other’ under the contract category.”}, {‘filename’: ‘0001144204-19-043489_tv528080_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document contains the Organizational Minutes and the full text of the ‘By-Laws of Incumaker, Inc.’, filed as SEC Exhibit 3.2. Per Rule 32, corporate governance records like bylaws attached as exhibits are classified as corporate_record. Per Rule 33, the doc_subclass is bylaws based on the explicit title within the document. No contract or other class signals are present.”}, {‘filename’: ‘0001079974-08-000839_artdimensionss1x32_9252008.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF ARTDIMENSIONS, INC.’ and filed as SEC Exhibit 3.2. Its substantive content covers standard corporate governance provisions including principal offices, registered agent, stockholder and director meetings, officer roles, capital stock, dividends, and indemnification. Per rules 32 and 33, the SEC exhibit wrapper does not change the substantive form, which is a corporate record classified as bylaws.”}, {‘filename’: ‘0001144204-19-030811_tv521467_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is filed as SEC Exhibit 3.2 and contains the Organizational Minutes of the Initial Board of Directors followed by the complete ‘BY-LAWS OF INCUMAKER, INC.’ Per Rule 32, corporate records like bylaws attached as SEC exhibits are classified as corporate_record. Per Rule 33, the subclass is bylaws based on the document’s own title and content.”}, {‘filename’: ‘0001469709-14-000529_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SWEETS & TREATS, INC.’ and filed as SEC Exhibit 3.2. It contains standard corporate governance provisions covering stockholders, directors, officers, meetings, voting, and amendment procedures. Per rules 32 and 33, this is classified as a corporate_record with the subclass bylaws.”}, {‘filename’: ‘shackleton-s/all_documents/3774.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is an internal business email coordinating the execution and routing of attached notices and letters. It features standard email phrasing (‘To catch everyone up’, ‘I will be out of the office’), direct routing instructions to multiple parties, and a professional signature block with contact details, firmly placing it in the correspondence class.”}, {‘filename’: ‘hyatt-k/to_do/51.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document consists solely of a task assignment header with administrative fields (status, priority, dates). It lacks substantive legal language, contractual obligations, or corporate governance content. Given the limited options, it best fits correspondence as an internal administrative memo/note. Confidence is low due to the minimal content and absence of clear legal context.’}, {‘filename’: ‘ermis-f/all_documents/303.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The text is a standalone corporate disclaimer clarifying the operational and regulatory separation between PG&E National Energy Group and the regulated utility. It functions as a business notice or email accompanying an invitation, lacking any contractual, corporate governance, M&A, or insurance claim characteristics.’}, {‘filename’: ‘taylor-m/all_documents/7663.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘press_release’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is an email forwarding a press release regarding bankruptcy legislation. It contains standard email routing headers (‘Forwarded by…’, ‘To:’, ‘Subject:’) and includes the full text of a press release, matching the correspondence class.”}, {‘filename’: ‘sanders-r/inbox/38.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a business email discussing litigation strategy, due diligence, and scheduling meetings with the plaintiff. It explicitly refers to itself as ‘this email’ and includes a standard email confidentiality notice at the end, firmly placing it in the correspondence category.”}, {‘filename’: ‘dasovich-j/all_documents/3005.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an internal business email with standard headers (To, cc, Subject) requesting meeting availability for a specific date. It clearly matches the correspondence class and specifically functions as an email/meeting request.’}, {‘filename’: ‘campbell-l/inbox/618.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an automated commercial email/notification from 1-800 CONTACTS to a customer regarding re-ordering contact lenses. It contains standard email formatting, a direct greeting, a secure link for ordering, customer support contact information, and an unsubscribe instruction, clearly fitting the correspondence class with an email subclass.’}, {‘filename’: ‘sanders-r/px/17.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘attorney_demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an internal email circulating revisions to a draft arbitration demand letter among committee members. Its format, salutation, and transmittal language clearly identify it as email correspondence rather than the underlying demand letter itself.’}, {‘filename’: ‘shackleton-s/inbox/675.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly formatted as a formal notice (labeled ‘Notice # 02-46’) with standard TO/FROM/RE headers, communicating procedural instructions to exchange members. It squarely fits the correspondence class.”}, {‘filename’: ‘kean-s/all_documents/1990.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The text is a brief internal communication/email discussing the distribution of a regulatory analysis memo to a stakeholder committee and government officials. It lacks any contractual, corporate, M&A, or insurance claim characteristics, fitting squarely into the correspondence category.’}, {‘filename’: ‘schoolcraft-d/inbox/junk/351.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a weekly industry newsletter/email digest from Rigzone featuring oil & gas headlines, market prices, event listings, and subscription management instructions. It contains no operative legal terms, agreements, or claims, squarely fitting the correspondence class as an email.’}, {‘filename’: ‘storey-g/sent_items/76.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a chain of internal corporate emails discussing terms for commodity trading confirmations (gas delivery points and pipeline nomination deadlines). It lacks operative contract language and clearly fits the correspondence class.’}, {‘filename’: ‘campbell-l/all_documents/1428.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a direct email notification from a customer relations representative to a discussion list regarding scheduled system maintenance. It lacks contractual, corporate, or M&A elements, fitting squarely into the correspondence class as an operational email.’}, {‘filename’: ‘kaminski-v/inbox/81.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a direct email exchange between two academics discussing conference travel and a seminar invitation. It features standard email formatting, conversational tone, and lacks any operative legal clauses, corporate governance records, or M&A/insurance structures.’}, {‘filename’: ‘staab-t/deleted_items/418.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a direct email from an executive to a specific recipient, containing standard email headers/footers (Message-Id, subscribe/unsubscribe instructions, ‘This email has been sent to…’). The content is a customer/stakeholder update regarding an aviation incident and precautionary inspections, fitting the correspondence class perfectly. No contractual or corporate record elements are present.”}, {‘filename’: ‘pde:233634491375303.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement containing a coverage determination (‘APPROVED’), prescription details, and cost-share breakdown. This directly aligns with the insurance_claim class, which encompasses coverage determinations and EOB-style claim documents.”}, {‘filename’: ‘carrier:887993387407016.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘carrier’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘MEDICARE SUMMARY NOTICE – PHYSICIAN/SUPPLIER CLAIM’ and contains standard claim adjudication details including beneficiary information, service dates, HCPCS codes, allowed/paid amounts, and a coverage determination. This matches the insurance_claim class (EOB/adjudication notice). Doc_subclass is set to other as it does not fit the specific carrier/pde/outpatient/inpatient categories.”}, {‘filename’: ‘pde:233804493732348.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a pharmacy statement and coverage determination for a Medicare Part D prescription drug claim, containing beneficiary details, prescription information, cost-sharing breakdown, and an explicit ‘APPROVED’ coverage outcome. This matches the format and content of an insurance claim document.”}, {‘filename’: ‘carrier:887433385493589.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘carrier’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Summary Notice detailing a physician/supplier claim, including adjudication results, allowed/paid amounts, and a coverage determination (‘APPROVED’). This squarely matches the insurance_claim category. Subclass is set to ‘other’ as it functions as a standard EOB/summary notice rather than fitting the specific inpatient, outpatient, PDE, or carrier sub-types.”}, {‘filename’: ‘pde:233934491832935.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement detailing a prescription fill, cost share, and explicitly includes a ‘COVERAGE DETERMINATION: APPROVED’ section. This format and content directly match insurance claim documentation, specifically an EOB or coverage determination record.”}, {‘filename’: ‘pde:233714489398465.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement containing beneficiary details, prescription information, cost-sharing breakdown, and a coverage determination (‘APPROVED’). This directly matches the insurance_claim class, specifically functioning as an EOB or coverage determination record.”}, {‘filename’: ‘carrier:887413386039256.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘carrier’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘outpatient’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Summary Notice for a physician/supplier claim, containing claim adjudication details, itemized service lines, payment amounts, and a coverage determination (‘APPROVED’). This aligns directly with the insurance_claim category, which encompasses claim documentation and coverage determinations.”}, {‘filename’: ‘pde:233464489735061.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement detailing a prescription fill, cost share, and explicitly includes a ‘COVERAGE DETERMINATION: APPROVED’. This directly matches the insurance_claim category, which covers claim documentation, EOB statements, and coverage determinations.”}, {‘filename’: ‘pde:233114489887590.txt’, ‘expected’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘pde’}, ‘predicted’: {‘doc_type’: ‘insurance_claim’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a Medicare Part D pharmacy statement/EOB detailing an insured beneficiary, prescription details, cost share, and a coverage determination (‘APPROVED’). This directly matches the insurance_claim class, which encompasses EOB statements and coverage determinations.”}, {‘filename’: ‘contract_123_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and details a tender offer followed by a statutory merger between Safari Parent, Inc., Safari Merger Subsidiary, Inc., and Seacor Holdings Inc. It contains standard M&A operative machinery including representations and warranties, conditions to closing, termination covenants, and specifies that shares will be converted into a fixed cash amount ($41.50 per share). Per Rule 31, this title and structure mandate the merger_agreement class. The consideration is exclusively cash, triggering the all_cash subclass under Rule 33. Embedded corporate records (e.g., Exhibit A Certificate of Incorporation) are governed by Rule 34 and do not change the parent classification.”}, {‘filename’: ‘contract_50_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery, including a Parent/Merger Sub structure, Effective Time mechanics, merger consideration, representations and warranties, and closing conditions. Per rule 31, this squarely fits the merger_agreement class. Sections 2.1(d) and 2.2(d) specify that eligible shares will be converted into the right to receive a fixed amount in cash (amended in the attached schedule to $12.15 and $8.35 per share respectively), with no stock component, satisfying the all_cash subclass definition under rule 33.”}, {‘filename’: ‘contract_119_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard M&A structure with defined parties (Company, Parent, Merger Subsidiary), Articles covering the Merger mechanics, Conversion of Shares, extensive Representations and Warranties, Conditions to Closing, and Termination provisions. Section 2.02 specifies that each share of Company Stock will be converted into the right to receive ‘$58.00 in cash’, establishing an all-cash consideration. Under Rules 31 and 33, this squarely fits the merger_agreement class with an all_cash subclass.”}, {‘filename’: ‘contract_26_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains classic M&A machinery: a ‘Parent’ and ‘Merger Sub’ counterparty, detailed representations and warranties, conditions to closing, termination fees, and no-shop/solicitation covenants. This directly triggers Rule 31 for merger_agreement. The transaction is funded through referenced Debt and Equity Commitment Letters and third-party Guarantors, which is characteristic of a private equity-led cash acquisition rather than a stock-based exchange, supporting the all_cash subclass.”}, {‘filename’: ‘contract_113_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure provisions, including a tender offer, merger mechanics, capital stock conversion, extensive representations and warranties, covenants, and closing conditions. The consideration is explicitly stated as ‘$22.50 per Share… in cash’, aligning with the all_cash subclass.”}, {‘filename’: ‘contract_69_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard structure of an M&A transaction agreement, including articles for the Merger, Delivery of Merger Consideration, Representations and Warranties, Covenants, Conditions to Closing, and Termination. The consideration section defines the ‘Per Share Merger Consideration’ as ‘$156.00 per Share in cash,’ confirming an all-cash deal. This squarely fits the merger_agreement class with an all_cash subclass.”}, {‘filename’: ‘contract_120_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains classic M&A operative machinery: parties designated as ‘Parent’ and ‘Merger Sub’, sections on ‘The Merger’ and ‘Effective Time’, extensive representations and warranties, no-shop/fiduciary out covenants, and termination fees. Per Rule 31, this title and structure strictly dictate the merger_agreement class. The deal structure reflects a private-equity-led acquisition (Mirasol Parent/Thoma Bravo acquiring RealPage), which utilizes cash consideration, justifying the all_cash subclass.”}, {‘filename’: ‘contract_22_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A machinery including a Parent, Merger Sub, and Company, with sections detailing the merger effective time, capital stock conversion, extensive representations and warranties, covenants, conditions to closing, and termination rights. Section 4.1(a) and 4.3 specify the merger consideration as ‘$25.75 per Share in cash’, making it an all-cash transaction. Embedded corporate documents like Annex B do not alter the parent classification per rule 34.”}, {‘filename’: ‘contract_16_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure machinery: a Parent/Merger Sub counterparty setup, Effective Time/Closing mechanics, extensive Representations and Warranties, no-shop/termination covenants, and disclosure schedules. Per Rule 31, this squarely falls under merger_agreement. Section 3.1(a)(i) specifies that each share of Common Stock shall convert into the right to receive ‘$56.00 in cash’, indicating an all-cash consideration structure, which triggers doc_subclass ‘all_cash’ per Rule 33.”}, {‘filename’: ‘contract_110_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure provisions, including parties designated as ‘Parent’ and ‘Merger Sub’, articles detailing ‘THE MERGER’, ‘MERGER CONSIDERATION’, extensive representations and warranties, covenants, closing conditions, and termination fees. This directly triggers rule 31 for the merger_agreement class. The specific form of merger consideration (cash vs. stock) is not detailed in the provided excerpt, so ‘other’ is assigned for doc_subclass per rule 33.”}, {‘filename’: ‘contract_93_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure provisions, including merger mechanics, stock exchange ratios, representations and warranties, covenants, and termination clauses. Under rule 31, this squarely falls under merger_agreement. The consideration section specifies that Company Common Stock will be converted into an exchange ratio of Buyer Common Stock (0.275 shares), indicating an all-stock transaction, which sets doc_subclass to all_stock.”}, {‘filename’: ‘contract_68_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal machinery, including a Parent and Merger Sub structure, Effective Time mechanics, exchange ratios for stock consideration, extensive representations and warranties, and closing conditions. Per Rule 31, this triggers the merger_agreement class. The consideration section specifies conversion of Company Common Stock into shares of Parent Common Stock, indicating an all-stock consideration type per Rule 33. Ancillary registration rights and adoption agreements are treated as embedded annexes and do not alter the primary classification.”}, {‘filename’: ‘contract_141_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative provisions, including a ‘Merger Sub’, ‘Effective Time’, share conversion mechanics, extensive representations and warranties, conditions to closing, and termination fees. Section 2.1(a) specifies that each share of Company Common Stock will be converted into ‘$177.50 in cash’, indicating an all-cash consideration structure. Under Rule 31 and Rule 33, this clearly classifies as a merger_agreement with an all_cash subclass.”}, {‘filename’: ‘contract_31_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard M&A structure with a Parent, Merger Sub, and Target Company. Article II outlines the merger mechanics, while Article III specifies that each share of Company Common Stock will be converted into a fixed number of shares of Parent Common Stock (Exchange Ratio of 1.46), with only standard cash-in-lieu-of-fractions provisions. This confirms an all-stock consideration structure, satisfying rule 31 for merger_agreement and rule 33 for doc_subclass.”}, {‘filename’: ‘contract_9_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AMENDED AND RESTATED AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery, including a cash tender offer, merger mechanics, representations and warranties, covenants, conditions precedent, and termination provisions. The consideration is strictly cash ($37.00 per share), satisfying the all_cash subclass criteria under Rule 33.”}, {‘filename’: ‘contract_76_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including the mergers, conversion of shares, representations and warranties, covenants, conditions, and termination. Section 2.1 specifies that shareholders will receive both Parent Common Shares and $90.00 in cash per share, indicating a mixed cash and stock consideration. Thus, it fits the merger_agreement class with doc_subclass mixed_cash_stock.”}, {‘filename’: ‘contract_59_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure machinery, including a Parent/Merger Sub counterparty setup, Effective Time mechanics, extensive representations and warranties, conditions precedent, and termination covenants. Section 2.1 specifies that each share of common stock will be converted into the right to receive $19.25 in cash, confirming an all-cash consideration type. Per Rule 31 and Rule 33, this is classified as a merger_agreement with doc_subclass all_cash.”}]

Per-subclass accuracy (second-level dimension)

subclass	accuracy	support (rows with GT)
Agency Agreements	1	2
Co_Branding	1	1
Collaboration	1	4
Endorsement	1	1
Franchise	1	2
Hosting	1	4
License_Agreements	1	2
Outsourcing	1	1
Reseller	1	1
Sponsorship	1	2
Strategic Alliance	1	2
Supply	1	1
Transportation	1	1
all_cash	0.8	5
all_stock	1	2
articles_of_incorporation	0.5833	12
attorney_demand	0.0	1
bylaws	1	1
carrier	0.25	4
demand	0.0	2
email	1	4
inpatient	1	8
letter	0.4	5
meeting_request	0.0	3
memo	0.0	2
mixed_cash_stock	1	1
notice	0.4	5
other	0.0	17
outpatient	1	6
pde	0.0	6
powers_of_attorney	0.0	2
press_release	0.5	2
rights_instrument	1	8

subclass_accuracy_equiv: 0.5833 — strict subclass OR a defensible equivalent family (mixed_cash_stock ↔︎ mixed_cash_stock_election); equiv-recovered rows: none
input-mode split: text 120

qwen_qwen3.7-flash_sorter_docclass_v7_docclass_ab120_s42 (docclass_classification)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T18:56:53.692191+00:00
Model	qwen/qwen3.7-flash
Prompt version	sorter: sorter_docclass_v7
Git commit	`388c466` (dirty tree)

Data source

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
ground_truth	doc_type + doc_subclass
ground_truth_mode	maud_consideration_gt / s1_record_type / cuad_subtype
dataset_fingerprint	05f05e9f45ce67c9880884f0378652369fda1782ebcd3141cfeea0259c1c490b
n_samples	120
sample_requested	0
stratified	0
seed	42

Parameters

Key	Value
datasets	mailroom-maud-contracts,mailroom-cuad-contracts-full,mailroom-s1-corporate-records
sample	0
stratified	0
seed	42
max_input_chars	100000
max_tokens	4096
reasoning_effort	medium
temperature	0.1
input_mode	text
vision_pages	all
max_concurrency	13
rate_limit_retries	0
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_sorter_docclass_v7_docclass_ab120_s42 · trace_name: docclass_classification · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
doc_type_accuracy	0.9917
subclass_accuracy	0.6917
subclass_accuracy_equiv	0.6833
exact_match	0.6833
confidence	0.9547
n_rows	120
n_errors	0
equiv_recovered	—

Scores — doc_type_accuracy_ci

Field	Score
half	0.0125
hi	1
lo	0.975
method	percentile-bootstrap
n	120
n_boot	2000
seed	42

Scores — subclass_accuracy_ci

Field	Score
half	0.0792
hi	0.7667
lo	0.6083
method	percentile-bootstrap
n	120
n_boot	2000
seed	42

Scores — exact_match_ci

Field	Score
half	0.0791
hi	0.7583
lo	0.6
method	percentile-bootstrap
n	120
n_boot	2000
seed	42

Scores — per_class_accuracy

Field	Score
contract	1
corporate_record	0.9583
correspondence	1
insurance_claim	1
merger_agreement	1

Scores — per_subclass_accuracy

Field	Score
Agency Agreements	1
Co_Branding	1
Collaboration	1
Endorsement	1
Franchise	1
Hosting	1
License_Agreements	1
Outsourcing	1
Reseller	1
Sponsorship	1
Strategic Alliance	1
Supply	1
Transportation	1
all_cash	0.8
all_stock	1
articles_of_incorporation	0.5833
attorney_demand	0.0
bylaws	1
carrier	1
demand	0.0
email	1
inpatient	1
letter	0.4
meeting_request	0.3333
memo	0.0
mixed_cash_stock	1
notice	0.4
other	0.0588
outpatient	1
pde	1
powers_of_attorney	0.0
press_release	1
rights_instrument	1

Scores — per_subclass_support

Field	Score
Agency Agreements	2
Co_Branding	1
Collaboration	4
Endorsement	1
Franchise	2
Hosting	4
License_Agreements	2
Outsourcing	1
Reseller	1
Sponsorship	2
Strategic Alliance	2
Supply	1
Transportation	1
all_cash	5
all_stock	2
articles_of_incorporation	12
attorney_demand	1
bylaws	1
carrier	4
demand	2
email	4
inpatient	8
letter	5
meeting_request	3
memo	2
mixed_cash_stock	1
notice	5
other	17
outpatient	6
pde	6
powers_of_attorney	2
press_release	2
rights_instrument	8

Scores — subclass_confusion

Field	Score
Agency Agreements	agency: 2
Co_Branding	co_branding: 1
Collaboration	collaboration: 4
Endorsement	endorsement: 1
Franchise	franchise: 2
Hosting	hosting: 4
License_Agreements	license: 2
Outsourcing	outsourcing: 1
Reseller	reseller: 1
Sponsorship	sponsorship: 2
Strategic Alliance	strategic_alliance: 2
Supply	supply: 1
Transportation	transportation: 1
all_cash	all_cash: 4 · other: 1
all_stock	all_stock: 2
articles_of_incorporation	articles_of_incorporation: 7 · rights_instrument: 1 · bylaws: 4
attorney_demand	email: 1
bylaws	bylaws: 1
carrier	carrier: 4
demand	memo: 1 · email: 1
email	email: 4
inpatient	inpatient: 8
letter	letter: 2 · email: 3
meeting_request	letter: 1 · email: 1 · meeting_request: 1
memo	notice: 1 · email: 1
mixed_cash_stock	mixed_cash_stock: 1
notice	email: 3 · notice: 2
other	articles_of_incorporation: 1 · all_cash: 11 · other: 1 · all_stock: 3 · mixed_cash_stock: 1
outpatient	outpatient: 6
pde	pde: 6
powers_of_attorney	bylaws: 2
press_release	press_release: 2
rights_instrument	rights_instrument: 7 · other: 1

Scores — input_mode_counts

Field	Score
text	120

Scores — sorter

failure_insights

mode_counts: {‘subclass_miss’: 37, ‘doc_type_miss’: 1} · n_failed: 38 · failures: [{‘filename’: ‘0001615774-14-000417_s100510_ex3-1.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is filed as Exhibit 3.1 and titled ‘AMENDED AND RESTATED MEMORANDUM AND ARTICLES OF ASSOCIATION’. Its content exclusively details corporate governance structures including the company name, registered office, share capital, director appointments and powers, general meetings, voting procedures, and indemnification. Per rules 32 and 33, this is a corporate record classified under articles_of_incorporation.”}, {‘filename’: ‘0001547903-13-000020_a41specimenclassacommonsto.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a specimen stock certificate for Class A Common Stock of NMI Holdings, Inc., filed as Exhibit 4.1. It functions as an instrument defining the ownership and rights of a securityholder, aligning with the corpus convention for securityholder instruments filed as SEC exhibits. Classified as corporate_record with doc_subclass rights_instrument.’}, {‘filename’: ‘0001469709-14-000003_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS FOR THE REGULATION OF NIGHTFOOD HOLDINGS, INC.’ and labeled as Exhibit 3.2. Its content consists entirely of standard corporate governance provisions governing offices, shareholder and director meetings, officer appointments, and amendment procedures. Per rules 32 and 33, the SEC exhibit wrapper does not change the substantive form, making this a corporate_record classified as bylaws.”}, {‘filename’: ‘0001062993-15-000198_s1011515_ex3z2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF REVENUE.COM CORPORATION’ and appears as Exhibit 3.2. Per rule 32, corporate governance records like bylaws filed as SEC exhibits are classified as corporate_record. Per rule 33, the subclass is determined by the document’s own heading, making it ‘bylaws’. Note: contract_subtype must be null for non-contracts, but the schema enum requires a value or null; I will set it to null as instructed.”}, {‘filename’: ‘0001144204-19-043489_tv528080_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is labeled Exhibit 3.2 and contains the Organizational Minutes of the Initial Board of Directors followed by the complete text of the ‘BY-LAWS OF INCUMAKER, INC.’ Under rules 32 and 33, corporate governance records such as bylaws filed as SEC exhibits are classified as corporate_record, with the subclass assigned based on the document’s own title and content.”}, {‘filename’: ‘0001079974-08-000839_artdimensionss1x32_9252008.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF ART DIMENSIONS, INC.’ and filed as Exhibit 3.2. Its content consists entirely of standard corporate governance provisions (offices, registered agent, stockholder/board meetings, officers, capital stock, indemnification), which directly matches the corporate_record class and bylaws subclass under rules 32 and 33.”}, {‘filename’: ‘0001615774-14-000417_s100510_ex4-7.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘rights_instrument’}, ‘predicted’: {‘doc_type’: ‘contract’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘doc_type_miss’, ‘reasoning’: “The document is a Unit Purchase Option agreement filed as Exhibit 4.7, detailing the terms, exercise mechanics, transfer restrictions, and registration rights for purchasing units of CB Pharma Acquisition Corp. It constitutes a formal agreement between the issuer and the holder. Because it governs securities purchase rights and does not align with any of the specific enumerated contract families (e.g., license, distribution, service, or endorsement), it is classified as a contract with the subtype ‘other’.”}, {‘filename’: ‘0001144204-19-030811_tv521467_ex3-2.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘articles_of_incorporation’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is filed as SEC Exhibit 3.2 and contains the ‘Organizational Minutes’ and the full text of the ‘By-Laws of Incumaker, Inc.’ Per Rule 32, corporate records such as bylaws attached to an SEC exhibit are classified as corporate_record. Rule 33 directs classifying the subclass based on the document’s own title, which explicitly names the bylaws, making ‘bylaws’ the correct doc_subclass.”}, {‘filename’: ‘0001469709-14-000529_ex3_2apg.htm’, ‘expected’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘powers_of_attorney’}, ‘predicted’: {‘doc_type’: ‘corporate_record’, ‘doc_subclass’: ‘bylaws’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘BYLAWS OF SWEETS & TREATS, INC.’ and filed as Exhibit 3.2. It contains standard internal governance provisions for stockholders, directors, officers, and meeting procedures. Per rules 32 and 33, corporate records like bylaws filed as SEC exhibits are classified as corporate_record with the doc_subclass ‘bylaws’.”}, {‘filename’: ‘shackleton-s/all_documents/3774.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is a business email coordinating the execution and routing of attached notices (Asset Notice, Prepayment Notice, Direction Letter). It features standard email conventions such as ‘Attached to this e-mail,’ direct instructional paragraphs to specific recipients, and contact/fax details. It lacks operative contractual terms, corporate governance language, or M&A/insurance claim structures, firmly placing it in the correspondence class.”}, {‘filename’: ‘hyatt-k/to_do/51.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a brief administrative record containing only task metadata (status, priority, dates). It lacks substantive legal terms, corporate governance language, or M&A/insurance structures. Given the available classes, it best fits correspondence as an internal memo/note, though extreme brevity limits confidence.’}, {‘filename’: ‘ermis-f/all_documents/303.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a standalone corporate disclaimer/announcement clarifying the operational and regulatory separation between PG&E National Energy Group and Pacific Gas and Electric Company. It contains no contractual obligations, corporate governance records, M&A transaction mechanics, or insurance claim data. Its formal, informational nature aligns with business correspondence, specifically a letter.’}, {‘filename’: ‘sanders-r/inbox/38.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘meeting_request’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is an informal internal communication discussing litigation preparation, due diligence, and scheduling a meeting with opposing counsel (‘Bid/Ask’). It contains standard email formatting and a confidentiality notice, matching the ‘email’ subclass under correspondence.”}, {‘filename’: ‘campbell-l/inbox/618.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a direct customer communication from 1-800 CONTACTS containing a personalized re-order link, instructions to reply or call, and an unsubscribe option, which clearly identifies it as an email correspondence.’}, {‘filename’: ‘sanders-r/px/17.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘attorney_demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an internal forwarding message/email circulating a revised draft of an arbitration demand letter. It lacks any operative contractual terms, corporate governance structure, or M&A deal mechanics. Its informal forwarding language and attachment reference clearly identify it as business correspondence, specifically an email.’}, {‘filename’: ‘shackleton-s/inbox/675.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘Notice # 02-46’ and uses a standard TO/FROM/RE format to communicate procedural updates regarding EFP/EFS trade execution to NYMEX members. It functions as a formal regulatory/internal notice rather than a contract, corporate record, or insurance claim.”}, {‘filename’: ‘kean-s/all_documents/1990.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘memo’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a brief business communication/email discussing the distribution of a prepared analysis memo to industry stakeholders and government officials regarding regulatory rule changes. It lacks operative agreement terms, corporate governance records, or claim documentation, clearly matching the correspondence class.’}, {‘filename’: ‘schoolcraft-d/inbox/junk/351.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘demand’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a daily industry newsletter/email digest from Rigzone featuring compiled oil and gas headlines, event listings, advertisements, and subscriber management/unsubscribe instructions. It lacks any contractual, corporate governance, M&A, or insurance claim substance, fitting squarely into the correspondence class as an email broadcast.’}, {‘filename’: ‘storey-g/sent_items/76.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a threaded email exchange between colleagues discussing specific contractual options (delivery point selections) to be added to commodity trading confirmations. Standard email headers (From, Sent, To, Subject) and informal business language clearly identify it as correspondence.’}, {‘filename’: ‘campbell-l/all_documents/1428.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘notice’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is formatted as an email sent to a discussion list (‘writes to the NYISO_TECH_EXCHANGE Discussion List’) announcing routine system maintenance. It contains standard email salutations and sign-offs, lacks any contractual, corporate record, merger, or insurance claim elements, and clearly fits the correspondence class with an email subclass.”}, {‘filename’: ‘kaminski-v/inbox/81.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is an informal email thread discussing academic conference attendance and a seminar invitation. It lacks any contractual obligations, corporate governance records, or M&A/insurance structures, clearly fitting the correspondence class as an email.’}, {‘filename’: ‘staab-t/deleted_items/418.’, ‘expected’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘letter’}, ‘predicted’: {‘doc_type’: ‘correspondence’, ‘doc_subclass’: ‘email’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: ‘The document is a direct electronic communication from an American Airlines executive to a specific recipient, providing operational updates and expressing gratitude following the 9/11 attacks. It contains explicit email metadata (Message-ID, unsubscribe instructions, recipient address) and follows a standard business email format, firmly placing it in the correspondence class under the email subclass.’}, {‘filename’: ‘contract_123_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative clauses including a tender offer, merger mechanics, capital stock treatment, representations and warranties, closing conditions, and termination provisions. The consideration is specified as $41.50 per share in cash, making it an all-cash transaction. Embedded exhibits like the Certificate of Incorporation do not alter the primary classification per corpus conventions.”}, {‘filename’: ‘contract_50_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including Article I (The Mergers), Article II (Effect on Capital Stock), Article III (Payment for Shares in cash at $11.69 and $7.81 per share), along with typical representations, warranties, conditions to closing, and termination provisions. Per Rule 31, this triggers the merger_agreement class. The consideration is exclusively cash, satisfying Rule 33 for doc_subclass all_cash.”}, {‘filename’: ‘contract_119_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative machinery including a ‘Parent’, ‘Merger Subsidiary’, and ‘Company’, along with articles detailing ‘THE MERGER’, ‘Conversion of Shares’, representations, covenants, conditions, and termination. Section 2.02 specifies that shares will be converted for ‘$58.00 in cash’, indicating an all-cash consideration structure. Therefore, it classifies as a merger_agreement with an all_cash subclass.”}, {‘filename’: ‘contract_113_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A machinery including a tender offer, merger mechanics, representations and warranties, covenants, and termination provisions. The consideration section specifies a fixed cash price of $22.50 per share, indicating an all-cash transaction. Per rule 31 and 33, this is classified as a merger_agreement with an all_cash subclass.”}, {‘filename’: ‘contract_69_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard structure of a public-company acquisition agreement, featuring parties designated as ‘Company’, ‘Parent’, and ‘Merger Sub’, along with standard M&A articles such as ‘The Merger’, ‘Closing’, ‘Representations and Warranties’, ‘Conditions to Closing’, and termination provisions. The consideration section explicitly defines the ‘Per Share Merger Consideration’ as ‘$156.00 per Share in cash,’ confirming an all-cash deal. This directly triggers rule 31 for merger_agreement and rule 33 for the all_cash subclass.”}, {‘filename’: ‘contract_120_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A structure including Parent/Merger Sub/Company parties, merger mechanics (Effective Time, Closing), extensive representations and warranties, conditions to closing, no-shop covenants, and termination fees. Per Rule 31, this is classified as a merger_agreement. The transaction involves a private equity acquirer (Thoma Bravo) with references to debt and equity financing commitments and lacks any stock exchange ratio or conversion mechanics, indicating an all-cash consideration structure for doc_subclass.”}, {‘filename’: ‘contract_22_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and features standard M&A parties (Company, Parent, Merger Sub). Operative clauses detail merger mechanics, closing procedures, representations and warranties, covenants, and conditions to closing. Section 4.3 specifies the merger consideration as ‘$25.75 per Share in cash,’ indicating an all-cash transaction. Under Rule 31, this title and structure mandate classification as merger_agreement. Under Rule 33, the consideration type is all_cash. Embedded corporate documents like Annex B are annex content per Rule 34.”}, {‘filename’: ‘contract_16_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard M&A structure with a Parent/Merger Sub counterparty, articles governing ‘THE MERGER’, ‘EFFECTIVE TIME’, capital stock conversion, representations and warranties, covenants, conditions to closing, and termination rights. Section 3.1(a) specifies that all outstanding shares will convert into the right to receive ‘$56.00 in cash’, confirming an all-cash consideration structure.”}, {‘filename’: ‘contract_110_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative clauses, including parties labeled as ‘Parent,’ ‘Merger Sub,’ and ‘Company,’ along with articles detailing ‘THE MERGER,’ ‘MERGER CONSIDERATION,’ representations, warranties, covenants, closing conditions, and termination fees. This squarely matches the merger_agreement class per Rule 31. The specific consideration breakdown (cash vs. stock) is not fully visible in the provided excerpt, so ‘other’ is assigned for doc_subclass.”}, {‘filename’: ‘contract_93_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative provisions including merger mechanics (Article I), stockholder approval, representations and warranties (Articles III-IV), covenants (Article V), closing conditions (Article VI), and termination fees (Article VII). Section 2.01 specifies that Company Common Stock will be converted into an exchange ratio of Buyer Common Stock, indicating a pure stock-for-stock transaction. This squarely matches the merger_agreement class under rule 31, with an all_stock subclass per rule 33.”}, {‘filename’: ‘contract_68_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A deal structure provisions (Parent/Merger Sub parties, Effective Time mechanics, stock exchange ratio for Merger Consideration, reps & warranties, covenants, conditions precedent). Per Rules 31 and 36, the primary M&A agreement governs the classification despite the inclusion of ancillary documents like a Registration Rights Agreement and Amendment No. 1 within the same filing package. The merger consideration consists of shares of Parent Common Stock (with cash only for fractional shares), supporting an all_stock subclass.”}, {‘filename’: ‘contract_141_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative provisions, including a ‘Merger Sub,’ ‘Effective Time,’ share conversion mechanics, extensive representations and warranties, and termination fees. Section 2.1(a) specifies that each share of Company Common Stock will be converted into ‘$177.50 in cash,’ confirming an all-cash consideration structure. Under Rule 31 and Rule 33, this is classified as a merger_agreement with doc_subclass all_cash.”}, {‘filename’: ‘contract_31_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A machinery including a Parent/Merger Sub/Company structure, Effective Time mechanics, Exchange Ratio for stock conversion, extensive representations and warranties, no-shop covenants, and conditions precedent. Per Rule 31, this triggers merger_agreement. Section 3.1 specifies that Company Common Stock will be converted solely into shares of Parent Common Stock at a fixed Exchange Ratio (1.46), with only standard cash-in-lieu-of-fractions provisions, indicating an all-stock consideration, which sets doc_subclass to all_stock per Rule 33.”}, {‘filename’: ‘contract_9_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document title explicitly identifies it as an ‘AMENDED AND RESTATED AGREEMENT AND PLAN OF MERGER’. The operative provisions detail a cash tender offer followed by a statutory merger, including standard M&A machinery such as representations and warranties, covenants, closing conditions, termination rights, and a fixed cash merger consideration ($37.00 per share). Under Rule 31, this falls squarely under merger_agreement. Per Rule 33, the consideration is exclusively cash, making the doc_subclass all_cash.”}, {‘filename’: ‘contract_76_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘mixed_cash_stock’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and contains standard M&A operative clauses including ‘THE MERGERS’, ‘CONVERSION OF SHARES’, representations and warranties, covenants, conditions to closing, and termination provisions. Section 2.1 specifies that shares will be converted into Parent Common Shares plus $90.00 in cash per share, indicating a mixed cash and stock consideration. Per Rules 31 and 33, this is classified as a merger_agreement with doc_subclass mixed_cash_stock.”}, {‘filename’: ‘contract_59_merger_agreement.txt’, ‘expected’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘other’}, ‘predicted’: {‘doc_type’: ‘merger_agreement’, ‘doc_subclass’: ‘all_cash’}, ‘failure_mode’: ‘subclass_miss’, ‘reasoning’: “The document is explicitly titled ‘AGREEMENT AND PLAN OF MERGER’ and follows the standard structure of a corporate acquisition agreement, featuring a ‘Parent’, ‘Merger Sub’, and ‘Company’ counterparty, along with articles covering the merger mechanics, capital stock conversion, representations and warranties, covenants, and closing conditions. Section 2.1(a) explicitly states that each share will be converted into the right to receive ‘$19.25 in cash’, confirming an all-cash consideration structure. This squarely fits the merger_agreement class with an all_cash subclass.”}]

Per-subclass accuracy (second-level dimension)

subclass	accuracy	support (rows with GT)
Agency Agreements	1	2
Co_Branding	1	1
Collaboration	1	4
Endorsement	1	1
Franchise	1	2
Hosting	1	4
License_Agreements	1	2
Outsourcing	1	1
Reseller	1	1
Sponsorship	1	2
Strategic Alliance	1	2
Supply	1	1
Transportation	1	1
all_cash	0.8	5
all_stock	1	2
articles_of_incorporation	0.5833	12
attorney_demand	0.0	1
bylaws	1	1
carrier	1	4
demand	0.0	2
email	1	4
inpatient	1	8
letter	0.4	5
meeting_request	0.3333	3
memo	0.0	2
mixed_cash_stock	1	1
notice	0.4	5
other	0.0588	17
outpatient	1	6
pde	1	6
powers_of_attorney	0.0	2
press_release	1	2
rights_instrument	1	8

subclass_accuracy_equiv: 0.6833 — strict subclass OR a defensible equivalent family (mixed_cash_stock ↔︎ mixed_cash_stock_election); equiv-recovered rows: none
input-mode split: text 120

qwen_qwen3.7-flash_contracts_specialist_docclass_v0_docclass_ab120_s42 (docclass_specialist_extraction)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T18:57:55.585037+00:00
Model	qwen/qwen3.7-flash
Prompt version	contracts_specialist: contracts_specialist_docclass_v0
Git commit	`388c466` (dirty tree)

Data source

Key	Value
local_dumps	data/datasets/docclass_merged_v5.jsonl
agent	contracts_specialist
ground_truth	docclass_merged_v5_gt_fields
dataset_fingerprint	34083b161a915165e1a4c6a0cfa01324f15d8310bf20427cea1e893dc4e1acec
n_samples	24
n_routed	48
filename_manifest	data/manifests/docclass_ab120_s42_filenames.jsonl
seed	42
stratified	0

Parameters

Key	Value
agent	contracts_specialist
prompt_version	contracts_specialist_docclass_v0
chunked	✓
max_input_chars	150000
max_tokens	32768
reasoning_effort	none
temperature	0.1
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_contracts_specialist_docclass_v0_docclass_ab120_s42 · trace_name: docclass_specialist_extraction · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
overall_extraction_score	0.6884
field_presence	0.9502
overall_verified_precision	0.9888
category_presence	0.3588
n_rows	24
n_errors	0
n_routed	48

qwen_qwen3.7-flash_contracts_specialist_docclass_v1_docclass_ab120_s42 (docclass_specialist_extraction)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T18:59:54.737881+00:00
Model	qwen/qwen3.7-flash
Prompt version	contracts_specialist: contracts_specialist_docclass_v1
Git commit	`388c466` (dirty tree)

Data source

Key	Value
local_dumps	data/datasets/docclass_merged_v5.jsonl
agent	contracts_specialist
ground_truth	docclass_merged_v5_gt_fields
dataset_fingerprint	34083b161a915165e1a4c6a0cfa01324f15d8310bf20427cea1e893dc4e1acec
n_samples	24
n_routed	48
filename_manifest	data/manifests/docclass_ab120_s42_filenames.jsonl
seed	42
stratified	0

Parameters

Key	Value
agent	contracts_specialist
prompt_version	contracts_specialist_docclass_v1
chunked	✓
max_input_chars	150000
max_tokens	32768
reasoning_effort	none
temperature	0.1
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_contracts_specialist_docclass_v1_docclass_ab120_s42 · trace_name: docclass_specialist_extraction · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
overall_extraction_score	0.8444
field_presence	0.9562
overall_verified_precision	0.968
category_presence	0.7449
n_rows	24
n_errors	0
n_routed	48

qwen_qwen3.7-flash_insurance_claims_specialist_docclass_v0_docclass_ab120_s42 (docclass_specialist_extraction)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T19:00:07.283289+00:00
Model	qwen/qwen3.7-flash
Prompt version	insurance_claims_specialist: insurance_claims_specialist_docclass_v0
Git commit	`388c466` (dirty tree)

Data source

Key	Value
local_dumps	data/datasets/docclass_merged_v5.jsonl
agent	insurance_claims_specialist
ground_truth	docclass_merged_v5_gt_fields
dataset_fingerprint	b238c80e946cfa0ece004c0129708a233e129f0168d7f02cf7002f78c45133ac
n_samples	24
n_routed	24
filename_manifest	data/manifests/docclass_ab120_s42_filenames.jsonl
seed	42
stratified	0

Parameters

Key	Value
agent	insurance_claims_specialist
prompt_version	insurance_claims_specialist_docclass_v0
chunked	✗
max_input_chars	150000
max_tokens	32768
reasoning_effort	none
temperature	0.1
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_insurance_claims_specialist_docclass_v0_docclass_ab120_s42 · trace_name: docclass_specialist_extraction · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
overall_extraction_score	0.6957
field_presence	0.7311
overall_verified_precision	1
category_presence	—
n_rows	24
n_errors	0
n_routed	24

qwen_qwen3.7-flash_insurance_claims_specialist_docclass_v1_docclass_ab120_s42 (docclass_specialist_extraction)

Interactive explorer detail →

Run metadata

Key	Value
Timestamp	2026-08-26T19:00:19.960201+00:00
Model	qwen/qwen3.7-flash
Prompt version	insurance_claims_specialist: insurance_claims_specialist_docclass_v1
Git commit	`388c466` (dirty tree)

Data source

Key	Value
local_dumps	data/datasets/docclass_merged_v5.jsonl
agent	insurance_claims_specialist
ground_truth	docclass_merged_v5_gt_fields
dataset_fingerprint	b238c80e946cfa0ece004c0129708a233e129f0168d7f02cf7002f78c45133ac
n_samples	24
n_routed	24
filename_manifest	data/manifests/docclass_ab120_s42_filenames.jsonl
seed	42
stratified	0

Parameters

Key	Value
agent	insurance_claims_specialist
prompt_version	insurance_claims_specialist_docclass_v1
chunked	✗
max_input_chars	150000
max_tokens	32768
reasoning_effort	none
temperature	0.1
tracing_backend	langfuse
tracing_meta	project: llm-dojo · environment: llm-dojo · base_url: https://us.cloud.langfuse.com · session_id: qwen_qwen3.7-flash_insurance_claims_specialist_docclass_v1_docclass_ab120_s42 · trace_name: docclass_specialist_extraction · disabled: False

Token usage

Stage	Prompt	Completion	Total	Mean cost $	Total cost $
all	—	—	—	—	—

Scores

Score	Value
overall_extraction_score	0.6904
field_presence	0.7273
overall_verified_precision	1
category_presence	—
n_rows	24
n_errors	0
n_routed	24