Measure usefulness, not just agreement.

Proposed protocol

Agree on the reference standard first.

• Define the intended task, eligibility boundaries, and job-related criteria.
• Select a representative, authorised sample before reviewing results, keeping a held-out evaluation set where appropriate.
• Have qualified reviewers adjudicate evidence with a documented disagreement-resolution process. Blind model and source labels where feasible.
• Compare the existing workflow or defined baseline against the proposed process on the same cases.
• Report denominators, failed runs, exclusions, uncertainty, and observed review effort, not a selected set of success stories.
• Document limits. A small pilot does not establish broad fairness, reliability, or regulatory compliance.

Separate model calls do not establish statistically independent errors. Treat model-provided confidence as self-reported, not calibrated accuracy.

Primary sources

What does agreement establish?

Agreement describes the returned opinions, not whether a candidate meets the criteria. Separate calls can share errors. Check every rationale against original evidence, keep missing responses visible, and record the human judgment separately.

In the fixed synthetic example, direct-report management is preferred, not essential. Treating its absence as an automatic rejection changes the stated criteria. This is a worked interpretation, not measured performance.

• NIST AI Risk Management Framework — risk measurement and management guidance, not product certification.
• EU AI Act: Regulation (EU) 2024/1689 — primary legislative text; applicability needs qualified review.
• EEOC: Employment tests and selection procedures — US selection guidance, not a statement that a deployment complies.

Published evidence

Results belong beside their method.