If your company uses AI in hiring, you've likely heard that "bias audits" are required. New York City, California, and Colorado all mandate some form of bias testing for AI hiring tools. But what does a bias audit actually entail? What data do you need? What methodologies are acceptable? How do you interpret the results? And critically—what do you do if the audit reveals discrimination?
This guide walks through the complete bias audit process from initial scoping to publication of results, with practical examples, statistical explanations in plain English, and decision frameworks for addressing findings. Whether you're preparing for compliance or seeking best practices, this resource equips HR and legal teams to navigate AI hiring risks effectively.
Who This Guide Is For:
- ✓ HR/Talent leaders responsible for AI hiring compliance
- ✓ Legal/compliance teams evaluating vendor tools
- ✓ In-house analysts tasked with conducting audits
- ✓ Anyone trying to understand what bias audits cost and deliver
What Is a Bias Audit? (Legal Definition)
A bias audit is a statistical analysis that evaluates whether an AI hiring tool produces disparate impact—meaning it disproportionately screens out candidates from protected classes, such as race, ethnicity, sex, age, or disability.
The legal framework stems from two key sources:
- Federal precedent: The "four-fifths rule" from the Uniform Guidelines on Employee Selection Procedures (1978), enforced by the EEOC.
- State and local laws: Mandates in NYC Local Law 144, California AB 2930, and Colorado's AI Act.
These laws generally require analysis of selection rates by race/ethnicity and sex at minimum, with expansions to age, disability, and intersectional categories (e.g., Black women) in some areas. For federal details, review the EEOC's guidance on employment selection procedures.
Step 1: Scope the Audit (What Tool, What Data, What Period)
Define the Tool Being Audited
Specify the tool clearly to ensure focused testing:
- Tool name and version: E.g., "HireVue Video Interview Platform v8.2"
- What it evaluates: E.g., "Candidate speech patterns, word choice, and verbal skills"
- How it's used: E.g., "Scores rank candidates; top 30% advance to interviews"
- Job categories covered: E.g., "Customer service and sales roles"
If the tool applies across diverse job families, conduct separate audits to align with rules like those from the NYC Department of Consumer and Worker Protection.
Determine the Audit Period
Laws like NYC's require data from the prior 12 months. For a February 2026 audit, use March 2025–February 2026 data. California and Colorado follow similar timelines.
Minimum sample size: At least 500 candidates per NYC guidelines for reliable analysis. If short, extend the window or aggregate similar roles—but document to avoid skewing results. See EEOC sample size guidance for validity tips.
Identify Required Demographic Data
Gather data on:
- Race/Ethnicity: EEOC categories (Hispanic/Latino, White, Black/African American, Asian, American Indian/Alaska Native, Native Hawaiian/Pacific Islander, Two or More Races)
- Sex: Male, female, non-binary
Challenges with data collection: Race/ethnicity is voluntary under EEOC rules, so gaps are common. Options include:
- Prospective collection: Implement voluntary self-identification forms from EEOC templates and audit after 12 months.
- Inference methods: Use proxies like names or zip codes (use cautiously; review FTC privacy resources).
- Vendor data: Leverage aggregated insights if permitted by your jurisdiction.
For geographic compliance, align forms with laws like California's CCPA, adding privacy notices.
Step 2: Collect and Prepare Data
Data Elements You Need
For each candidate:
- Anonymized ID (per EEOC privacy guidelines)
- Job title/category
- Application date
- AI usage (yes/no)
- AI output (score/ranking)
- Outcome (advanced/hired/rejected)
- Demographics (race/ethnicity, sex)
Comply with FERPA for education data or HIPAA for disabilities.
Data Cleaning and Validation
Address issues like:
- Missing demographics: Exclude or impute (follow NIST AI fairness guidelines); document choices.
- Inconsistent categories: Standardize via O*NET.
- Duplicates: Decide on handling multiple applications.
- Incomplete outcomes: Track full process.
This phase consumes 20-30% of time; use tools like Python's pandas for efficiency, documenting for transparency. Platforms like EmployArmor automate ATS data pulls to streamline.
Benchmark against EEOC charge data for industry context.
Step 3: Calculate Selection Rates
Selection rate = (Selected from group) / (Total in group)
Example Calculation
For 1,000 customer service candidates:
Sample Data:
- White: 400 evaluated → 160 advanced (40%)
- Black: 250 → 50 (20%)
- Hispanic: 200 → 60 (30%)
- Asian: 150 → 75 (50%)
Sex:
- Male: 450 → 180 (40%)
- Female: 550 → 165 (30%)
Step 4: Calculate Impact Ratios
Impact ratio = (Group A's rate) / (Highest rate)
Applying the Four-Fifths Rule
Disparate impact flags if below 80% of the top rate (EEOC Uniform Guidelines).
Example:
- Highest: Asian (50%)
- Black: 20%/50% = 0.40 (40%) → Fails
- Hispanic: 30%/50% = 0.60 (60%) → Fails
- White: 40%/50% = 0.80 (80%) → Passes
Sex: Female 30%/40% (male highest) = 0.75 (75%) → Fails
Failing triggers Title VII defenses: prove job-relatedness and no alternatives. Align with Colorado's AI Act.
Step 5: Statistical Significance Testing
Test if differences are random using:
- Chi-square: For group differences
- Fisher's exact: Small samples
- Z-test: Two-group comparisons
P-value < 0.05 signals significance. Combine with four-fifths failure for evidence. Engage experts via SIOP. Use R/SPSS; keyword "AI bias statistical testing" aids internal SEO.
Step 6: Intersectional Analysis (Emerging Requirement)
Analyze overlaps like Black women, per California's AB 2930.
Example:
- White men: 45%
- White women: 38%
- Black men: 25%
- Black women: 15% (severe)
- Hispanic men: 32%
- Hispanic women: 28%
Use U.S. Census data for tailoring.
Step 7: Document Findings and Prepare Report
Required Report Elements (NYC LL144 Standard)
Include:
- Audit date
- Selection rates/impact ratios
- Sample details
- Methodology/limitations
- Independent certification
See NYC DCA resources.
Recommended Additions
- Trends vs. prior audits
- Plain-language summary
- Mitigation proposals
Add charts for engagement.
Step 8: Decide What to Do With the Results
Options for disparate impact:
Option 1: Stop Using the Tool
Quick risk reduction, but disrupts workflows.
Option 2: Modify the Tool
Collaborate on tweaks; re-audit post-changes.
Option 3: Validate Job-Relatedness
Conduct validity studies ($50K–$250K); prove under Title VII. Note EEOC v. iTutorGroup skepticism.
Option 4: Accept and Publish
High-risk; seek counsel. Consult Colorado AG AI resources.
Step 9: Publish Results (Where Required)
Disclose publicly in NYC/California via a transparency page (e.g., yourcompany.com/ai-hiring-transparency). Use accessible language, update annually, link from careers. Optimize with schema and keywords like "AI bias audit results NYC".
Sample Format
AI Hiring Tool Bias Audit Results
Tool: HireVue v8.2
Date: January 15, 2026
Period: February 2025–January 2026
Auditor: [Firm Name]
Summary: Analyzed 1,247 candidates...
[Tables]
Full report: [email]
Step 10: Establish Ongoing Monitoring
- Annual re-audits
- Quarterly reviews
- Trigger re-audits for changes
- Vendor alerts
EmployArmor provides automated dashboards.
Who Should Conduct the Audit?
In-House vs. External
Laws demand independence. External offers expertise; find via SIOP directory. Seek EEOC/SIOP credentials.
Cost Expectations
- Simple: $15K–$30K
- Moderate: $30K–$75K
- Complex: $75K–$250K+
Re-audits: 30-50% cheaper. Higher in NYC.
Common Pitfalls to Avoid
- ❌ Vendor audits without verification (EEOC standards)
- ❌ Test data use
- ❌ Skipping intersections (NIST)
- ❌ Undocumented limitations
- ❌ Unreviewed publication
How EmployArmor Simplifies Bias Audits
EmployArmor handles:
- Auditor matching
- Data automation from ATS
- Progress tracking
- Compliant publications
- Monitoring dashboards
Simplify Your Bias Audit
Connect with auditors and manage compliance seamlessly.
Start Your Audit →
Frequently Asked Questions
How often must bias audits be conducted?
Most laws require annual audits, plus re-audits for material changes. See NYC LL144 FAQ.
Can we use the same audit for multiple jurisdictions?
Yes, if it meets the strictest standards (e.g., NYC covers CA/CO). Verify with NAAG.
What if we don't have 500+ candidates in a 12-month period?
Extend time or combine roles; document. Small samples limit detection (EEOC advisories).
Do we need separate audits for each AI tool?
Yes, per tool/algorithm.
What if candidates don't provide demographic data?
Use inference or build samples longer; follow EEOC best practices.
For SEO, add this JSON-LD to your page head:
Related Resources
- Complete AI Hiring Compliance Guide 2026
- Do I Need an AI Bias Audit?
- First NYC LL144 Enforcement Actions
- 2026 AI Hiring Laws: What Changed
This content is for informational purposes only and does not constitute legal advice. Consult qualified employment law counsel for specific guidance. EmployArmor is not a law firm. For official interpretations, refer to EEOC.gov and state agencies.
(Word count: Approximately 2,250)