Every HR decision an algorithm makes is, eventually, a decision about a person. A rejected resume is a rejected job-seeker. A low performance score is a missed promotion. A flight-risk flag is a manager pulled into an awkward conversation. When the system that produced that decision can’t show its workings, the cost falls on the employee, and the legal and reputational liability falls on the employer.
At its core, the black-box problem is the gap between what an AI system decides and what anyone can explain about how it arrived at that decision. The user sees the verdict. The logic that produced it stays out of reach.
In Indian HR, that gap runs especially wide right now. According to the Boston Consulting Group’s AI at Work 2025 report, 92% of Indian employees use generative AI regularly, making India the world’s most AI-saturated workforce. Indeed’s 2025 Smarter Hiring Report found that only 5% of Indian employers aren’t using AI-powered hiring tools today. In other words, the AI infrastructure is in.
The trust hasn’t kept pace. EY’s 2025 Work Reimagined India data shows that 83% of Indian employees flag explainability as a concern, the single biggest perception gap between employers and employees in the survey. Most AI tools in Indian HR still operate as opaque systems; HR teams can’t always justify the decisions these tools surface, and regulators are starting to catch up.
What the Black-Box Problem Actually Is
A black-box AI system is one where the relationship between inputs and outputs can’t be inspected. The model takes in data, runs calculations across millions of parameters, and returns a result. You see the result. You don’t see the reasoning.
In simpler statistical models, you could point to a coefficient and say, “candidates with 5 years of experience are weighted more heavily than those with 2.” In modern machine learning, especially deep learning and large language models, interpretability collapses. The decision emerges from a web of weighted connections that no human directly designed.
For HR, that creates problems on multiple fronts. There’s a technical one: the people running the tool often can’t articulate what it’s optimising for. There is an operational one too, where a candidate or employee asks why, and no one has a clean answer. And then there is a legal one, where, under emerging Indian and global rules, “the algorithm decided” is no longer a defensible position.
A useful working definition comes from India’s own regulatory language. The India AI Governance Guidelines push for explainable AI (XAI) designs under the principle of “Understandability”, which means systems should be built so their decisions can be traced, examined, and explained to the people they affect. If your current AI stack can’t do that, it’s a black box, regardless of what the vendor promises.
Why Explainability Hits Harder in HR Than Anywhere Else
A black-box recommender system on an OTT platform mis-suggests a movie. A black-box hiring algorithm misjudges a candidate’s career. The asymmetry is the entire point.
HR sits on some of the most consequential decision categories in any organisation: who gets in, who moves up, and what they get paid. Each one carries legal, ethical, and reputational weight that other functions don’t share.
Hiring Fairness
Resume screeners, interview scoring tools, and candidate-ranking engines now sit between most Indian applicants and the recruiter who’d otherwise read their CV. A recent investigation found that 40% of AI-driven rejections in India disproportionately affected women and marginalised groups, signalling systemic flaws in the training data these systems learn from.
When the model can’t explain why it scored Candidate A higher than Candidate B, the organisation has no way to check whether the scoring was based on skills or on something it shouldn’t have been weighing at all.
Performance Evaluations
AI is increasingly used to score productivity, flag underperformance, and generate review summaries. An opaque score is hard to contest, hard to coach against, and easy to weaponise in disputes. Employees who can’t see what’s being measured can’t improve it.
Promotions and Pay
Models that predict promotion-readiness or recommend pay bands have become standard in larger Indian organisations. When these recommendations harden into decisions without explanation, you’ve effectively automated a career-shaping moment with no audit trail.
The deeper issue: If an algorithm makes the call and no one can show how, the employer carries the burden of proving it didn’t discriminate. That’s a hard burden to discharge when the system is, by design, unexplainable.
What Can Actually Go Wrong
The risks of running opaque AI in HR aren’t theoretical. They’ve already played out in well-documented cases and are quietly reshaping the legal exposure of Indian companies.
A useful starting point is the most-cited case in this space. Amazon began developing an automated hiring system in 2014 to rank job seekers with one to five stars, but scrapped the project after discovering it had developed a preference for male candidates in technical roles.
The reason came down to training data. The tool was trained on 10 years of resumes, and because tech is a male-dominated industry, the system was unintentionally taught to choose male candidates over female ones. It penalised resumes containing the word “women’s” or the names of certain all-women colleges.
Amazon couldn’t fix the issue because the bias was woven through layers of weighted features that no one could fully untangle. They shut the project down.
That case became the textbook reference point for the black-box problem in HR, and a reminder that opaque systems can absorb bias faster than anyone can audit it out.
| Risk | What It Looks Like in Practice | Why Explainability Helps |
| Hidden discrimination | Models trained on biased historical data replicate that bias beneath a layer of mathematical neutrality. Female candidates, older workers, or candidates from non-Tier-1 colleges get systematically lower scores. | Feature importance analysis surfaces which inputs are driving decisions, allowing teams to spot and correct proxy discrimination. |
| Legal exposure | An employee or candidate challenges a decision under the Equal Remuneration Act, Code on Wages, or the DPDP Act. The company can’t produce evidence of how the decision was made. | Audit logs and decision rationale make the case defensible, or surface a genuine problem before it goes to court. |
| Employee distrust | Workers stop engaging honestly with AI-driven tools (pulse surveys, performance check-ins, internal mobility platforms) because they don’t trust how their data is being used. | Visible decision logic and human review checkpoints rebuild engagement. |
| Reputation damage | A bias incident becomes public. Indian media picks it up. Recruitment pipeline dries up. Glassdoor scores drop. | Explainable systems generate the documentation needed to respond credibly and reduce the odds of the incident occurring. |
There’s also a quieter cost that doesn’t make headlines. When HR teams don’t trust the AI they’re running, they double-check everything manually, which means the efficiency gains the tool was bought for never fully materialise.
The pattern shows up in the EY Work Reimagined India data: HR teams still double-check AI outputs even where adoption is officially complete. Opaque AI doesn’t just create risk. It quietly destroys ROI.
What Explainable AI Looks Like in Practice
Explainable AI isn’t a single product or feature. It’s a set of capabilities that, together, let a human understand why a model produced a specific output. A few of these matter especially in HR contexts.
Decision Rationale Visibility
For every individual decision the model makes, the system should be able to surface the top factors that drove it. If a candidate was ranked 7th, the system should be able to say which features pushed them up and which pushed them down.
Local explanation techniques like SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic Explanations) provide candidate-level feedback by attributing contributions to specific features or tokens. These aren’t experimental anymore. They’re standard tooling for any vendor doing this seriously.
Feature Importance Analysis
At the model level, HR teams need to know what the algorithm weighs most heavily across all decisions. If “years of experience” is the top feature, that’s defensible. If “graduating institution tier” is in the top three, the team has a problem worth investigating. Hyring’s HR guide notes that if zip code, university name, or years of experience are top features, teams should investigate whether they’re serving as proxies for protected characteristics.
Human Review Checkpoints
No algorithm should be the final word on a hiring, performance, or pay decision. The model surfaces the analysis. A human takes the call. The point isn’t to slow hiring down but to keep accountability with a person who can be questioned, trained, and held responsible. Under the EU AI Act framework, human oversight is treated as a baseline requirement, not an optional feature, for AI in HR contexts.
A practical way to think about it: if you can’t sit across from a rejected candidate and walk them through the specific factors that drove their score, the system isn’t explainable enough.
How Indian HR Leaders Can Evaluate Explainability
Most HR leaders don’t build AI models. They buy them. Which means explainability is, in practice, a vendor question, and the evaluation moves through distinct phases: what to test before signing, what to require in the contract, and what to monitor after the system goes live.
Before Signing
The test is whether the vendor can actually show its working. A good one should walk a buyer through how the model arrives at a decision and what it weighs to get there. The questions worth running them through include:
- Can you surface the top features driving decisions, both globally and for any individual case?
- How do you handle proxy variables for protected characteristics like gender, caste, age, or region?
- Where was the training data sourced, and what’s its demographic composition?
- When was your last bias audit, and how often is it run?
- What’s your incident response process when bias is detected after deployment?
A vendor with a real plan for the day something goes wrong has usually thought hardest about explainability. A vendor who deflects on the data-provenance or incident-response questions is selling a black box with a friendlier UI.
At the Contract Stage
The documentation needs to be a deliverable, not a marketing claim. Any vendor selling AI for hiring, performance, or pay decisions in 2026 should be able to produce three artefacts on request:
- A model card: a short technical summary of what the model does, its training data, and its known limitations
- A bias audit report from the last 12 months
- A data flow diagram showing where employee or candidate data moves through the system
If these aren’t delivered on request, treat the gap as a red flag, not a minor inconvenience.
After Deployment
The audit trail is what protects the organisation when something goes wrong. Every AI-influenced decision should generate a record that captures:
- Timestamp and model version used
- The inputs the model saw
- The top features driving the output
- The human reviewer’s action and any override reasoning
High-risk AI systems must log decisions, document explanation methodology, and record feature importance for audits. Treat this as the minimum standard, not the ceiling.
Building Trustworthy AI Systems in HR
Treating explainability as a procurement checkbox is a recipe for missing the point. The companies getting this right are building it into governance, not bolting it on after deployment.
The starting move is an AI governance council. This isn’t another committee that meets quarterly to nod at slide decks. It’s a small, cross-functional group, typically the CHRO, CTO or CIO, a legal lead, and a data protection officer, that owns the AI inventory for HR. They sign off on new tools, review bias audits, and sit on the incident response team if something goes wrong.
Sitting alongside that council, there has to be a written responsible AI policy. The exact wording matters less than the principles being concrete and enforceable. A workable policy covers:
- Which decisions can be automated and which require a human in the loop
- What documentation must each AI tool produce
- How and when bias audits happen
- What employees and candidates are told about AI in their experience
- The escalation path for contested decisions
The piece most organisations underplay is communication. Most Indian employees haven’t been told, in plain language, where AI is showing up in their work life, and that information vacuum is what’s fuelling the trust gap.
A short, honest disclosure (“we use AI to help screen resumes, here’s what it considers, here’s what a human reviews”) does more to build confidence than any vendor demo, as TPB’s earlier coverage of why Indian employees still don’t trust HR AI lays out in detail.
Infrastructure isn’t the constraint it once was. Indian HR tech firms like Darwinbox, Keka, PeopleStrong, and Leena AI have started building governance and explainability features directly into their platforms, and the broader Indian HR tech ecosystem leveraging AI has matured fast enough that “we couldn’t find a vendor that supports this” is a harder excuse to make than it was 18 months ago.
In the End…
The black-box problem isn’t really a technology problem. It’s a trust problem dressed in technical language. And in HR, where every decision the system makes affects someone’s livelihood, trust isn’t a soft metric. It’s the only thing that keeps the function functioning.
India’s AI adoption story is genuinely impressive. The infrastructure is in place, the workforce is engaged, and the regulatory framework is more thoughtful than that of most major economies. But the next chapter isn’t about adoption. It’s about whether the tools we’ve deployed at speed can be defended at depth, in a court, in a press cycle, in a one-on-one with a rejected candidate who deserves an answer.
The HR leaders who’ll come out ahead aren’t the ones with the most AI. They’re the ones who can show their working. Explainability is the new minimum, and the cost of treating it as optional is rising faster than most teams realise.
FAQs
What is the black-box problem in HR AI?
The black-box problem is the gap between what an AI system decides and what anyone can explain about how it reached that decision. In HR, the tool surfaces a verdict, such as a candidate ranking or a flight-risk flag, but the reasoning behind it stays hidden. That matters because each decision affects a person’s livelihood, and the employer carries the liability when the logic can’t be shown.
Why does explainable AI matter more in HR than in other functions?
HR controls some of the most consequential decisions in any organisation: who gets hired, who gets promoted, and what people are paid. Each carries legal, ethical, and reputational weight that a movie recommender or ad-targeting tool doesn’t. When an HR algorithm can’t explain its scoring, the company can’t verify whether decisions were based on skills or on something it shouldn’t be weighing at all.
What does India’s regulatory framework say about explainable AI?
The India AI Governance Guidelines push for explainable AI (XAI) under the principle of “Understandability,” meaning systems should be built so their decisions can be traced, examined, and explained to the people they affect. Decisions can also be challenged under the DPDP Act, the Code on Wages, and the Equal Remuneration Act, where “the algorithm decided” is no longer a defensible position.
How can HR leaders evaluate whether an AI vendor is explainable?
Test it across three stages. Before signing, ask whether the vendor can surface the top features driving any individual decision, how it handles proxy variables for protected characteristics, and where its training data came from. At the contract stage, require a model card, a bias audit from the last 12 months, and a data flow diagram. After deployment, ensure every AI-influenced decision generates an audit log with timestamp, inputs, top features, and the human reviewer’s action.
What is the most common real-world example of the black-box problem?
Amazon’s scrapped hiring tool is the textbook case. Built from 2014 to rank candidates one to five stars, it was trained on a decade of resumes from a male-dominated industry and learned to penalise resumes containing the word “women’s” or the names of certain all-women colleges. Amazon couldn’t isolate the bias because it was woven through layers of weighted features, so the project was shut down.
