Loan Approval: Bias & Compliance Audit
A hands-on exercise in building a model for a regulated domain from the ground up — where compliance and fairness questions shape every decision from the first line of code, not just the final review.
The Business Problem & Why It Matters
A loan officer manually reviewing every application is slow, expensive, and inconsistent. An ML model that automates the first-pass decision solves the speed problem — but introduces three others simultaneously: financial risk (the model might approve loans that default), applicant harm (the model might deny loans to creditworthy applicants unfairly), and legal exposure (if the model's decisions correlate with protected characteristics, the lender is in Equality Act 2010 territory regardless of whether bias was "intended").
This project was a deliberate exercise in treating those three concerns as constraints from the first line of code — not as a compliance checklist appended to a model that was already built. The goal was to understand what it feels like to let regulatory framing shape modeling decisions: which features are off-limits before training begins, which correlations need to be audited before feature selection, and what "good enough" accuracy actually means when subgroup fairness is unresolved.
Financial Risk
Approving loans that default harms the lender. The model needs to be genuinely predictive, not just high-accuracy on a skewed dataset.
Applicant Harm
Denying creditworthy applicants — especially along demographic lines — is both ethically wrong and legally actionable.
Legal Exposure
The Equality Act 2010 makes it unlawful to discriminate on the basis of protected characteristics in the provision of financial services. Intent doesn't matter; disparate impact does.
Regulatory & Compliance Framing
Before any exploratory data analysis began, I set the compliance constraints. This sequencing was deliberate — the alternative (discovering a problem after building a model on prohibited features) is a worse outcome both ethically and practically.
Note: this project's compliance framing was originally built around US law (ECOA/FCRA), then revised to UK law (Equality Act 2010, Data Protection Act 2018) to better match this portfolio's target market — an example of catching and correcting a real assumption partway through a project, not getting it right on the first pass.
Equality Act 2010
In the UK, the Equality Act 2010 makes it unlawful to discriminate based on nine protected characteristics, including sex and marriage/civil partnership — both directly relevant here. Official UK government guidance on the Act gives an example almost identical to this project's scenario: a credit union refusing a woman a loan because "a woman is less likely to have a job and be able to repay the loan" — explicitly named as direct sex discrimination.
A UK-specific nuance worth noting: indirect discrimination — a neutral-looking feature that disadvantages a protected group without directly using it (the exact pattern found with Dependents and CoapplicantIncome) — is genuinely allowed under UK law if the company can show it's "a proportionate means of achieving a legitimate aim." This means retaining these features isn't automatically a compliance violation the way using Gender directly would be — but it requires a real, defensible business justification for each one, not just improved accuracy.
Enforcement is split across different bodies: the Financial Conduct Authority (FCA) regulates lending conduct but does not directly enforce the Equality Act — that's the Equality and Human Rights Commission's (EHRC) role. The Financial Ombudsman Service (FOS) handles individual complaints on a "fair and reasonable" basis but can't make a formal legal finding of a breach — that's reserved for the courts.
Gender & Married: Excluded Before Modeling
Both Gender and Married correspond to the protected characteristics of sex and marriage/civil partnership under the Equality Act 2010. Both were dropped from the feature set before any model training. This was a non-negotiable constraint, not a modeling choice — documenting it explicitly is part of the compliance trail a regulated lender would need.
Dependents & Property_Area: Proxy Flags
Dependents and Property_Area were flagged as potential proxies for protected attributes at this stage — before data exploration, not during it. Both were kept conditionally, pending a proper proxy audit. Under the Equality Act's indirect discrimination provisions, retaining them would require a documented, proportionate business justification — not just a marginal accuracy gain.
In a real production setting, these decisions would involve legal counsel and a documented model card, not just a notebook comment. The framing here is intentionally simplified to practice the reasoning, not to simulate a complete compliance process.
Reframing as an ML Problem
After setting compliance constraints, the ML framing itself is relatively straightforward: binary classification (Loan_Status: Y/N) on structured tabular data, with a mix of numeric and categorical features.
Target: Loan_Status — approved (68.7% of labeled records) or denied (31.3%). The class imbalance isn't severe enough to require resampling, but it does mean accuracy can flatter a model that just approves everything. A lazy baseline was established first.
Features available after compliance exclusions: ApplicantIncome, CoapplicantIncome, LoanAmount, Loan_Amount_Term, Credit_History, Dependents, Self_Employed, Property_Area, Education.
The Data — Including a Pre-Modeling Bias Audit
Subgroup Outcome Gaps (Before Any Model)
Before fitting any model, I checked whether approval rates already differed by protected group in the raw data. Two findings stood out — with meaningfully different levels of confidence:
Gender Gap: 2.4 pts
Female approval rate vs. male was 2.4 percentage points lower. But the female sample was small (n≈100 vs. n≈370 male), making this gap statistically inconclusive. It's not evidence of no problem — it's evidence we can't conclude either way with this sample size.
Married Gap: 8.7 pts
Unmarried applicants were approved 8.7 pts less often than married ones. The sample here was larger and the gap bigger — a more trustworthy signal that something worth investigating is present in the data, even though Married is already excluded from the model.
Proxy Audit Results
Excluding Gender and Married directly doesn't guarantee fairness — features that correlate strongly with protected attributes can re-introduce the same signal through the back door. I ran a proxy check on the remaining candidates:
| Feature Pair | Correlation Signal | Verdict |
|---|---|---|
| Dependents → Married | 42 pts gap | Strong proxy — flag for review |
| Dependents → Gender | 17.5 pts gap | Moderate proxy — flag for review |
| Property_Area → Gender | 10 pts gap | Weaker, borderline |
| CoapplicantIncome → Gender | p = 0.0000003 | Very strong proxy — flag |
| ApplicantIncome → Gender | p = 0.068 | Borderline — watch |
CoapplicantIncome and Dependents were kept in the model (removing every correlated feature would collapse predictive power), but their correlation with protected attributes is documented here and would require explicit justification in a compliance review.
The Dependents Flip-Flop
Dependents caused three reversals before I reached for a proper statistical test. First pass: it looked predictive, keep it. Second pass: it correlated with Married (42 pts gap), flag it for exclusion. Third pass: I excluded it, model performance dropped noticeably, so I reconsidered. Fourth pass (the right move): ran a chi-square test. The test gave p=0.21 against Gender — not a statistically significant proxy for the gender axis specifically, which is the axis that matters most given the Equality Act constraint.
The flip-flop happened because I was making the decision by intuition ("this feels like it might correlate") rather than evidence. The lesson isn't that Dependents is safe — the Married correlation is still real — it's that the sequence should have been: define the test, run the test, then decide. Not: decide, then look for evidence to confirm.
I didn't catch this on the first pass. The proxy audit for Gender came after I'd already made a preliminary inclusion decision. That order was wrong.
Missing Data: Cleaning Decisions
| Column | Fill Method | Missingness p-value | Flag Kept? |
|---|---|---|---|
| LoanAmount | Median | p=0.67 (not sig.) | No — clean impute |
| Loan_Amount_Term | Mode | p=0.81 (not sig.) | No — clean impute |
| Credit_History | Mode | p<0.001 (sig.) | Yes — Had_Credit_History flag added |
| Dependents | Mode | p=0.21 (not sig.) | No — clean impute |
| Self_Employed | Mode | p=0.43 (not sig.) | No — clean impute |
The Credit_History flag was the only missingness pattern that was itself statistically significant — whether someone had credit history recorded at all was predictive of the outcome, so a binary flag (Had_Credit_History) was added alongside the imputed value.
Why Accuracy Can Hide Unfairness
Before showing any model results, it's worth building the intuition for why overall accuracy is an insufficient fairness check. Here's a concrete worked example:
The 200-Person Problem
Imagine a model tested on 200 people: 100 women, 100 men. Overall accuracy: 83%. Sounds fine.
Men (n=100)
92 correct → 92% accuracy
Women (n=100)
74 correct → 74% accuracy
83% overall, but the model is 18 percentage points worse for women than men. The aggregate number hid a real disparity because the two groups averaged together. This is why subgroup accuracy checks are not optional in a regulated domain — they're the actual test of whether the model is performing fairly, not just performing well.
This explainer frames what the subgroup results below are actually measuring: not a bonus metric, but a first-class outcome that determines whether the model is fit for use.
Model Choice — Through a Compliance Lens
In a regulated domain, interpretability isn't a nice-to-have — it's close to a legal requirement. The UK doesn't have a direct equivalent to the US's adverse-action-notice requirement. The closer match is the Data Protection Act 2018 / UK GDPR's provisions on automated decision-making: a right not to be subject to a solely-automated decision with a significant effect, and a right to "meaningful information about the logic involved." This is genuinely weaker and less specific than the US requirement; recent academic review concludes the UK lacks a clear statutory right to explanation for credit decisions.
This actually strengthens the case for an interpretable model rather than weakening it: since the legal bar ("meaningful information") is loosely defined, building on a model with real, named, explainable coefficients makes it easier to satisfy in good faith than relying on a black-box model and a post-hoc explanation technique of uncertain adequacy. A Random Forest with hundreds of trees can't provide that in a human-readable form. Logistic Regression can: each feature has a coefficient, and a denial reason can be traced to the top contributing features.
This compliance framing made Logistic Regression the preferred model from the start — before any numbers were compared. The fact that it also outperformed Random Forest on the test set made the decision unambiguous.
Model Comparison
| Model | Validation Accuracy | Test Accuracy | Note |
|---|---|---|---|
| Logistic Regression (tuned C=1.0) | 83.7% | 82.9% | Final model |
| Random Forest (tuned max_depth=5) | 84.2% | 82.1% | Stronger on val, weaker on test |
Unlike the Titanic project where both models tied exactly on test (81.0%/81.0%), here Logistic Regression won outright on the honest evaluation — 82.9% vs. 82.1%. The interpretability argument and the performance argument pointed in the same direction.
The Experiment
Split
60 / 20 / 20
Train / Validation / Test — stratified to preserve the 68.7% approval rate across all three sets
Lazy Baseline
68.3%
Accuracy from always predicting 'Approved' — the floor every real model has to beat
Final Test Accuracy
82.9%
+14.6 pts over lazy baseline, on data the model never saw during tuning
Convergence Warning & Feature Scaling
The initial Logistic Regression run returned a convergence warning — the solver didn't reach a stable solution within the default iteration limit. Root cause: the numeric features (ApplicantIncome, LoanAmount) were on very different scales. After applying StandardScaler, the warning disappeared and the model reached a stable solution. This is a small but real technical judgment call: ignoring a convergence warning and treating the output as trustworthy would have been wrong.
Results — Including a Subgroup Breakdown
Confusion Matrix
18
True Negatives (correct denials)
21
False Positives (wrongly approved)
0
False Negatives (wrongly denied)
84
True Positives (correct approvals)
Zero false negatives means the model never wrongly denied someone who should have been approved — a strong result for applicant harm. But 21 false positives (loans approved that may default) represents financial risk. The model leans heavily toward approval, which is the less harmful direction for applicants but not a free pass from a credit-risk perspective.
Gender Subgroup Accuracy
| Subgroup | n | Test Accuracy |
|---|---|---|
| Female | 28 | 75.0% |
| Male | 93 | 84.9% |
Chi-Square Result: p = 0.350
A chi-square test on the gender accuracy gap returned p=0.350 — not statistically significant.
This does NOT mean the model is fair. It means we cannot prove it is unfair at this sample size. These are not the same thing.
With only 28 female test examples, a 9.9 percentage point accuracy gap is statistically indistinguishable from random noise — not because the gap is small, but because there isn't enough data to distinguish a real disparity from chance variation. "Not statistically significant" is a statement about evidence quality, not about the underlying reality. A larger test set might return a very different p-value on the same 9.9 pt gap.
Go / No-Go: Would I Sign Off for Production?
No. Not as-is.
The model's overall accuracy (82.9%) is real and stable — it clears the lazy baseline by nearly 15 points and held up honestly on untouched test data. The zero false negatives on applicant harm is a genuinely good result. But the subgroup fairness story is unresolved, and in a regulated domain, "unresolved" is not the same as "acceptable." A 9.9 percentage point accuracy gap between female and male applicants, on a test set with only 28 women, tells us almost nothing about whether the model is treating those groups fairly. We don't have enough evidence to conclude fairness — and that absence of evidence is the problem.
A Go recommendation here would require: a larger, more balanced test set; a formal fairness metric (four-fifths rule or equivalent); and a documented legal review of the proxy variables still in the feature set. Without those, any Go sign-off would be premature.
For this exercise — a practice project, not a production system — the honest conclusion is: the model works well enough to document as a learning artifact, but not to recommend for deployment. The value of the exercise was learning to reach that specific, bounded conclusion rather than rounding it into something cleaner.
What I'd Do Next
The gap between this exercise and production-grade practice is real and worth naming explicitly, not glossing over. These aren't aspirational improvements — they're the minimum bar for responsible deployment in a regulated domain:
Formal fairness metrics
The four-fifths (80%) rule is a standard EEOC threshold for adverse impact. A proper audit would compute this for each protected class, not just eyeball an accuracy gap.
Fairlearn or AIF360
Specialist fairness toolkits that run calibrated subgroup analysis, compute fairness metrics across multiple definitions simultaneously, and can apply in-processing or post-processing fairness interventions.
Power analysis before the project
The female test set (n=28) was too small to detect a real disparity. A power analysis at the start would have flagged this sample size as insufficient for any meaningful subgroup test — and prompted either more data collection or a different experimental design.
Actual legal review
No proxy-variable decision made in a notebook substitutes for a legal opinion. In production, the compliance documentation and feature decisions would go to counsel, not just the model card.
A documented model card
The decisions, limitations, and fairness findings from this notebook would need to exist as a formal model card before any deployment — the artifact that a regulator, auditor, or downstream team would actually review.
Dependents capping / training-serving skew
The Dependents column had unusual values in a few rows. In a real deployment, outlier capping and explicit documentation of the training distribution would be required to prevent training-serving skew from producing unexpected behavior in production.
Reflection
The most instructive part of this project wasn't the model — it was the three reversals on Dependents. I made an inclusion decision by intuition, then reversed it when a correlation surfaced, then reversed again when performance dropped, and only on the fourth pass did I stop and run a proper statistical test. The test gave a defensible answer (p=0.21, not a significant Gender proxy) that I could have had from the start if I'd led with the right question instead of trying to reason my way to a conclusion from gut feel.
There were also two proxy checks I missed the first time. The Gender income proxy (CoapplicantIncome, p=0.0000003) and the Dependents-vs-Gender gap (17.5 pts) were both caught late — after I'd already made preliminary feature decisions. The right sequencing was: set compliance constraints, run the full proxy audit, then do feature selection. I did those in the wrong order and had to loop back.
What felt most different about this project compared to a standard ML exercise was how the compliance framing changed the sequencing of the whole thing. In a typical accuracy-first workflow, you build the model, then check whether it has problems. Here, the regulatory constraints came first: which features are off-limits, which ones might be proxies, what test the model has to pass before it can be used. That reordering didn't make the modeling harder — it made the modeling decisions more defensible, because each one had a reason that could be explained to someone outside the notebook.
Section 9 deliberately doesn't round the fairness finding into a clean answer. A 9.9 percentage point accuracy gap on n=28 is genuinely ambiguous — statistically inconclusive, practically concerning. The honest response to that ambiguity is to name it, not to paper over it by pointing at the p-value and declaring success. In a regulated domain, a No-Go recommendation on unresolved fairness evidence isn't a failure of the model. It's the right call.
"The compliance framing didn't constrain the modeling — it clarified it. Every feature decision had to be defensible, not just technically justifiable. That's a different and harder standard, and it's the right one for a model that affects real credit decisions."
Let's Connect
I am actively seeking Junior AI PM / Technical PM roles at companies building AI-powered products in media, events, e-commerce, or consumer applications. Let's connect if you're hiring or want to discuss AI product strategy.
Contact Info
© 2025 Ogbebor Osaheni. Built with Next.js, React, and Tailwind CSS.