Titanic Survival Prediction
A guided, hands-on ML fundamentals exercise built to strengthen the technical judgment behind AI product decisions — not a shipped product, but a documented practice case study in data cleaning, feature engineering, model comparison, and honest evaluation.
Executive Summary
Working from the classic Kaggle Titanic dataset, I ran a complete, disciplined ML workflow end-to-end: cleaned and investigated messy real-world data (including a deliberate check for confounding), engineered a new feature from raw text, tuned two different model types against a held-out validation set, and evaluated the final choice exactly once against an untouched test set. The goal wasn't the dataset — it was building the fluency to ask sharp, well-grounded questions of a data science team as an AI PM, rather than taking model performance claims at face value.
Final Test Accuracy
Honest, one-time evaluation on data the model never saw during tuning
Over Lazy Baseline
vs. 61.5% from always predicting the majority class
Models Compared
Logistic Regression vs. Random Forest, tied on test despite a validation gap
From Feature Engineering
Extracting passenger Title from free-text Name — a gain that held up on test
Problem Statement
AI Product Managers don't need to write model training code, but they do need enough technical fluency to evaluate what a data science team hands them: is 82% accuracy actually good, or is it barely beating a coin flip? Did that validation improvement survive contact with real, unseen data, or was it an illusion of overfitting? Is the model's error pattern acceptable given who it affects? Without hands-on practice, these questions stay abstract.
This project was built specifically to close that gap — using a well-understood, low-stakes dataset (Titanic passenger records) as a safe space to practice the exact judgment calls that come up in real production ML work: handling missing data, choosing what to optimize for, comparing models honestly, and knowing when a performance gain is real versus an artifact of tuning against the wrong data.
Every step below was done manually, from scratch, in a live notebook — including several genuine mistakes and corrections along the way (an early file-path error, an initial skip of proper baseline-setting, a redundant single-model run before the tuning loop) — because the reasoning behind a decision, including the wrong turns, is the actual skill being demonstrated.
"The most valuable finding wasn't a model metric — it was discovering that a feature (Has_Cabin) which looked strongly predictive of survival was actually a confound: a proxy for passenger class, not an independent signal. Catching that distinction before trusting the feature is a rehearsal for exactly the kind of fairness and bias question a real AI PM has to raise before a model ships."
Solution Overview
Rather than jumping straight to a finished model, I treated every step as a deliberate, defensible decision: what to do with missing data, which metric to trust, which hyperparameter to prefer among ties, and which model to ship given identical test performance. Each choice is documented with its reasoning, not just its result.
Confound Investigation
Found that 'Has_Cabin' predicted survival (66.7% vs. 29.9%), then tested and confirmed it was really a proxy for Pclass (81.5% of 1st class had known cabins vs. 2.4% of 3rd class) — a direct rehearsal for spotting proxy-variable bias.
Honest Model Comparison
Tuned Logistic Regression and Random Forest independently against validation data. Random Forest looked stronger on validation (+2.3 pts) but the two models tied exactly on the untouched test set — the validation lead didn't generalize.
Real Feature Engineering
Extracted passenger Title (Mr/Miss/Mrs/Master/Other) from the free-text Name field. Unlike higher model complexity, this gain held up on the test set (+1.7 pts) — a concrete example of real signal vs. an overfitting artifact.
Data & Methodology
Data Dictionary
| Feature | Type | Description | Source |
|---|---|---|---|
| Pclass | Ordinal (1/2/3) | Ticket class, proxy for socio-economic status | Kaggle Titanic dataset |
| Sex | Binary | Encoded male=0, female=1 | Kaggle Titanic dataset |
| Age | Numeric | 177 missing values (20%), imputed with column median | Kaggle Titanic dataset |
| SibSp / Parch | Numeric | Siblings/spouses and parents/children aboard | Kaggle Titanic dataset |
| Fare | Numeric | Passenger fare paid | Kaggle Titanic dataset |
| Has_Cabin | Engineered, Binary | 1 if Cabin was recorded; investigated as a Pclass proxy rather than dropped outright | Derived from Cabin (77% missing) |
| Embarked_Q / Embarked_S | Engineered, One-hot | Port of embarkation, one-hot encoded to avoid implying false ordering | Derived from Embarked (2 missing, filled with mode) |
| Title_Miss / Title_Mr / Title_Mrs / Title_Other | Engineered, One-hot | Extracted from free-text Name; rare titles grouped into 'Other' | Derived from Name (previously unused) |
Methodology
891 labeled passengers were split 60/20/20 into train/validation/test using stratified sampling, preserving the ~38% survival rate across all three sets. Logistic Regression and Random Forest were each tuned against the validation set only (C for Logistic Regression; max_depth for Random Forest), and the test set was touched exactly once, after all tuning decisions were locked in, to produce an honest final score.
Validation Approach
- •Stratified 60/20/20 train/validation/test split to preserve class balance across all subsets
- •Lazy baseline (always predict majority class) calculated first, to give every subsequent accuracy number a meaningful floor to beat
- •Hyperparameter tuning restricted entirely to the validation set — the test set was never used to select a setting
- •Test set evaluated exactly once, after final model and hyperparameters were chosen, to avoid test-set contamination
- •Confusion matrix built on final test predictions to separate error types (false positives vs. false negatives) rather than relying on accuracy alone
Try the Model Yourself
Adjust the passenger details below to see what my trained Logistic Regression model would have predicted — computed live, using its real coefficients.
In the training data, this was largely a proxy for passenger class rather than an independent factor — see the Data section above.
9% predicted chance of survival
This runs your actual trained model's coefficients directly in your browser — no server involved.
Proof of Impact
81.0%
Final test accuracy — Logistic Regression + Title feature, evaluated once on untouched data
Results Comparison
| Metric | Before | After | Change |
|---|---|---|---|
| Lazy baseline (always predict majority class) | — | 61.5% | floor |
| Logistic Regression, default settings | — | 82.0% (val) | +20.5 pts vs. baseline |
| Logistic Regression, tuned (C=0.1) | 82.0% (val) | 83.1% (val) → 79.3% (test) | +1.1 pt val |
| Random Forest, tuned (max_depth=5) | 83.1% (val, LR) | 85.4% (val) → 79.3% (test) | tied LR on test |
| Logistic Regression + Title feature (final) | 79.3% (test) | 81.0% (test) | +1.7 pts, held on test |
Key Insights
Random Forest's validation advantage over Logistic Regression (85.4% vs. 83.1%) completely disappeared on the honest test set — both models scored identically at 79.3%. Random Forest's larger validation-to-test drop (-6.1 pts vs. -3.8 pts) is consistent with more flexible models being more prone to fitting quirks of a specific validation split.
The final confusion matrix (97 true negatives, 13 false positives, 21 false negatives, 48 true positives) showed nearly twice as many false negatives as false positives — the model was more likely to miss a survivor than to falsely predict survival, a pattern worth flagging in any higher-stakes real-world analogue (e.g., a triage or eligibility model).
Since both tuned models tied on test accuracy, Logistic Regression was selected for the final model on interpretability grounds — added model complexity earned no measurable performance benefit in this case.
Would I Ship This?
As a standalone accuracy number, 81% sounds shippable — it clears the lazy baseline by nearly 20 points and holds up on data the model never saw during tuning. But accuracy alone isn't the whole story. The confusion matrix shows the model makes almost twice as many false negatives (21) as false positives (13) — it is more likely to miss a real survivor than to falsely flag one.
In a real-world analogue with the same error shape — a hospital triage model, or an eligibility screener deciding who gets flagged for urgent review — that imbalance would be disqualifying on its own. Under-flagging the group you most need to catch is a worse failure mode than over-flagging, and no amount of overall accuracy makes up for it. I would not ship this model as-is into a high-stakes decision. I would ship it as a transparent, well-understood baseline — one explicitly optimized next for recall on the positive class, re-evaluated with a cost-weighted metric rather than plain accuracy, and reviewed against real subgroup performance (not just Sex and Pclass in aggregate) before any higher-stakes use.
For this specific dataset — a low-stakes, historical practice exercise — the honest answer is: yes, publish it as a documented learning artifact. The exercise here was building the judgment to make that distinction correctly, not proving the Titanic model itself is production-ready.
Learnings & Reflections
What Went Well
- •Establishing a lazy baseline (61.5%) before trusting any accuracy number made every later result interpretable — an 82% accuracy figure means very little on its own without that floor for comparison.
- •Investigating why Has_Cabin correlated with survival, rather than accepting the correlation at face value, surfaced a real confound (Pclass) — good practice for spotting proxy variables before they mislead a fairness or explainability review.
- •Re-running hyperparameter tuning after adding the Title feature (rather than assuming the old best setting still applied) revealed the optimal C actually shifted — a reminder that tuning decisions are tied to a specific feature set, not permanent.
Challenges Faced
- •Early on, a model was trained and validated without first establishing what accuracy would look like from doing nothing intelligent at all (the lazy baseline) — a step later corrected, but a useful reminder that context should be established before results are judged.
- •Distinguishing a real, generalizable model improvement (Title feature engineering) from a fake one (Random Forest's validation-only lead) required actually running the honest test evaluation for both — the two looked similar at the validation stage alone.
- •Understanding False Positive vs. False Negative terminology took several passes with concrete before/after examples (spam filters, medical triage) rather than the formal definition alone — a reminder that plain-language framing matters as much as technical accuracy when explaining model behavior to non-technical stakeholders.
What I'd Do Differently
- •Establish the lazy baseline as the very first step in any future model evaluation, before training the first real model, rather than adding it in retrospectively.
- •Extract and test Title (and similar free-text-derived features) earlier in the process, since it was the single highest-leverage, cheapest improvement in the entire project — more valuable than any hyperparameter tuning attempted.
- •Build the confusion-matrix error breakdown alongside the very first model, not only at the end, so that error-type tradeoffs (false positives vs. false negatives) inform earlier modeling decisions, not just the final write-up.
"The technical mechanics of a model — accuracy, tuning, cross-validation — are learnable in an afternoon. The harder, more valuable skill is the discipline to keep asking 'compared to what?' and 'does this improvement actually hold up on data it wasn't tuned against?' at every step. That discipline, not the specific Titanic result, is what transfers directly to evaluating a real production model."
Let's Connect
I am actively seeking Junior AI PM / Technical PM roles at companies building AI-powered products in media, events, e-commerce, or consumer applications. Let's connect if you're hiring or want to discuss AI product strategy.
Contact Info
© 2025 Ogbebor Osaheni. Built with Next.js, React, and Tailwind CSS.