Part 5: Interactive Notebook - Exploration & Preprocessing

Open your notebook.ipynb file in VS Code and run the following code cells.

5.1 Understanding Exploratory Data Analysis (EDA) & Preprocessing

Before building a machine learning model, we must load and understand our dataset. This is called Exploratory Data Analysis (EDA). EDA helps us discover missing values, identify column structures, and locate anomalies (like outliers or incorrect values). Preprocessing is the step where we clean up these anomalies and prepare the data so it fits perfectly into our machine learning models.

---

[JUPYTER CELL 1] Ingesting Datasets

Let’s import our libraries, load the raw CSV files, and inspect their dimensions:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load records
app_df = pd.read_csv('data/application_record.csv')
credit_df = pd.read_csv('data/credit_record.csv')

print(f"Application Records: {app_df.shape}")
print(f"Credit History Records: {credit_df.shape}")

• Why we do this: This verifies that the files are placed correctly in the data/ directory and outputs the row and column dimensions.
• What to look for: Check if both files loaded successfully without errors. You should see thousands of application records.

---

[JUPYTER CELL 2] Label Engineering & Merging

We calculate the default risk for each applicant and join the tables together:

# Label overdue status '2', '3', '4', or '5' (60+ days overdue) as risky (1)
credit_df['is_risky'] = credit_df['STATUS'].isin(['2', '3', '4', '5']).astype(int)

# Group by ID to find if they defaulted at least once in their history
user_target = credit_df.groupby('ID')['is_risky'].max().reset_index()
user_target.rename(columns={'is_risky': 'target'}, inplace=True)

# Merge datasets on the unique ID column
df = pd.merge(app_df, user_target, on='ID', how='inner')
print(f"Merged Dataset Shape: {df.shape}")
print("Target Distribution:")
print(df['target'].value_counts(normalize=True) * 100)

• Why we do this:
  • We group the credit behavior records by user ID and find if they had a default history (is_risky == 1).
  • Merging this target back with application_record gives us our final labeled training dataset.
• What to look for: The target distribution prints out. You will notice that a huge majority of applicants are labeled 0 (Safe). This is called class imbalance and is something we must configure our machine learning model to handle.

---

5.2 What is Class Imbalance?

In our dataset, over 98% of applicants are labeled 0 (Safe/Approved), while less than 2% are labeled 1 (Risky/Rejected). This is a severe Class Imbalance. If we trained a model on this directly without adjustments, the model could simply guess 0 for every single applicant and achieve 98% accuracy, while failing to detect the actual risky applicants. We will resolve this during model training by using stratified splits and class weights.

---

[JUPYTER CELL 3] Preprocessing and Fixing Anomalies

We clean up missing values, handle offsets, and drop unneeded identifiers:

# Resolve null occupations
df['OCCUPATION_TYPE'] = df['OCCUPATION_TYPE'].fillna('Unknown')

# Convert birth offset to positive age in years
df['AGE'] = (df['DAYS_BIRTH'] / -365.25).astype(int)

# Convert employment offset to years. Positive values are marked as unemployed (0 years).
df['YEARS_EMPLOYED'] = df['DAYS_EMPLOYED'].apply(lambda x: 0 if x > 0 else x / -365.25)
df['UNEMPLOYED'] = (df['DAYS_EMPLOYED'] > 0).astype(int)

# Drop redundant raw columns
df_clean = df.drop(columns=['ID', 'DAYS_BIRTH', 'DAYS_EMPLOYED', 'FLAG_MOBIL'])
print(df_clean.head())

• Why we do this:
  • Missing occupation fields are filled with 'Unknown' to avoid dropping rows.
  • Age and employment length are converted from negative days into readable years.
  • The positive values in DAYS_EMPLOYED indicate unemployment, so we create a binary UNEMPLOYED indicator flag.

---