Part 5: Interactive Notebook - Exploration & Preprocessing

Open your notebook.ipynb file in VS Code and run the following code cells.

5.1 Understanding Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is the first phase of any data science project. It allows us to view basic statistical metrics, check distributions, look for missing fields, and visualize relationships between features using correlation matrices.

[JUPYTER CELL 1] Ingesting Data & Handling Empty Cells

Let’s import our scientific libraries, load our CSV file, and print basic dataset details:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
df = pd.read_csv('data/hdi_data.csv')

# Clean and patch any empty lines with the median
for col in df.columns:
    if df[col].isnull().sum() > 0:
        df[col] = df[col].fillna(df[col].median())

print("Dataset Ingested & Missing Values Handled Successfully!")
print(f"Dataset Shape: {df.shape} (Rows, Columns)")
print("\nFirst 5 Records:")
print(df.head())

5.2 Line-by-Line Code Breakdown & Real-World Analogy (Cell 1)

Let’s understand exactly what each block in this cell accomplishes:

import pandas as pd: Imports the Pandas library, renaming it to pd for brevity. Think of Pandas as Python’s version of Microsoft Excel. It allows us to view and manipulate tabular data sheets (called DataFrames).
df = pd.read_csv('data/hdi_data.csv'): Opens the raw spreadsheet and loads it into memory as a DataFrame named df.
df.fillna(df[col].median()): Patches any blank values (NaN) in our dataset with the column median.
- Real-world analogy: If a country forgot to record their average years of schooling, we patch it with the middle value of schooling from all other countries so our mathematical models don’t crash.
df.head(): Displays the top 5 rows of our virtual spreadsheet so we can visually inspect the layout.

[JUPYTER CELL 2] Statistical Summaries & Data Quality Audits

We inspect statistical ranges and target score distributions:

print("Data Summary Statistics:")
print(df.describe())

print("\nTarget Score Statistics (HDI):")
print(df['hdi'].describe())

5.3 Line-by-Line Code Breakdown & Real-World Analogy (Cell 2)

Let’s look at how we audit our data quality in this cell:

df.describe(): Generates statistical metrics for every numeric column (including the mean, standard deviation, minimum value, maximum value, and quartiles).
- Real-world analogy: Imagine a principal reviewing a report card summary. This tells us that GNI per capita ranges from very low values up to $60,000+, with average life expectancy sitting around 70 years.
df['hdi'].describe(): Checks that our target variable hdi ranges correctly between 0.0 and 1.0. If we see numbers outside this range, it indicates raw data anomalies.

[JUPYTER CELL 3] Correlation Heatmap & Bivariate Scatter

We compute and plot the linear relationships between numeric variables:

plt.figure(figsize=(10, 8))
sns.heatmap(df.corr(), annot=True, cmap='viridis', fmt=".2f")
plt.title('Correlation Heatmap of HDI Indicators')
plt.show()

# Visualize income non-linearity
plt.figure(figsize=(8, 4))
sns.scatterplot(x='gross_inc_percap', y='hdi', data=df, color='indigo')
plt.title('Income vs Human Development Index')
plt.xlabel('GNI per Capita')
plt.ylabel('HDI')
plt.show()

5.4 Line-by-Line Code Breakdown & Real-World Analogy (Cell 3)

Let’s understand how variables relate to each other:

sns.heatmap(..., annot=True): Draws the matrix as a colored grid.
- Real-world analogy: Think of a weather radar map. Yellow/warm blocks indicate extremely high positive correlation (close to 1.0), while dark purple/cool blocks represent weak or negative connections.
sns.scatterplot(x='gross_inc_percap', y='hdi', ...): Plots a coordinate dot for each country.
- What it shows: Notice how income curves non-linearly against HDI. A $10,000 increase has a massive impact on a low-income country’s HDI, but almost no impact on a wealthy country’s HDI. To model this mathematically, we must apply a logarithmic transform.

[JUPYTER CELL 4] Split, Log Transform, and Train Baseline Regression Model

We split our features, apply a logarithmic transform to GNI per Capita, partition train/test subsets, and train a baseline Linear Regression model:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Apply Log Transform to stabilize exponential income distribution
df['log_gross_inc_percap'] = np.log(df['gross_inc_percap'])

# Split features (X) and target (y)
X = df[['life_expectancy', 'expec_yr_school', 'mean_yr_school', 'log_gross_inc_percap']]
y = df['hdi']

# Train-Test Split (80% training, 20% validation testing)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a baseline Linear Regression model
reg = LinearRegression()
reg.fit(X_train, y_train)

# Evaluate
y_pred = reg.predict(X_test)
print(f"Baseline R2 Score: {r2_score(y_test, y_pred) * 100:.2f}%")
print(f"Mean Squared Error (MSE): {mean_squared_error(y_test, y_pred):.5f}")

5.5 Line-by-Line Code Breakdown & Real-World Analogy (Cell 4)

Let’s inspect our model preparation and training steps:

np.log(df['gross_inc_percap']): Calculates the natural logarithm of GNI.
- Why we do it: Log transformations convert exponential growth profiles into linear relationships. This allows standard algorithms to fit stable prediction lines.
train_test_split(..., test_size=0.2): Partitions our dataset. The model trains on 80% of countries, and we reserve 20% to test its predictive performance on unseen countries.
LinearRegression(): Fits a linear equation to the data.
- Real-world analogy: Drawing a “best-fit line” through a cloud of scatter points on a graph, allowing us to predict where new dots will fall.
r2_score: Calculates the R-Squared coefficient, representing the proportion of variance explained by our features. A score of 95% means the model captures almost all the variance in human development scores.
mean_squared_error: Calculates the average squared difference between predictions and actual values. Lower values mean higher model accuracy.

Human Development Index Predictor: End-to-End Machine Learning Guide