Part 5: Interactive Notebook - Exploration & Preprocessing

Open your notebook.ipynb file in VS Code and run the following code cells.

5.1 Understanding Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is the first phase of any data science project. It allows us to view basic statistical metrics, check distributions, look for missing fields, and visualize relationships between features using correlation matrices.

---

[JUPYTER CELL 1] Ingesting & Checking Dataset Shape

Let’s import our libraries, load the raw CSV file, and print basic dataset details:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
df = pd.read_csv('data/flood_prediction.csv')

print("Dataset Loaded Successfully!")
print(f"Dataset Shape: {df.shape} (Rows, Columns)")
print("\nFirst 5 Records:")
print(df.head())

• Why we do this: This verifies that the file is placed correctly in the data/ directory and outputs the row and column dimensions.
• What to look for: Check if the file loaded successfully without errors. You should see 117 rows and 11 columns.

---

[JUPYTER CELL 2] Statistical Summary & Target Class Balance

We check the summary statistics of all climate features and inspect if the target classes are balanced:

print("Data Summary Statistics:")
print(df.describe())

print("\nMissing Values per Column:")
print(df.isnull().sum())

print("\nTarget Class Distribution (Flood vs. No Flood):")
print(df['flood'].value_counts(normalize=True) * 100)

• Why we do this:
  • describe() helps us verify the ranges of our features (e.g., temperatures around 28–31°C, humidity around 70–80%).
  • isnull().sum() checks if there are empty rows that could crash model training.
  • Class distribution prints the ratio of flood events, helping us see if the target classes are balanced.

---

[JUPYTER CELL 3] Correlation Heatmap Visualization

We visualize how different rainfall seasons and weather variables correlate with flood occurrences:

plt.figure(figsize=(10, 8))
sns.heatmap(df.corr(), annot=True, cmap='coolwarm', fmt=".2f")
plt.title('Correlation Heatmap of Weather Features and Flood Status')
plt.show()

• Why we do this: Correlation maps show the linear relationship between variables (from -1 to 1). Features with high positive numbers (close to 1.0) have a strong correlation with flooding.
• What to look for: Notice how the monsoon season rainfall (Jun-Sep and avgjune) has a very strong positive correlation with the flood target. This confirms our meteorological hypothesis that seasonal rain clusters trigger flood disasters.

---

[JUPYTER CELL 4] Split, Standard Scale, and Train Baseline Model

We prepare our features, split our data into training and test partitions, apply a Standard Scaler, and train a baseline Random Forest:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, accuracy_score

# Split features (X) and target (y)
X = df.drop(columns=['flood'])
y = df['flood']

# Train-Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# Apply StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train a baseline Random Forest Classifier
clf = RandomForestClassifier(random_state=42, n_estimators=100)
clf.fit(X_train_scaled, y_train)

# Evaluate
y_pred = clf.predict(X_test_scaled)
print(f"Baseline Accuracy: {accuracy_score(y_test, y_pred) * 100:.2f}%")
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

• Why we do this:
  • StandardScaler: Centers each column’s values around 0 with a standard deviation of 1. This prevents larger variables (like ANNUAL rainfall of 3000mm) from dominating smaller variables (like Temp of 28°C) during model calculation.
  • Stratified Split: Ensures the train and test subsets have the same proportion of flood events.
  • Data Leakage Prevention: We fit our scaler only on the training split (X_train) and apply the transformation to both splits. Fitting on the test set is a major beginner mistake because it leaks statistical information from the unseen test set into the training loop!

---