Part 5: Interactive Notebook - Exploration & Preprocessing

Open your notebook.ipynb file in VS Code and run the following code cells.

5.1 Understanding Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is the first phase of any data science project. It allows us to view basic statistical metrics, check distributions, look for missing fields, and visualize relationships between features using correlation matrices.

---

[JUPYTER CELL 1] Ingesting the Soil Dataset

Let’s import our scientific libraries, load the raw CSV file, and print basic dataset details:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
df = pd.read_csv('data/Crop_recommendation.csv')

print("Dataset Loaded Successfully!")
print(f"Dataset Shape: {df.shape} (Rows, Columns)")
print("\nFirst 5 Records:")
print(df.head())

5.2 Line-by-Line Code Breakdown & Real-World Analogy (Cell 1)

Let’s understand exactly what each block in this cell accomplishes:

• import pandas as pd: Imports the Pandas library, renaming it to pd for brevity. Think of Pandas as Python’s version of Microsoft Excel. It allows us to view and manipulate tabular data sheets (called DataFrames).
• df = pd.read_csv('data/Crop_recommendation.csv'): Opens the raw spreadsheet and loads it into memory as a DataFrame named df.
• df.shape: Returns the dimensions of the spreadsheet.
  • Expected Output: You should see (2200, 8), meaning the table contains 2,200 historical crop records and 8 columns of data.
• df.head(): Displays the top 5 rows of our virtual spreadsheet so we can visually inspect the layout.

---

[JUPYTER CELL 2] Statistical Summaries & Quality Audits

We inspect statistical ranges, audit missing values, and check for duplicate rows:

print("Data Summary Statistics:")
print(df.describe())

print("\nMissing Values per Feature:")
print(df.isnull().sum())

print("\nTarget Class Distribution (Unique Crops):")
print(df['label'].value_counts())

5.3 Line-by-Line Code Breakdown & Real-World Analogy (Cell 2)

Let’s look at how we audit our data quality in this cell:

• df.describe(): Generates statistical metrics for every numeric column (including the mean, standard deviation, minimum value, maximum value, and quartiles).
  • Real-world analogy: Imagine a principal reviewing a report card summary. This tells us the average temperature is around 25.6°C, and soil pH averages 6.47 (slightly acidic, which is normal for agricultural land).
• df.isnull().sum(): Checks for empty cells (NaN values) and outputs the total count for each column.
  • Real-world analogy: Checking an application form to ensure the applicant didn’t leave any fields blank. Machine learning algorithms cannot calculate blank values and will crash if they encounter them.
• df['label'].value_counts(): Counts the number of occurrences of each crop type.
  • What it shows: It outputs exactly 100 records for each of the 22 unique crops (like rice, maize, mango, and coffee). This is a perfectly balanced dataset, meaning our model will learn about all crops equally.

---

[JUPYTER CELL 3] Correlation Heatmap of Soil & Weather Factors

We compute and plot the linear relationships between numeric variables:

plt.figure(figsize=(10, 8))
numeric_cols = df.select_dtypes(include=[np.number])
sns.heatmap(numeric_cols.corr(), annot=True, cmap='coolwarm', fmt=".2f")
plt.title('Correlation Heatmap of Environmental Features')
plt.show()

5.4 Line-by-Line Code Breakdown & Real-World Analogy (Cell 3)

Let’s understand how variables relate to each other:

• numeric_cols.corr(): Computes the Pearson Correlation coefficient matrix. A value close to 1.0 means strong positive correlation (when feature A rises, feature B also rises), while 0 means no connection.
• sns.heatmap(..., annot=True): Draws the matrix as a colored grid.
  • Real-world analogy: Think of a weather radar map. Red/warm blocks indicate high correlation, while blue/cool blocks represent weak or negative connections. In our soil data, Nitrogen, Phosphorus, and Potassium show relatively low correlations with each other, meaning they act as independent nutrients in the soil.

---

[JUPYTER CELL 4] Split, Scale, and Train Baseline Random Forest

We prepare our features, split our data into training and test partitions, apply a Standard Scaler, and train a baseline classifier:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, accuracy_score

# Split features (X) and target (y)
X = df.drop(columns=['label'])
y = df['label']

# Train-Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# Apply StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train a baseline Random Forest Classifier
clf = RandomForestClassifier(random_state=42, n_estimators=100)
clf.fit(X_train_scaled, y_train)

# Evaluate
y_pred = clf.predict(X_test_scaled)
print(f"Baseline Accuracy: {accuracy_score(y_test, y_pred) * 100:.2f}%")
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

5.5 Line-by-Line Code Breakdown & Real-World Analogy (Cell 4)

Let’s inspect our model preparation and training steps:

• X = df.drop(...) and y = df[...]: Separates our questions (features X like soil nutrients and climate parameters) from our answer key (the target y representing crop labels).
• train_test_split(..., test_size=0.2, stratify=y): Partitions our dataset. The model trains on 80% of agricultural profiles (1,760 rows), and we reserve 20% (440 rows) to test its predictive accuracy on unseen soils.
• StandardScaler(): Normalizes feature scales.
  • Real-world analogy: Imagine comparing a student’s grade out of 100 (e.g. 85%) to another student’s grade out of 10 (e.g. 9/10). To evaluate them fairly, you must scale both scores to percentages. In our data, rainfall can exceed 300mm while pH values are small numbers like 6.5. Scaling prevents the model from assuming rainfall is more important simply because its numbers are larger.
• RandomForestClassifier(n_estimators=100): Builds an ensemble model consisting of 100 independent decision trees.
  • Real-world analogy: Instead of relying on a single agricultural expert, we ask 100 soil managers to evaluate the parameters and vote, taking the majority decision as the final recommended crop.
• Data Leakage Prevention: We fit our scaler only on the training split (X_train) and apply the transformation to both splits. Fitting on the test set is a major beginner mistake because it leaks statistical information from the unseen test set into the training loop!

---