Walkthroughs & Exercises: Machine Learning for Data Analytics with Python

Interactive, in-browser edition: fill in the code as we go. Nothing to install.

Author

Dr. Chester Ismay

How to use this page

Everything runs in your browser: there’s no Python or Jupyter to install.

Click Run Code on the import cell and the data-loading cells near the top first.
Then work through the page top to bottom, filling in each cell as we live-code together.
Your edits are saved automatically in this browser. Use Start Over on a cell to reset it.
The very first run takes a few seconds while Python loads in the background.

Stuck? Open the completed solutions in another tab.

How this notebook is structured for class: Several walkthrough cells come pre-filled so that class time goes to the core machine learning steps; read and run those, and we will write the remaining walkthrough cells together. The exercises stay mostly blank on purpose: use the walkthrough code as your guide.

Intro: Getting Started with Machine Learning for Data-Driven Decisions

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  D[("Business data:<br/>churn, campaigns")]:::data --> M1["Module 1<br/>Understand it<br/>and clean it"]:::proc
  M1 --> Q{"Is there a labeled<br/>answer to predict?"}:::decision
  Q -->|"yes"| SUP["Module 2: supervised<br/>regression and<br/>classification"]:::proc
  Q -->|"no"| UNS["Module 3: unsupervised<br/>clustering and<br/>market basket"]:::accent
  SUP --> M4["Module 4<br/>Evaluate honestly,<br/>then tune"]:::proc
  UNS --> M4
  M4 --> O[/"A model you would<br/>defend to a stakeholder"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

The whole course in one picture. The fork in the middle is the question that decides which family of methods you reach for.

Walkthrough: Setting Up the Python Environment for ML

If you haven’t already installed Python, Jupyter, and the necessary packages, there are instructions on the course repo in the README to do so here.

You can also install the packages directly in a Jupyter notebook with

Run the following code to check that each of the needed packages are installed. If you get an error, you may need to install the package(s) again.

Exercise: Setting Up the Python Environment

By completing this exercise, you will be able to

Import necessary Python packages
Check for successful package loading
Load datasets into Python

Follow the instructions above in Walkthrough to check for correct installation of necessary packages.

Module 1: Data Understanding and Preprocessing for Machine Learning

Walkthrough 1.1: Exploring and Preprocessing Data with Pandas & Seaborn

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  R[("Raw CSV")]:::data --> I["Inspect<br/>head, info, isnull,<br/>duplicated"]:::proc
  I --> F["Fix<br/>fill gaps,<br/>drop duplicates"]:::proc
  F --> E["Encode<br/>Yes and No<br/>become 1 and 0"]:::proc
  E --> N["Engineer<br/>TotalSpent,<br/>ServiceCount"]:::accent
  N --> V["Visualize<br/>violin, histogram,<br/>scatter"]:::proc
  V --> M[/"A model-ready<br/>table"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

No model appears in this module. Everything here turns a raw export into a table a model can read.

Inspect a dataset using Pandas

Handle missing values and clean data

Create visualizations to identify key business trends

Exercise 1.1: Exploring and Preprocessing Data with Pandas & Seaborn

Inspect a dataset using Pandas

Handle missing values and clean data

Common Pitfall: The Income column has missing values. If you skip this step and try to build models later, you’ll get errors. Always check isnull().sum() after cleaning to verify zeros.

Create visualizations to identify key business trends

Interpretation Questions

Looking at the violin plot of Income by Response, which group (responders or non-responders) shows more variability in income?
Based on your scatter plot, do higher-income customers tend to spend more? Is this relationship strong or weak?
Which education level shows the highest response rate? What marketing implications might this have?

Self-Check

By the end of this module, you should be able to:

Load a dataset and inspect its structure with .info() and .describe()
Identify and handle missing values before modeling
Create at least 3 types of visualizations (histogram, scatter, violin/box)
Articulate one business insight from your exploratory analysis

Module 2: Supervised Learning for Business Decisions

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  Q{"What is the<br/>target column?"}:::decision
  Q -->|"a number<br/>MonthlyCharges, TotalSpent"| R["Regression<br/>LinearRegression"]:::proc
  Q -->|"a yes or no label<br/>Churn, Response"| C["Classification<br/>RandomForestClassifier"]:::proc
  R --> RM[/"Score it with<br/>R-squared and MAE"/]:::good
  C --> CM[/"Score it with accuracy,<br/>precision, and recall"/]:::good
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

One question picks both the model family and the scorecard: is the thing you are predicting a number or a label?

Walkthrough 2.1: Build a Regression Model for Pricing Optimization

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '15px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  Q{"How does the model<br/>read a feature?"}:::decision
  Q -->|"by distance<br/>k-means, k-NN, SVM"| Y1["Scale it. A big-range<br/>feature would otherwise<br/>decide the answer."]:::key
  Q -->|"with a penalty on the<br/>coefficients: logistic,<br/>ridge, lasso"| Y2["Scale it. The penalty<br/>is unit-dependent."]:::key
  Q -->|"one threshold at a time<br/>tree, random forest"| N1["No need. Rescaling moves<br/>the threshold and leaves<br/>the split identical."]:::good
  Q -->|"plain OLS, no penalty"| N2["Optional. The fit is the<br/>same either way; only the<br/>coefficient's units change."]:::accent
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef key fill:#fff3cd,stroke:#d39e00,stroke-width:2px,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a

Scaling is not a formatting habit. Whether a model needs it depends on how that model reads a feature.

Split the data into training and validation sets

Common Pitfall: Fitting StandardScaler on the full dataset before the train/test split leaks information from the validation set into training. Scaling is not a formatting step, it is a learned step: the scaler learns a mean and a standard deviation from whatever you show it. Show it everything and your validation rows have helped decide how the training rows are represented. So split first, fit_transform on train, and transform (never fit) on validation. Module 4 shows the Pipeline version, which enforces this for you.

Why scale at all, and does every model need it? No. It depends on how the model reads a feature.

Distance-based models need it. K-means, k-NN, and SVMs measure distance between points. A feature measured in thousands (Income) swamps one measured in single digits (TotalChildren), so the large-range feature silently decides the answer. This is why we scale before k-means in Module 3.

Regularized linear models need it. Logistic regression and ridge/lasso apply a penalty to the size of the coefficients. Coefficient size depends on feature units, so without scaling the penalty falls unevenly across features. LogisticRegression regularizes by default, which is why Module 4 scales.

Gradient-descent solvers converge faster with it, and PCA needs it because it chases variance, which is unit-dependent.

Tree models do not need it. A decision tree, random forest, or gradient booster splits one feature at a time at a threshold. Rescaling moves the threshold but produces the identical split, so the model is unchanged. Notice we do not scale for the Random Forest in Walkthrough 2.2 or the grid search in 4.2.

Plain OLS is the subtle case. Ordinary least squares with no penalty is invariant to scaling: predictions, R-squared, and MAE come out identical either way. Only the coefficient changes, because it is now “per standard deviation” instead of “per raw unit.” We scale here for interpretability and habit, not because the fit needs it.

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '15px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  ALL[("All rows")]:::data --> SP["train_test_split FIRST"]:::key
  SP --> TR[("Training<br/>80%")]:::data
  SP --> VA[("Validation<br/>20%")]:::data
  TR --> FT["scaler.fit_transform(X_train)<br/>learns the mean and SD here"]:::proc
  FT --> TF["scaler.transform(X_val)<br/>applies them, never re-fits"]:::proc
  VA --> TF
  TF --> OK[/"Validation stays<br/>genuinely unseen"/]:::good
  LEAK["Scale before splitting and the<br/>validation rows help decide<br/>how training is represented"]:::bad -.->|"the mistake"| SP
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef key fill:#fff3cd,stroke:#d39e00,stroke-width:2px,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef bad fill:#fde0e0,stroke:#c62828,color:#1a1a1a

Split first, fit the scaler on the training rows only, then apply it. Fitting on everything leaks the validation set into training.

What is reset_index doing, and do we need it here? Honestly, in this notebook, no. It is a habit worth understanding rather than a fix for a problem we have.

A pandas Series and DataFrame carry row labels (the index) separately from row positions. reset_index(drop=True) throws the old labels away and renumbers the rows 0, 1, 2, and so on. drop=True says discard the old labels instead of keeping them as a new column.

It matters when rows have been removed, because pandas keeps the original labels of whatever survives. Drop rows 3 and 4 from a 10-row table and the index reads 0, 1, 2, 5, 6, …, with gaps. Line up that Series against a fresh NumPy array or another differently-indexed object and pandas aligns on those labels, not on position, which quietly produces NaN rows where the labels do not match.

Here nothing ever removes a row. We filled the missing values in with the median rather than dropping them, so telco_churn still has all 7,043 rows and its index is already 0 through 7,042 with no gaps. These two lines change nothing at all, and the model comes out identical without them. Keep them as insurance for the day you add a .dropna() or a filter above this cell, and recognize the pattern when you meet it in real code.

Train a linear regression model

Evaluate model performance on the validation set

Exercise 2.1: Build a Regression Model for Pricing Optimization

Split the data into training and validation sets

Common Pitfall: Forgetting to scale features before linear regression can work, but it makes the coefficient harder to interpret. Always scale when comparing feature importance.

Train a linear regression model

Evaluate model performance on the validation set

Interpretation Questions

Is your R-squared higher or lower than the telco churn model? What might explain the difference?
If MAE is $200, what does that mean in practical terms for predicting customer spending?
Would you trust this model for making budget decisions? Why or why not?

Walkthrough 2.2: Implement a Classification Model for Customer Churn

Split the data into training and validation sets

Scaling is not as important for tree-based models.

Train a Random Forest classification model

Evaluate model performance on the validation set

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '15px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  A[/"Customer actually<br/>CHURNED"/]:::anchor --> TP["Predicted churn:<br/>caught it"]:::good
  A --> FN["Predicted stay: MISSED it,<br/>no retention offer went out"]:::bad
  K[/"Customer actually<br/>STAYED"/]:::anchor --> TN["Predicted stay:<br/>correct"]:::good
  K --> FP["Predicted churn: false alarm,<br/>a discount you did not<br/>need to give"]:::warn
  classDef anchor fill:#eceff1,stroke:#546e7a,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef bad fill:#fde0e0,stroke:#c62828,color:#1a1a1a
  classDef warn fill:#ffe8cc,stroke:#e8730c,color:#1a1a1a

The confusion matrix in business terms. Recall is about the red box; precision is about the orange one.

Quick Reference: When to Use Which Metric

Situation	Metric	Why
Predicting continuous values	R-squared, MAE	Measures prediction error
Balanced classes	Accuracy	Overall correctness
Cost of false positives is high	Precision	Minimize wrong positive predictions
Cost of false negatives is high	Recall	Catch all actual positives
Need balance	F1-Score	Harmonic mean of precision/recall

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  Q{"Which mistake<br/>costs you more?"}:::decision
  Q -->|"missing a real churner<br/>a false negative"| RE["Optimize RECALL<br/>catch them all"]:::accent
  Q -->|"chasing someone who was<br/>never leaving<br/>a false positive"| PR["Optimize PRECISION<br/>be right when you act"]:::proc
  Q -->|"both, about equally"| F1["Use F1"]:::good
  Q -->|"classes are balanced and<br/>the errors are symmetric"| AC["Accuracy is fine"]:::good
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

Accuracy is the default, not the answer. Let the cost of each kind of mistake choose the metric.

Exercise 2.2: Implement a Classification Model for Customer Churn

Split the data into training and validation sets

Common Pitfall: The Response column is imbalanced (many more 0s than 1s). This is why accuracy alone can be misleading. A model predicting “No Response” every time would still get ~85% accuracy!

Train a Random Forest classification model

Evaluate model performance on the validation set

Interpretation Questions

Compare your precision and recall. Which is higher, and what does that imply about the model’s tendencies?
Looking at your confusion matrix, is the model better at identifying responders or non-responders?
If running a marketing campaign costs $50 per contact, which metric matters more: precision or recall?

Self-Check

By the end of this module, you should be able to:

Explain the difference between regression and classification
Split data into training and validation sets with a fixed random_state
Interpret R-squared, MAE, accuracy, precision, and recall
Explain why we don’t evaluate a model on its training data
Recognize when accuracy is misleading on imbalanced classes

Module 3: Unsupervised Learning and Pattern Discovery in Business

Walkthrough 3.1: Exploring K-Means Clustering for Customer Segmentation

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  X[("Customer<br/>features")]:::data --> S["Standardize<br/>k-means measures<br/>distance"]:::key
  S --> K["Try k = 1 to 10,<br/>record inertia"]:::proc
  K --> E{"Elbow plot:<br/>where does<br/>it bend?"}:::decision
  E --> SIL["Silhouette score<br/>breaks the tie"]:::proc
  SIL --> FIT["Fit at the chosen k,<br/>label every customer"]:::proc
  FIT --> P[/"Segments you can<br/>describe and act on"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef key fill:#fff3cd,stroke:#d39e00,stroke-width:2px,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

Choosing k is the whole job. Scale first, let the elbow narrow the range, and let the silhouette and business sense settle it.

Apply K-Means clustering to segment customers

Common Pitfall: K-means uses Euclidean distance, so an unscaled feature with a large range (like MonthlyCharges) will dominate the clusters. Always standardize features before clustering, and remember the cluster labels are arbitrary integers with no inherent order.

Determine the optimal number of clusters using the Elbow Method

Verify using the silhouette score (optional but recommended)

Fit K-means and assign cluster labels to each customer

Visualize customer segments using a 2D plot

Exercise 3.1: Exploring K-Means Clustering for Customer Segmentation

Apply K-Means clustering to segment customers

Determine the optimal number of clusters using the Elbow Method

Verify using the silhouette score (optional but recommended)

Choosing Your Optimal k

The “right” answer here is somewhat subjective. Look for:

Where the elbow curve bends (typically k=3 to 5 for this data)
The highest silhouette score
If they disagree, silhouette score is often more reliable

Pick a k and justify your choice. There’s no single correct answer.

Fit K-means and assign cluster labels to each customer

Visualize customer segments using a 2D plot

Interpretation Questions

How did you decide on the optimal k? Did the elbow method and silhouette scores agree?
Looking at your 2D visualization, do the clusters seem well-separated or do they overlap?
Can you describe what “type” of customer each cluster might represent based on their TotalChildren and TotalSpent values?

Walkthrough 3.2: Market Basket Analysis with Apriori Algorithm

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  T[("One row per customer,<br/>True or False per service")]:::data --> AP["apriori(min_support=0.2)"]:::proc
  AP --> FI[/"Frequent itemsets:<br/>combinations that<br/>show up often"/]:::accent
  FI --> AR["association_rules(<br/>metric='confidence')"]:::proc
  AR --> CH{"Is lift<br/>above 1?"}:::decision
  CH -->|"yes"| ACT[/"A real association,<br/>worth acting on"/]:::good
  CH -->|"no"| SKIP["Coincidence: the<br/>consequent is just popular"]:::bad
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef bad fill:#fde0e0,stroke:#c62828,color:#1a1a1a

Support finds what is common, confidence finds what follows from it, and lift is the check that stops you bundling two popular things that have nothing to do with each other.

Prepare transactional data (services as items)

Common Pitfall: Setting min_support too low floods you with rules (many spurious), while setting it too high can return no itemsets at all. Also, a rule can have high confidence simply because the consequent is popular: always check lift (> 1) to confirm a real association rather than a coincidence.

Apply the Apriori algorithm to identify frequent itemsets

Generate association rules from frequent itemsets

Interpret insights

Key Metrics:

Support: How often items appear together (0.25 = 25% of customers)
Confidence: If A, how likely B? (0.8 = 80% chance)
Lift: How much more likely than random? (>1 = positive association)

Example interpretation: If rule shows {PhoneService} -> {InternetService} with confidence=0.75 and lift=1.3: “Customers with phone service are 75% likely to also have internet service, and this is 30% more likely than if the purchases were independent.”

Exercise 3.2: Market Basket Analysis with Apriori Algorithm

Prepare transactional data (product categories as items)

Note on Thresholds

The min_support=0.2 and min_threshold=0.6 are starting points. If you get:

Too few rules: lower the thresholds
Too many rules: raise them
All trivial rules: look for higher lift values

Feel free to experiment with different values.

Apply the Apriori algorithm to identify frequent itemsets

Generate association rules from frequent itemsets

Interpretation Questions

Which product category appears in the most frequent itemsets? What does this suggest about purchasing patterns?
Find a rule with high confidence but low lift. Why might this rule be less useful despite high confidence?
Identify one actionable insight: what product bundle would you recommend based on these rules?

Self-Check

By the end of this module, you should be able to:

Explain the difference between supervised and unsupervised learning
Scale features before clustering and explain why it matters
Use the elbow method and silhouette score to choose k
Interpret support, confidence, and lift in association rules
Describe a business application for clustering and market basket analysis

Module 4: Implementing and Evaluating ML Models

Walkthrough 4.1: Exploring Cross-Validation for Model Evaluation

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '15px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  TR[("Training set")]:::data --> F["Cut into 5 folds"]:::proc
  F --> R1["Round 1: fold 1 validates,<br/>folds 2 to 5 train"]:::proc
  F --> R2["Round 2: fold 2 validates,<br/>the rest train"]:::proc
  F --> R5["and so on through<br/>round 5"]:::proc
  R1 --> AVG["5 scores:<br/>a mean and a<br/>standard deviation"]:::accent
  R2 --> AVG
  R5 --> AVG
  AVG --> O[/"How much your score depends<br/>on which rows you happened<br/>to hold out"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

Every row gets a turn in validation. The spread across folds tells you how much a single split was luck.

Split data into training and validation sets

Common Pitfall: Reporting a single train-test split can be lucky or unlucky depending on which rows land where. Cross-validation averages over several folds for a more honest estimate, and forgetting random_state makes those folds (and your results) non-reproducible.

Train a classification model using logistic regression

What the Pipeline actually does. make_pipeline(StandardScaler(), LogisticRegression(...)) glues the two together into a single object that still has .fit() and .predict(), so it behaves exactly like a model. The point is that it enforces the fit/transform asymmetry for you:

pipe.fit(X, y) runs scaler.fit_transform(X), then fits the logistic regression on the result. The scaler learns its mean and SD here, from whatever rows it was handed.

pipe.predict(X_new) runs scaler.transform(X_new), never fit, then predicts. New data is scaled using the numbers learned during fit.

That matters most under cross-validation. cross_validate builds 5 different training sets, and it calls .fit() on the pipeline separately for each fold. So each fold fits its own scaler on that fold’s 80% and applies it to that fold’s held-out 20%. Scale by hand beforehand and you get one scaler fit on all the data, meaning every fold trains on rows whose scaling was informed by the rows it is about to be tested on. There is no seam to leak through here, because the scaler is part of the thing being refit.

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '15px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  P(["make_pipeline(StandardScaler(),<br/>LogisticRegression())"]):::proc --> FIT["pipe.fit(X, y)<br/>scaler.fit_transform,<br/>then model.fit"]:::key
  P --> PRED["pipe.predict(X_new)<br/>scaler.transform only,<br/>never fit"]:::proc
  FIT --> CV["cross_validate refits the<br/>WHOLE pipeline on each fold"]:::accent
  CV --> G[/"Each fold's scaler sees only<br/>that fold's training rows"/]:::good
  H["Scale by hand first and each fold<br/>trains on rows scaled with the<br/>very rows it is tested on"]:::bad -.->|"the seam this closes"| CV
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef key fill:#fff3cd,stroke:#d39e00,stroke-width:2px,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef bad fill:#fde0e0,stroke:#c62828,color:#1a1a1a

Putting the scaler in the pipeline is what makes cross-validation honest: the scaler is refit inside every fold.

Apply k-fold cross-validation to evaluate model performance

Compare metrics across folds

Interpretation

Accuracy -> Overall correctness of predictions

Precision -> How many predicted churns were actual churns

Recall -> How many churns were correctly identified

F1-Score -> Balances precision & recall

Cross-validation ensures that your model generalizes better to unseen data by reducing the risk of overfitting on a single split.

Exercise 4.1: Exploring Cross-Validation for Model Evaluation

Split data into training and validation sets

Train a classification model using logistic regression

Apply k-fold cross-validation to evaluate model performance

Compare metrics across folds

Interpretation

Accuracy -> Overall correctness of predictions

Precision -> How many predicted responders were actual responders

Recall -> How many actual responders were correctly identified

F1-Score -> Harmonic mean of precision and recall

Interpretation Questions

How consistent are your metrics across folds? (Look at the standard deviation values.)
Which metric shows the most variability? What might cause this?
Compare your cross-validation results to the single train-test split in Exercise 2.2. Are they similar?

Walkthrough 4.2: Hyperparameter Tuning with Grid Search

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  G[/"param_grid:<br/>2 x 2 x 2 x 2 = 16<br/>combinations"/]:::data --> S["GridSearchCV(cv=5)<br/>16 x 5 = 80 fits"]:::proc
  S --> B["best_params_:<br/>the highest mean recall"]:::accent
  B --> E["Refit on the<br/>training data"]:::proc
  E --> H["Score on rows the<br/>search never saw"]:::key
  H --> O[/"An honest estimate"/]:::good
  L["Tune and report on the same rows<br/>and the number is inflated"]:::bad -.-> H
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef key fill:#fff3cd,stroke:#d39e00,stroke-width:2px,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef bad fill:#fde0e0,stroke:#c62828,color:#1a1a1a

Grid search is brute force with a fold count attached. Keep a held-out set the search never touched.

Train a Random Forest classifier

Common Pitfall: Tuning hyperparameters and then evaluating on the same data leaks the answer and overstates performance. The honest grid-search workflow selects parameters using cross-validation on the training set, then reports the final score on a held-out test set the search never saw.

Apply grid search to find optimal hyperparameters

Evaluate model improvement using accuracy and recall

Interpret the best hyperparameter combination

Exercise 4.2: Hyperparameter Tuning with Grid Search

Train a Random Forest classifier

Apply grid search to find optimal hyperparameters

Evaluate model improvement using accuracy and recall

Interpret the best hyperparameter combination

Interpretation Questions

Did hyperparameter tuning improve recall compared to the default model in Exercise 2.2?
Look at the best parameters found. Are they at the edges of your grid (suggesting you should expand the search)?
Was the computational cost of grid search worth the performance improvement?

Self-Check

By the end of this module, you should be able to:

Explain why cross-validation is better than a single train-test split
Interpret the mean and standard deviation of cross-validation scores
Define what hyperparameters are and why we tune them
Use GridSearchCV to find optimal model settings
Explain why tuning and reporting on the same data inflates performance

Bonus Challenge: End-to-End ML Pipeline

If you finish early or want additional practice, try this integration challenge using the marketing_campaign data:

Goal: Build the best model to predict Response using everything you’ve learned.

Feature Engineering: Create at least one new feature beyond TotalChildren and TotalSpent (e.g., spending per child, years as customer from Dt_Customer)
Model Comparison: Train both Logistic Regression and Random Forest, use cross-validation to compare them fairly
Optimization: Use GridSearchCV on your better-performing model
Interpretation: Write 2-3 sentences explaining which model you’d recommend and why

This is open-ended. There’s no single right answer. The goal is to practice the full workflow.