scikit-learn Part 2 — The Core Algorithms: Linear Models, SVM, Trees, Ensembles & k-NN

A machine-learning algorithm is a recipe for drawing a decision boundary through data — a line, a curve, a stack of yes/no questions, a vote among a hundred little trees. Part 1 of this arc did the unglamorous, essential half of the job: it turned a messy spreadsheet into a clean numeric matrix and wrapped the whole thing in a Pipeline so no test data could leak into training. This lesson is the half everyone actually shows up for — the models that do the learning, and the judgement to pick the right one.

There are more than fifty classifiers and regressors in scikit-learn, but you will reach for maybe eight of them ninety-nine percent of the time, and they fall into five families: linear models (draw a line), k-nearest neighbors (ask the neighbours), support vector machines (find the widest street between classes), decision trees and their ensembles (twenty questions, then vote), and a lightweight probabilistic baseline, Naive Bayes. By the end of this lesson you will know, for each one, how it decides, when it shines and when it breaks, the two or three hyperparameters that matter, and the single question that governs half of them — does it need its features scaled?

This is Part 2 of a three-part scikit-learn arc. Part 1 was preprocessing: scaling, encoding and pipelines — link it in your head as the thing that produces the X this lesson consumes. Part 3 is cross-validation and hyperparameter tuning — measuring honestly and turning the knobs you meet here. This lesson lives in the middle: it assumes you can build a Pipeline, and it fills that Pipeline with algorithms.

Every code block below was run on Python 3.12.3 with scikit-learn 1.9.0, NumPy 2.5.1 and pandas 3.0.3. The accuracy and coefficient numbers are reproducible (every model is seeded); the millisecond fit-times are not — they depend on your machine and vary run to run, so read them as ratios, never absolutes. Set up a clean environment before you start:

python3 --version                                     # Python 3.12.3
python3 -m venv .venv && source .venv/bin/activate    # Windows: .venv\Scripts\activate
pip install scikit-learn numpy pandas matplotlib
python -c "import sklearn; print(sklearn.__version__)" # 1.9.0

⚠️ Install into a virtual environment, never the system Python — a stray pip install scikit-learn into /usr/bin/python3 is how you end up with three broken NumPy versions and a lost afternoon.

---

Why this matters

Beginners obsess over the algorithm and ignore the data; that is exactly backwards, and Part 1 was the correction. But once your data is clean, the algorithm choice is real, and it is not a matter of taste. The wrong model on the right data can turn a 96% classifier into a 20% one — you saw a version of that in Part 1, where ordinal-encoding cities crippled a linear model. The encoding was the culprit there; here the same size of swing comes from picking a model whose assumptions don’t match your data.

Here is the mental model that organises the whole lesson. Every algorithm makes an assumption about the shape of the boundary between the answers. A linear model assumes the boundary is a straight line (or flat plane). k-NN assumes points near each other share a label. An SVM assumes there is a wide gap it can wedge a boundary into. A decision tree assumes the world can be carved into axis-aligned rectangles. When the assumption fits your data, the model is fast, accurate and often interpretable. When it doesn’t, the model is slow, wrong, or both — and no amount of hyperparameter tuning rescues a fundamentally mismatched assumption. Choosing an algorithm is choosing an assumption about your data, and the skill is matching the two.

Two questions decide most of it, and you should ask them of every dataset before you fit anything. First: is the relationship roughly linear? If a straight line through the features can separate the classes, a linear model will be fast, interpretable and hard to beat — and you will see exactly that later, when logistic regression wins a fair fight against a neural network on a near-linear medical dataset. If the relationship is full of interactions and non-linear kinks, trees and boosting will pull ahead. Second: does the model measure distance? If it does — k-NN, SVM — then features on different scales sabotage it, and you must scale. If it splits on thresholds instead — any tree — scaling is pointless. That one question, “does it measure distance,” is the thread running through this entire lesson, and it is the first thing the decision map at the end will ask.

Hold three sentences for the rest of the lesson. Start with a linear baseline — sometimes it wins. Scale the distance-based models or watch them collapse. When in doubt on tabular data, gradient boosting is the safe default, and a neural network is usually not.

---

Linear models: draw a line, then tame it

A linear model is the oldest idea in the book and still the first thing you should try. It predicts by taking a weighted sum of the features — prediction = w₁·x₁ + w₂·x₂ + … + b — and learning the weights w (the coef_) and the intercept b from data. For regression that sum is the prediction; for classification it gets squashed into a probability. That is the whole model: one weight per feature, a bias term, and the arithmetic a spreadsheet could do.

Its virtues are exactly what a first model should have. It is fast (fitting is essentially solving a system of equations), it is interpretable (each weight tells you how much that feature moves the answer, and in which direction), and it cannot overfit wildly the way a deep tree can, because a straight line has nowhere to hide. Its vice is equally simple: it can only draw straight boundaries. If the truth curves, a plain linear model misses it, and you must either engineer features that straighten the curve (Part 1’s PolynomialFeatures) or switch to a model that bends.

Ordinary least squares, and why it needs help

LinearRegression fits the plain weighted sum by minimizing squared error — ordinary least squares (OLS). It has no hyperparameters worth touching, which is either restful or a warning sign depending on your data. Its failure mode is overfitting through large, unstable coefficients, especially when features are correlated. Watch it on the built-in diabetes dataset (442 patients, 10 features, predicting disease progression). Two of the features, s1 and s2, are blood serum measurements that correlate strongly — and OLS reacts by blowing up their coefficients in opposite directions:

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso

X, y = load_diabetes(return_X_y=True)
names = load_diabetes().feature_names            # ['age','sex','bmi','bp','s1'...'s6']
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0)
sc = StandardScaler().fit(Xtr)                   # scale so coefficients are comparable
Xtr_s, Xte_s = sc.transform(Xtr), sc.transform(Xte)

ols   = LinearRegression().fit(Xtr_s, ytr)
ridge = Ridge(alpha=10.0).fit(Xtr_s, ytr)
print(f"{'feature':>6} | {'OLS':>7} | {'Ridge':>7}")
for i in [4, 5]:                                 # s1 and s2, the correlated pair
    print(f"{names[i]:>6} | {ols.coef_[i]:>7.2f} | {ridge.coef_[i]:>7.2f}")

feature |     OLS |   Ridge
     s1 |  -26.52 |   -7.39
     s2 |   12.38 |   -2.52

OLS gives s1 a coefficient of −26.5 and s2 one of +12.4 — big, opposed numbers that partly cancel. That is the model straining to fit noise in two correlated columns, and it is fragile: a slightly different training set would swing those numbers wildly. This is where regularization earns its place.

Regularization: penalize big weights to fight overfitting

The idea is one sentence: add a penalty for large coefficients to the thing the model minimizes, so it only grows a weight when the data really justifies it. Left alone, a linear model will happily use enormous coefficients to chase every wrinkle in the training data — that is overfitting. Regularization taxes coefficient size, so the model keeps weights small unless a feature pays for itself in accuracy. There are two taxes, and the difference between them is the single most useful thing to understand about linear models.

Ridge (L2) penalizes the sum of squared coefficients. It shrinks every weight toward zero, smoothly and proportionally, but never quite to zero. In the table above, Ridge pulled s1 from −26.5 to −7.4 and s2 from +12.4 to −2.5 — the unstable pair is tamed, the coefficients are smaller and more believable, and the model generalizes better. Ridge is the safe default regularizer: it keeps all your features but stops any one of them from dominating.

Lasso (L1) penalizes the sum of absolute coefficients. This sounds like a small mathematical change and it produces a dramatic behavioural one: L1 drives weak coefficients exactly to zero. A Lasso model doesn’t just shrink features, it deletes them — which makes Lasso a feature-selection tool as much as a regularizer. Watch it happen. As we raise alpha (the penalty strength), Lasso zeros out more and more of the ten diabetes features:

print(f"{'alpha':>6} | {'# kept':>6} | features Lasso keeps")
for a in [0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0]:
    m = Lasso(alpha=a).fit(Xtr_s, ytr)
    kept = [names[i] for i in range(len(names)) if abs(m.coef_[i]) > 1e-6]
    print(f"{a:>6} | {len(kept):>6} | {kept}")

 alpha | # kept | features Lasso keeps
  0.01 |     10 | ['age','sex','bmi','bp','s1','s2','s3','s4','s5','s6']
   0.1 |     10 | ['age','sex','bmi','bp','s1','s2','s3','s4','s5','s6']
   0.5 |      9 | ['age','sex','bmi','bp','s1','s3','s4','s5','s6']
   1.0 |      8 | ['age','sex','bmi','bp','s1','s3','s5','s6']
   2.0 |      7 | ['sex','bmi','bp','s1','s2','s3','s5']
   5.0 |      5 | ['sex','bmi','bp','s3','s5']
  10.0 |      4 | ['bmi','bp','s3','s5']

At alpha=0.5 the first feature drops out; by alpha=10 only four survive — bmi, bp, s3, s5 — and notice they are medically plausible drivers of diabetes progression. Lasso has performed automatic feature selection, and the survivors at high penalty are the features that carry real signal. This is the property to remember: when you need a sparse, interpretable model that uses only a handful of features, reach for Lasso. The danger is over-shrinking — push alpha too high and Lasso zeros out everything, leaving a model that predicts the mean.

Lining all three up on the full ten features makes the difference visceral — Ridge shrinks, Lasso deletes:

lasso = Lasso(alpha=1.0).fit(Xtr_s, ytr)          # ols, ridge fit above
for i, nm in enumerate(names):
    print(f"{nm:>4} | OLS {ols.coef_[i]:>7.2f} | Ridge {ridge.coef_[i]:>7.2f} | Lasso {lasso.coef_[i]:>7.2f}")

OLS’s wild ±26/±12 swing on the correlated s1/s2 pair is tamed by Ridge (−7.4 / −2.5), and Lasso goes further — it deletes s2 and s4 outright (exactly 0.00) while keeping the strong bmi and s5 drivers near their OLS values. L2 shrinks everything a little; L1 removes the weak entirely. That single visual difference — a column of near-zeros versus a column of exact zeros — is the whole distinction between the two penalties.

ElasticNet is the compromise: it blends both penalties, controlled by l1_ratio (0 = pure Ridge, 1 = pure Lasso). You use it when you want Lasso’s feature selection but you have groups of correlated features — pure Lasso tends to arbitrarily pick one from each correlated group and zero the rest, while ElasticNet keeps or drops them together, which is often what you want.

⚠️ Every regularized linear model must be scaled first. The penalty is applied per-coefficient, so a feature measured in large units (rupees) gets an unfairly small coefficient and is under-penalized, while a small-unit feature (a 0–5 rating) is over-penalized. Scaling puts every feature on equal footing before the tax is levied. This is the “regularized linear models need scaling” row from Part 1, and it is not optional.

Logistic regression: the linear classifier

For classification, the linear model gets one extra step. The weighted sum can be any real number from −∞ to +∞, but a probability must live in [0, 1]. Logistic regression feeds the weighted sum through the sigmoid function, σ(z) = 1 / (1 + e⁻ᶻ), which squashes any real number into a probability:

def sigmoid(z): return 1 / (1 + np.exp(-z))
for z in [-6, -2, -0.5, 0, 0.5, 2, 6]:
    print(f"sigmoid({z:>4}) = {sigmoid(z):.4f}")

sigmoid(  -6) = 0.0025
sigmoid(  -2) = 0.1192
sigmoid(-0.5) = 0.3775
sigmoid(   0) = 0.5000
sigmoid( 0.5) = 0.6225
sigmoid(   2) = 0.8808
sigmoid(   6) = 0.9975

A large positive weighted sum → probability near 1; a large negative sum → near 0; a sum of exactly 0 sits on the fence at 0.5, which is the decision boundary. Despite the name, logistic regression is a classifier, and it draws a straight-line boundary just like linear regression draws a straight fit — the sigmoid only converts the line’s output into a probability. It is, for the same reasons as its regression cousin, the best default classifier to start with: fast, interpretable, calibrated probabilities out of the box.

Its one hyperparameter you will actually tune is C, and its meaning trips up everyone once: C is the inverse of regularization strength. Small C means strong regularization (small coefficients, simpler model); large C means weak regularization (the model is freer to fit, coefficients grow). Watch C sweep on the breast-cancer dataset — as C rises, the total size of the coefficients explodes from 1.4 to 95:

from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression

bc = load_breast_cancer()
Xtr, Xte, ytr, yte = train_test_split(bc.data, bc.target, test_size=0.25,
                                      random_state=0, stratify=bc.target)
for C in [0.001, 0.01, 0.1, 1.0, 100.0]:
    pipe = make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=5000))
    pipe.fit(Xtr, ytr)
    coef = pipe.named_steps["logisticregression"].coef_.ravel()
    print(f"C={C:>7} | test acc={pipe.score(Xte, yte):.3f} | sum|coef|={np.abs(coef).sum():6.2f}")

C=  0.001 | test acc=0.881 | sum|coef|=  1.39
C=   0.01 | test acc=0.951 | sum|coef|=  3.85
C=    0.1 | test acc=0.958 | sum|coef|=  8.47
C=    1.0 | test acc=0.958 | sum|coef|= 17.13
C=  100.0 | test acc=0.951 | sum|coef|= 95.00

Too small a C (0.001) over-regularizes — the coefficients are tiny and the model underfits at 0.881. Too large a C (100) lets the coefficients run to 95 and the model slightly overfits, dipping to 0.951. The sweet spot is in the middle. That U-shape — underfit at one extreme, overfit at the other, best in between — is the shape of every regularization hyperparameter you will ever tune, and Part 3 is about finding its bottom automatically.

The one warning that bites beginners: logistic regression is fit by an iterative optimizer, and on unscaled or badly-conditioned data it can hit its iteration cap before converging, printing ConvergenceWarning: lbfgs failed to converge. The fix is almost always scale your features (which straightens the loss surface — you saw 132 iterations drop to 9 in Part 1) and only then, if needed, raise max_iter. A convergence warning is a signal to scale, not merely to crank the iteration count.

---

k-Nearest Neighbors: the model that doesn’t learn

k-NN is the one algorithm here that does no real training at all. Its fit method just memorizes the training data. All the work happens at prediction time: to classify a new point, k-NN finds the k closest training points (by straight-line distance) and takes a majority vote of their labels. That is the entire algorithm — “you are what your neighbours are.” Because it defers everything to prediction, it is called a lazy or instance-based learner, in contrast to the eager learners (linear, trees) that do their work up front in fit and then predict cheaply.

This design has sharp consequences. k-NN makes no assumption about the shape of the boundary — it can trace any wiggly, non-linear frontier, because the boundary is defined implicitly by where the neighbourhoods flip. That flexibility is its strength on small, low-dimensional data with irregular class shapes. But it has three serious weaknesses, and you must know all three.

First, and non-negotiably: k-NN is driven entirely by distance, so it must be scaled. If one feature ranges 0–2500 (tumour area) and another 0–0.2 (smoothness), the straight-line distance is completely dominated by the big-range feature and the small one is invisible — exactly the collapse you saw in Part 1. Any k-NN that is not preceded by a StandardScaler is broken, whether or not the number looks plausible.

Second, k is a bias-variance dial. A small k (like 1) makes the model sensitive to every single point, including noise — high variance, a jagged boundary that overfits. A large k averages over a big neighbourhood — high bias, a smooth boundary that can underfit. Watch it sweep, scaled, on breast cancer:

from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
for k in [1, 3, 5, 11, 21, 51, 101]:
    pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))
    cv = cross_val_score(pipe, bc.data, bc.target, cv=5).mean()
    print(f"k={k:>4} | CV acc={cv:.3f}")

k=   1 | CV acc=0.954
k=   3 | CV acc=0.960
k=   5 | CV acc=0.965
k=  11 | CV acc=0.965
k=  21 | CV acc=0.956
k=  51 | CV acc=0.951
k= 101 | CV acc=0.924

k=1 is good but slightly jagged (0.954); the sweet spot is k=5–11 (0.965); by k=101 the neighbourhood is so large it smooths across the class boundary and underfits to 0.924. The usual starting point is a small odd number (odd avoids ties) around 5–15, then tune. Note that the best k scales loosely with dataset size — bigger data supports bigger neighbourhoods.

Third, k-NN suffers the curse of dimensionality. In high dimensions, distance stops being meaningful: everything becomes roughly equidistant from everything else, and the “nearest” neighbours are barely nearer than the farthest. Add pure-noise features and watch k-NN rot — the same informative signal, drowned in useless dimensions:

from sklearn.datasets import make_classification
rng = np.random.default_rng(0)
Xg, yg = make_classification(n_samples=800, n_features=5, n_informative=5,
                             n_redundant=0, random_state=0)
for extra in [0, 20, 100, 500]:
    Xd = np.column_stack([Xg, rng.normal(size=(800, extra))]) if extra else Xg
    pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(7))
    print(f"{5+extra:>4} features ({extra} noise) | kNN CV={cross_val_score(pipe, Xd, yg, cv=5).mean():.3f}")

   5 features (0 noise) | kNN CV=0.921
  25 features (20 noise) | kNN CV=0.805
 105 features (100 noise) | kNN CV=0.661
 505 features (500 noise) | kNN CV=0.591

From 0.921 down to 0.591 — barely above a coin flip — purely from irrelevant dimensions. Trees and linear models with regularization tolerate junk features far better because they can learn to ignore them; k-NN cannot, because every feature contributes to the distance. The lesson: k-NN wants few, informative, scaled features. Give it hundreds of raw columns and it drowns.

---

Support Vector Machines: the widest street

A support vector machine draws a boundary too, but with a specific, elegant goal: it finds the boundary that leaves the widest possible margin — the biggest empty street — between the two classes. Of all the lines that separate the classes, an SVM picks the one that is as far as possible from the nearest points of either side. Those nearest points, the ones touching the edge of the street, are the support vectors, and they alone define the boundary — move any other point and nothing changes. The intuition is that the widest-margin boundary is the most robust one, the least likely to misclassify a new point that lands near the frontier.

That handles data separable by a straight line. The reason SVMs became famous is the kernel trick, which handles data that isn’t. Instead of drawing a straight boundary in the original features, an SVM can implicitly map the data into a much higher-dimensional space where it becomes linearly separable, draw the flat boundary there, and project it back — where it appears as a curve. The “trick” is that it does this without ever actually computing the high-dimensional coordinates, using a kernel function as a shortcut. You choose the kernel, and the choice sets the family of shapes the boundary can take:

from sklearn.svm import SVC
for kern in ["linear", "poly", "rbf", "sigmoid"]:
    pipe = make_pipeline(StandardScaler(), SVC(kernel=kern))
    print(f"kernel={kern:>8} | CV acc={cross_val_score(pipe, bc.data, bc.target, cv=5).mean():.3f}")

kernel=  linear | CV acc=0.972
kernel=    poly | CV acc=0.903
kernel=     rbf | CV acc=0.974
kernel= sigmoid | CV acc=0.953

The rbf (radial basis function) kernel is the default and the right first choice — it can form smooth, curved, local boundaries and works well across a huge range of problems. linear is what you use when you suspect the data is linearly separable (and here, on near-linear breast cancer, it nearly ties rbf) or when you have so many features that a straight boundary is already flexible enough. poly fits polynomial curves; sigmoid is rarely the best. The two hyperparameters that make or break an rbf SVM are C and gamma.

C is the same inverse-regularization dial as in logistic regression, controlling the softness of the margin: a small C allows a wide margin that tolerates some misclassified points (more regularization, simpler), a large C insists on classifying every training point correctly, even at the cost of a narrow, contorted margin (less regularization, prone to overfit). gamma sets the reach of each training point’s influence in the rbf kernel: a small gamma means each point influences a wide region (smooth, global boundary), a large gamma means each point only influences its immediate vicinity (wiggly, local boundary that memorizes the training set). Watch gamma go from underfit to catastrophic overfit — train accuracy climbs to a perfect 1.000 while test accuracy collapses to 0.636:

for g in ["scale", 0.001, 0.01, 0.1, 1.0, 10.0]:
    pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf", gamma=g, C=1.0)).fit(Xtr, ytr)
    print(f"gamma={str(g):>7} | train={pipe.score(Xtr, ytr):.3f} test={pipe.score(Xte, yte):.3f}")

gamma=  scale | train=0.993 test=0.958
gamma=  0.001 | train=0.951 test=0.958
gamma=   0.01 | train=0.986 test=0.958
gamma=    0.1 | train=0.995 test=0.923
gamma=    1.0 | train=1.000 test=0.636
gamma=   10.0 | train=1.000 test=0.629

At gamma=1.0 the model scores a perfect 1.000 on training data and a dismal 0.636 on test — the textbook signature of overfitting, a model that has memorized rather than learned. The default gamma='scale' (which sets gamma from the data’s variance) is a well-chosen starting point; when you tune, C and gamma are tuned together, because they interact. Like every distance-and-kernel method, an SVM must be scaled — the rbf kernel measures distance, and gamma assumes features share a comparable spread.

C tells the same over/under-fit story from the margin side. Too small and the margin is so soft the SVM stops separating the classes at all — it collapses to predicting the majority; too large and it contorts a narrow margin to classify every training point:

At C=0.01 the margin is so forgiving the SVM gives up and labels every tumour benign (0.627 — exactly the base rate); at C=100 it reaches a perfect training score and sheds test accuracy. Because C and gamma push against each other, Part 3 tunes them on a two-dimensional grid rather than one at a time.

The honest limitation: SVMs do not scale to large datasets. Training is roughly quadratic-to-cubic in the number of samples, so an SVM that fits in a blink on 1,000 rows can take minutes on 100,000 and is impractical on millions. On the 4,000-row set you will meet shortly, the rbf SVM already took ~50× longer to fit than logistic regression. For large-n linear problems reach for LinearSVC or SGDClassifier instead; for large-n non-linear problems, gradient boosting is both faster and usually better.

---

Decision Trees: questions all the way down

A decision tree is the most human-readable model there is: it is a flowchart of yes/no questions. “Is worst_perimeter > 106? If yes, is worst_concave_points > 0.14? If yes → malignant.” The tree learns which questions to ask, and in what order, by recursively splitting the data: at each node it picks the single feature-and-threshold that best separates the classes (measured by Gini impurity or entropy), splits the data into two, and repeats on each half until the leaves are pure or a stopping rule fires. To predict, you drop a new sample in at the top and follow the answers down to a leaf.

This gives trees three properties nothing else here matches. They are fully interpretable — you can print the tree and read the exact rule for any prediction. They need no scaling — a split asks “is feature > threshold,” and multiplying that feature by 1000 or adding 100 to it doesn’t change which side of the threshold a point falls on, so trees are perfectly scale-invariant. And they capture non-linear boundaries and interactions natively, by stacking axis-aligned splits into any staircase shape.

Their fatal flaw, left unchecked, is overfitting: a tree grown to full depth will keep splitting until every training point sits in its own pure leaf, memorizing the training set completely — 100% train accuracy, mediocre test accuracy. The cure is pruning: stop the tree early. Watch a single tree overfit on breast cancer, and watch depth control it:

from sklearn.tree import DecisionTreeClassifier
for d in [1, 2, 3, 5, None]:
    t = DecisionTreeClassifier(max_depth=d, random_state=0).fit(Xtr, ytr)
    print(f"max_depth={str(d):>4} | train={t.score(Xtr, ytr):.3f} "
          f"test={t.score(Xte, yte):.3f} | leaves={t.get_n_leaves()}")

max_depth=   1 | train=0.930 test=0.888 | leaves=2
max_depth=   2 | train=0.932 test=0.888 | leaves=4
max_depth=   3 | train=0.977 test=0.916 | leaves=8
max_depth=   5 | train=1.000 test=0.902 | leaves=18
max_depth=None | train=1.000 test=0.902 | leaves=18

max_depth=1 (a single question, a “decision stump”) underfits at 0.888. max_depth=3 is the sweet spot here at 0.916. Let the tree grow unbounded and it hits 1.000 on training — perfect memorization — but falls to 0.902 on test, because the deep splits are fitting noise. That gap between perfect train and mediocre test is the unmistakable fingerprint of an overfit tree. The two main pruning knobs are max_depth (cap how many questions deep the tree can go) and min_samples_leaf (require every leaf to hold at least N samples, so the tree can’t carve out a leaf for a single outlier).

The deeper truth, though, is that a single decision tree is rarely the best model — it is too high-variance, its predictions swinging on small data changes. Its real value is twofold: as an interpretable model when you must explain every decision, and as the building block for ensembles, which is where trees become genuinely powerful.

---

Ensembles: the wisdom of many trees

One tree is weak; a crowd of trees is formidable. Ensemble methods combine many trees into one predictor, and they dominate tabular machine learning so thoroughly that on most real business datasets the practical question isn’t whether to use a tree ensemble but which one. There are two ways to build the crowd, and the difference between them is the difference between the two best models you will ever throw at a spreadsheet.

Random Forest: bagging, and the safe default

A random forest builds many deep trees (default 100) and averages their votes. To make the trees usefully different from each other — averaging identical trees gains nothing — it uses two sources of randomness: each tree is trained on a random bootstrap sample of the rows (bagging = bootstrap aggregating), and at each split it considers only a random subset of the features. The individual trees overfit in different directions, and averaging cancels their errors, producing a model far more stable than any single tree. A random forest is the closest thing machine learning has to a “just works” button: it is robust, hard to overfit badly, needs no scaling, tolerates junk features and outliers, and its defaults are usually decent.

Its second gift is feature_importances_ — a ranking of which features the forest actually relied on, computed from how much each feature reduced impurity across all the splits. On breast cancer it surfaces exactly the clinically sensible drivers:

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(Xtr, ytr)
print(f"RF test acc = {rf.score(Xte, yte):.3f}")
for i in np.argsort(rf.feature_importances_)[::-1][:6]:
    print(f"  {bc.feature_names[i]:24} {rf.feature_importances_[i]:.3f}")

RF test acc = 0.944
  worst perimeter          0.145
  worst radius             0.134
  worst concave points     0.121
  worst area               0.101
  mean concave points      0.100
  mean concavity           0.066

The forest leans hardest on the “worst” (largest) tumour measurements — perimeter, radius, concave points — which is medically reasonable, and the importances sum to 1.0. This is genuine, free interpretability, though it comes with a caveat we will sharpen shortly.

Gradient Boosting: sequential error-correction, the usual winner

Random forests build their trees in parallel and independently. Gradient boosting builds them sequentially, each new tree correcting the errors of the ones before it. The first tree makes a rough prediction; the second tree is trained to predict the first tree’s residual errors; the third corrects what’s left, and so on, each tree nudging the ensemble a little closer to the truth. The nudges are deliberately small — scaled by a learning_rate — so the model improves gradually and doesn’t overshoot. This sequential error-correction is more powerful than bagging’s averaging, and on tabular data gradient boosting is usually the most accurate model you can train.

scikit-learn’s modern implementation is HistGradientBoostingClassifier (and its regressor twin). The “Hist” is for histogram — it buckets continuous features into a few hundred bins, which makes it dramatically faster than the older GradientBoostingClassifier on large data, and it handles missing values natively. Compare the two, then watch the learning_rate–iterations interplay that governs boosting:

from sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier
for name, mdl in [("GradientBoosting", GradientBoostingClassifier(random_state=0)),
                  ("HistGradientBoosting", HistGradientBoostingClassifier(random_state=0))]:
    print(f"{name:22} CV acc={cross_val_score(mdl, bc.data, bc.target, cv=5).mean():.3f}")

for lr in [0.01, 0.1, 0.3, 1.0]:
    m = HistGradientBoostingClassifier(learning_rate=lr, max_iter=200, random_state=0).fit(Xtr, ytr)
    print(f"learning_rate={lr:>5} | train={m.score(Xtr, ytr):.3f} test={m.score(Xte, yte):.3f}")

GradientBoosting       CV acc=0.963
HistGradientBoosting   CV acc=0.965
learning_rate=  0.01 | train=0.988 test=0.937
learning_rate=  0.1 | train=1.000 test=0.937
learning_rate=  0.3 | train=1.000 test=0.944
learning_rate=  1.0 | train=1.000 test=0.930

The governing trade-off in boosting is learning_rate × n_estimators (called max_iter in HistGB): a small learning rate needs many trees but generalizes better; a large learning rate needs few trees but risks overshooting and overfitting (note lr=1.0 scores a perfect 1.000 on train but the worst test score, 0.930). The standard recipe is a smallish learning rate (0.05–0.1) with as many trees as your compute budget allows, using early stopping to find the right number.

The honest note on XGBoost, LightGBM and CatBoost. These three third-party libraries are, in practice, what wins Kaggle competitions and powers a great deal of production tabular ML. They are all gradient boosting — the same sequential error-correction — with better engineering: faster training, smarter handling of categoricals (CatBoost especially), and more tuning knobs. HistGradientBoosting was scikit-learn’s answer to LightGBM and is competitive with all three for most purposes, so start with HistGradientBoostingClassifier (no extra install, sklearn-native, in your Pipeline) and reach for XGBoost or LightGBM when you are squeezing out the last points or need their specific features. They are not a different idea; they are a faster, more tunable version of the idea you just learned.

---

Naive Bayes: the fast baseline

Naive Bayes is the sprinter of classifiers: it computes class probabilities directly from the data using Bayes’ theorem, under the “naive” assumption that all features are independent given the class. That assumption is almost always false, yet the model is often surprisingly decent — and it is blisteringly fast to train, because there is no iterative optimization, just counting and a little arithmetic. In the bake-off below, GaussianNB fits in 0.3 milliseconds, faster than anything else.

Its natural home is text classification — spam filters, topic tagging, sentiment — where MultinomialNB (for word counts) or ComplementNB (for imbalanced text) run over a bag-of-words or TF-IDF matrix and give a strong baseline in milliseconds. For continuous features, GaussianNB models each feature as a bell curve per class. Reach for Naive Bayes as a fast baseline to beat, as a text-classification default, or when you have very little data and training time matters more than the last few points of accuracy. Note that scaling does not help Naive Bayes — it works on per-feature distributions, not cross-feature distance.

---

A fair fight: comparing algorithms honestly

Theory tells you which model should win; only a fair experiment tells you which does. “Fair” means one dataset, one split, the same evaluation, and every model in a Pipeline that scales it if and only if it needs scaling — so no algorithm is handicapped by a preprocessing choice that suits another. Here are eight algorithms on breast cancer (569 samples, 30 features), reporting held-out test accuracy, 5-fold cross-validated accuracy, and fit time:

A) BREAST CANCER (569×30, nearly linearly separable)
model                  |  test |    CV |  fit ms | scaled
LogisticRegression     | 0.958 | 0.981 |     1.9 | yes
kNN (k=7)              | 0.958 | 0.970 |     0.5 | yes
SVM (rbf)              | 0.958 | 0.974 |     1.6 | yes
GaussianNB             | 0.923 | 0.939 |     0.3 | no
DecisionTree           | 0.902 | 0.917 |     2.3 | no
RandomForest           | 0.944 | 0.963 |   199.4 | no
HistGradientBoosting   | 0.937 | 0.965 |   306.4 | no
MLP (neural net)       | 0.951 | 0.977 |   120.9 | yes

Read this table slowly, because it contains a genuine surprise and an important lesson. Logistic regression wins — 0.981 cross-validated, ahead of the random forest (0.963), gradient boosting (0.965), and even a neural network (0.977) — and it fits in under two milliseconds. Why? Because the breast-cancer classes are very nearly linearly separable, so the simplest model, which assumes a straight boundary, is exactly right, and everything fancier is solving a harder problem than the data poses. This is why you always run a linear baseline first: sometimes it wins, and it wins cheap. The lone decision tree is the weakest (0.917) — a single tree is high-variance — while the ensembles that wrap many trees recover most of the gap. And the neural net, at 60× the fit time of logistic regression, does not beat it.

But breast cancer is near-linear, which flatters the linear model. Change the data to something genuinely non-linear — a synthetic set with feature interactions and clustered classes that no straight line can separate — and run the exact same loop (plus an MLPClassifier neural-net row). The ranking inverts:

from sklearn.datasets import make_classification
Xb, yb = make_classification(n_samples=4000, n_features=20, n_informative=8,
                             n_redundant=2, n_clusters_per_class=4, class_sep=0.9,
                             flip_y=0.02, random_state=42)   # clusters + interactions, not linearly separable
# ... same models, same 5-fold cross_val_score as above ...

B) SYNTHETIC NON-LINEAR (4000×20, interactions + clusters)
model                  |  test |    CV |  fit ms | scaled
LogisticRegression     | 0.674 | 0.689 |     1.9 | yes
kNN (k=7)              | 0.731 | 0.751 |     0.7 | yes
SVM (rbf)              | 0.782 | 0.787 |   102.7 | yes
GaussianNB             | 0.703 | 0.708 |     0.6 | no
DecisionTree           | 0.681 | 0.701 |    32.5 | no
RandomForest           | 0.791 | 0.810 |  1419.2 | no
HistGradientBoosting   | 0.800 | 0.828 |   468.7 | no
MLP (neural net)       | 0.798 | 0.802 |  1511.7 | yes

Now HistGradientBoosting wins (0.828), the random forest is close behind (0.810), and logistic regression has collapsed to 0.689 — the linear assumption simply cannot bend around the interactions. This is the general rule the industry lives by: on tabular data, gradient boosting is the model to beat. And look at the neural network: 0.802, close but not ahead of boosting, at three times the fit time. That is not a fluke of this dataset — it is a robust, repeatedly-published finding that on ordinary tabular data, gradient-boosted trees generally match or beat deep neural networks, which is why the DL lessons that follow this arc are aimed at images, text and audio, not spreadsheets. A neural network is the right tool when features can’t be hand-crafted; on a table with named columns, boosting usually wins for a fraction of the effort.

Two honest caveats keep this from being folklore. First, the winner is data-dependent — that is the entire point of running the comparison rather than trusting a leaderboard. Second, fit times are illustrative, not portable: they depend on your CPU, core count and system load, and they wobble run to run (the ensembles’ hundreds of milliseconds versus the linear models’ single digits is the durable signal, not the exact figures). The reproducible part is the accuracy ordering, and its lesson is: run the simple model and the boosting model, always; add others when you have a reason.

The fit-time column also hides a second cost — prediction time — and the two do not move together. The honest complexity story per family (n = samples, p = features):

Two rows deserve a flag. kNN is free to train and expensive to predict — it does all its work per query, so a model that fits instantly can be too slow to serve at scale. SVM is the opposite trouble: its training is quadratic-to-cubic in the number of rows, so it simply does not finish on large datasets. Everything tree-based sits comfortably in the middle — a third reason gradient boosting is the tabular default. (For the formal Big-O treatment behind this table, the algorithms-and-complexity lesson linked at the end is the companion.)

---

Reading the model: interpretability, and its traps

A prediction you can’t explain is a hard thing to trust, deploy, or debug. The good news is that two of the model families here are interpretable for free; the important news is that the free interpretability can quietly mislead you.

Linear models expose their reasoning as coefficients. Each coef_ is the weight on a feature: its sign says which class the feature pushes toward, its magnitude (on scaled data) says how strongly. On breast cancer, the logistic model’s strongest coefficients are all negative on “large tumour” measurements, meaning bigger values push toward the malignant class — exactly the direction a doctor would expect. Coefficients are the gold standard of interpretability: precise, signed, and directly tied to the model’s arithmetic. Their limitation is that they only tell the linear story; if the truth is non-linear, the coefficients describe a boundary that doesn’t really fit.

Tree ensembles expose feature_importances_ — how much each feature reduced impurity across all splits — which you already saw rank the breast-cancer features sensibly. But importances carry a real trap: they are computed from the model’s structure, not from causation, and correlated features split their importance. Duplicate a strong feature and watch its apparent importance get cut in half, though nothing about the feature changed:

Xg, yg = make_classification(n_samples=2000, n_features=6, n_informative=4,
                             n_redundant=0, random_state=0)
rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(Xg, yg)
top = int(np.argmax(rf.feature_importances_))
print(f"before: feature {top} importance = {rf.feature_importances_[top]:.3f}")
Xd = np.column_stack([Xg, Xg[:, top]])            # duplicate the strongest feature
imp = RandomForestClassifier(n_estimators=300, random_state=0).fit(Xd, yg).feature_importances_
print(f"after duplicating it: {imp[top]:.3f} + copy {imp[-1]:.3f} = {imp[top]+imp[-1]:.3f}")

before: feature 2 importance = 0.268
after duplicating it: 0.146 + copy 0.142 = 0.288

The feature that scored 0.268 now scores 0.146 while its identical twin scores 0.142 — the forest split the credit between them, and a naive reading would conclude the feature is “half as important” when it is exactly as important as before. So: never read feature_importances_ as a causal or stable ranking. Two correlated real-world features (height and weight, say) will each look weaker than either truly is. For a more robust view, prefer permutation importance (shuffle a feature and measure how much accuracy drops — model-agnostic and less fooled by correlation) or SHAP values (a game-theoretic per-prediction attribution, the current standard for explaining individual predictions in production). SHAP lives in the third-party shap library and works with any model; it is the tool to reach for when “why did the model decide this for this customer” is a question you must answer.

The honest summary: interpretability is a spectrum, not a switch. Linear coefficients and a shallow tree are transparent; a random forest is semi-transparent through importances (with the caveat above); a tuned gradient-boosting model or a neural net is opaque and needs SHAP to explain. If regulatory or ethical constraints require you to explain every decision, that requirement narrows your model choice as much as accuracy does — and it belongs in the decision map next.

---

Choosing an algorithm: the decision map

Put it all together and algorithm selection becomes a short series of questions about your data’s shape. The map below traces them left to right: start from what you have (a table of n rows and p columns), fit a linear baseline, and move rightward only when the data demands more — to the distance-based models if the set is small and you scale, to trees and boosting as the tabular default, and out to neural networks only for unstructured data like images and text. Each node marks the one thing you must remember about it: whether it needs scaling, and its key hyperparameter.

The six badges mark the decisions and traps. Start with a linear baseline (1) — scale it, tune C, and sometimes it simply wins. Lasso does feature selection (2) by zeroing weak coefficients. kNN and SVM must be scaled (3) — the single most common way to break them. Trees ignore scaling (4) but memorize the training set if you don’t prune. Gradient boosting usually wins on tabular data (5) and needs no scaling. And neural networks are for unstructured data (6) — images, text, audio — not ordinary tables, where boosting beats them at a fraction of the cost. That last badge is the bridge to the deep-learning lessons.

Here is the same logic as a lookup table — problem shape in, algorithm out:

And the single most practical table in this lesson — does it need scaling? — because forgetting it is the number-one silent bug in applied ML:

---

Hands-on lab

You will run the full comparison end to end on a real dataset, then pull apart why the models rank as they do — scaling, feature importances, coefficients, Lasso-style selection, and an overfitting tree. Everything runs on the venv from the top of the lesson (scikit-learn, numpy, pandas installed). Create algos.py and build it up step by step; every output below is exact and reproducible (all models are seeded).

Step 1 — Load a real dataset and split it.

import time
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier

bc = load_breast_cancer()
X, y = bc.data, bc.target
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
print(f"shape {X.shape} | classes {list(bc.target_names)} | benign rate {y.mean():.3f}")

shape (569, 30) | classes ['malignant', 'benign'] | benign rate 0.627

What just happened: a real medical dataset — 569 tumours, 30 numeric measurements, a binary malignant/benign target — split with stratify=y so train and test share the 62.7% benign rate. The 30 features live on wildly different scales (areas in the thousands, smoothness near 0.1), which is exactly what makes the scaling story visible.

Step 2 — A fair comparison: each algorithm in a Pipeline, scaled only if it needs it.

models = [
    ("LogisticRegression",   make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),   "yes"),
    ("kNN (k=7)",            make_pipeline(StandardScaler(), KNeighborsClassifier(7)),             "yes"),
    ("SVM (rbf)",            make_pipeline(StandardScaler(), SVC()),                                "yes"),
    ("GaussianNB",           GaussianNB(),                                                          "no"),
    ("DecisionTree",         DecisionTreeClassifier(random_state=0),                                "no"),
    ("RandomForest",         RandomForestClassifier(n_estimators=300, random_state=0),              "no"),
    ("HistGradientBoosting", HistGradientBoostingClassifier(random_state=0),                        "no"),
]
print(f"{'model':22} | {'test':>5} | {'CV':>5} | {'fit ms':>7} | scaled")
for name, mdl, scaled in models:
    mdl.fit(Xtr, ytr)                                    # warm up (ignore first timing)
    t0 = time.perf_counter(); mdl.fit(Xtr, ytr); fit_ms = (time.perf_counter() - t0) * 1000
    acc = mdl.score(Xte, yte)
    cv = cross_val_score(mdl, X, y, cv=5).mean()
    print(f"{name:22} | {acc:>5.3f} | {cv:>5.3f} | {fit_ms:>7.1f} | {scaled}")

model                  |  test |    CV |  fit ms | scaled
LogisticRegression     | 0.958 | 0.981 |     1.9 | yes
kNN (k=7)              | 0.958 | 0.970 |     0.5 | yes
SVM (rbf)              | 0.958 | 0.974 |     1.6 | yes
GaussianNB             | 0.923 | 0.939 |     0.4 | no
DecisionTree           | 0.902 | 0.917 |     2.3 | no
RandomForest           | 0.944 | 0.963 |   200.1 | no
HistGradientBoosting   | 0.937 | 0.965 |   307.0 | no

What just happened: seven algorithms, one split, each wrapped in a Pipeline that scales it only if it needs scaling. Logistic regression tops the cross-validated ranking (0.981) because this data is near-linearly separable — the simplest model wins, and it fits in under 2 ms. Your fit ms will differ; the ratio (ensembles ~100× the linear models) is the durable point.

Step 3 — Prove that kNN/SVM break on unscaled data while the tree does not care. Simulate real-world unit mismatch by multiplying each column by a random power of ten, then fit with and without a scaler:

rng = np.random.default_rng(0)
factors = 10.0 ** rng.integers(-3, 4, size=X.shape[1])       # per-column unit mismatch
Xtr_u, Xte_u = Xtr * factors, Xte * factors
for name, mk in [("DecisionTree", lambda: DecisionTreeClassifier(random_state=0)),
                 ("kNN (k=7)",   lambda: KNeighborsClassifier(7)),
                 ("SVM (rbf)",   lambda: SVC())]:
    raw = mk().fit(Xtr_u, ytr).score(Xte_u, yte)
    scaled = make_pipeline(StandardScaler(), mk()).fit(Xtr_u, ytr).score(Xte_u, yte)
    print(f"  {name:14} unscaled={raw:.3f}  scaled={scaled:.3f}")

  DecisionTree   unscaled=0.902  scaled=0.902
  kNN (k=7)      unscaled=0.937  scaled=0.958
  SVM (rbf)      unscaled=0.937  scaled=0.958

What just happened: the tree is identical unscaled and scaled (0.902 both) — it splits on thresholds, immune to the unit mismatch. kNN and SVM both recover accuracy when scaled (0.937 → 0.958), because scaling restores the distance metric the mangled units had distorted. On this data the drop is moderate because the big-scale features are also informative; on data where the loud feature is noise, the same mistake collapses kNN to near-random (you saw 0.53 in Part 1). The rule stands: scale the distance-based models, never bother for trees.

Step 4 — Pull the forest’s feature_importances_.

rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(Xtr, ytr)
for i in np.argsort(rf.feature_importances_)[::-1][:6]:
    print(f"  {bc.feature_names[i]:24} {rf.feature_importances_[i]:.3f}")

  worst perimeter          0.145
  worst radius             0.134
  worst concave points     0.121
  worst area               0.101
  mean concave points      0.100
  mean concavity           0.066

What just happened: the forest ranks the “worst” (largest) tumour measurements as most predictive — clinically sensible, and free interpretability. Remember the trap from earlier: had two of these features been near-duplicates, they’d split the credit and each look weaker than it is.

Step 5 — Linear coefficients, and Lasso-style L1 feature selection. Pull the logistic model’s strongest signed coefficients, then use an L1 penalty to zero features out — the classification cousin of Lasso:

scaler = StandardScaler().fit(Xtr)
Xtr_s, Xte_s = scaler.transform(Xtr), scaler.transform(Xte)
l2 = LogisticRegression(max_iter=5000).fit(Xtr_s, ytr)
coef = l2.coef_.ravel()
print("strongest coefficients (negative → pushes toward 'malignant'):")
for i in np.argsort(np.abs(coef))[::-1][:5]:
    print(f"  {bc.feature_names[i]:24} {coef[i]:+.3f}")

print("L1 penalty zeros weak features as C shrinks (feature selection):")
for C in [1.0, 0.1, 0.05, 0.02]:
    m = LogisticRegression(solver="saga", l1_ratio=1.0, C=C, max_iter=20000,
                           random_state=0).fit(Xtr_s, ytr)      # l1_ratio=1 → pure L1
    nz = int(np.sum(np.abs(m.coef_.ravel()) > 1e-6))
    print(f"  C={C:>4} | nonzero features={nz:>2}/30 | test acc={m.score(Xte_s, yte):.3f}")

strongest coefficients (negative → pushes toward 'malignant'):
  radius error             -1.213
  worst texture            -1.107
  mean concave points      -1.106
  worst radius             -0.995
  worst concave points     -0.925

L1 penalty zeros weak features as C shrinks (feature selection):
  C= 1.0 | nonzero features=14/30 | test acc=0.958
  C= 0.1 | nonzero features= 6/30 | test acc=0.937
  C=0.05 | nonzero features= 4/30 | test acc=0.951
  C=0.02 | nonzero features= 3/30 | test acc=0.916

What just happened: the linear model’s coefficients are signed and readable — all the top ones are negative on “large/irregular tumour” features, meaning higher values push toward malignant, which matches medical intuition. Then the L1 penalty performs feature selection: as C shrinks (stronger regularization), it drives more coefficients to exactly zero — from 14 nonzero features down to 3 — and accuracy holds around 0.95 until, at C=0.02, it has thrown away too much and drops to 0.916. This is Lasso’s zeroing behaviour in a classifier: regularize and select features in one step.

Version note: on scikit-learn 1.9 the clean way to get a pure-L1 logistic model is solver="saga" with l1_ratio=1.0 (the penalty= argument is deprecated as of 1.8). On older versions use penalty="l1", solver="liblinear", which does the same thing.

Step 6 — Watch an unpruned tree memorize the training set.

for d in [2, 3, None]:
    t = DecisionTreeClassifier(max_depth=d, random_state=0).fit(Xtr, ytr)
    print(f"  max_depth={str(d):>4} | train={t.score(Xtr, ytr):.3f} test={t.score(Xte, yte):.3f}")

  max_depth=   2 | train=0.932 test=0.888
  max_depth=   3 | train=0.977 test=0.916
  max_depth=None | train=1.000 test=0.902

What just happened: the unbounded tree hits a perfect 1.000 on training — it has memorized every case — yet scores only 0.902 on unseen data, worse than the pruned max_depth=3 tree at 0.916. A perfect training score next to a mediocre test score is the unmistakable signature of overfitting, and the cure is pruning (max_depth, min_samples_leaf). You now have the whole toolkit: a fair comparison, the scaling boundary, tree importances, linear coefficients, L1 selection, and the overfitting fingerprint — the exact instincts you need to pick and diagnose a model.

---

Common mistakes and troubleshooting

Four of these deserve extra words, because they are the ones that cost the most trust.

1. Forgetting to scale a distance-based model is silent. Nothing raises, nothing warns — kNN and SVM will happily return a number, and the number is just quietly worse than it should be, or catastrophically worse if a loud feature is uninformative. The only reliable defence is structural: put every kNN and SVM inside a Pipeline that begins with a StandardScaler, so you cannot forget. Trees, by contrast, are immune — and this asymmetry is worth internalizing, because it means “should I scale?” has a crisp per-model answer, not a vibe.

2. Accuracy lies on imbalanced data. If 95% of your rows are the negative class, a model that always predicts “negative” scores 95% accuracy while catching none of the positives — useless for fraud, disease, or churn, where the rare class is the whole point. In one imbalanced test, a default logistic model scored 0.974 accuracy but caught only 59% of the rare positives; setting class_weight='balanced' (which up-weights the rare class) traded a little accuracy for a recall jump to 81%. When classes are imbalanced, stop looking at accuracy, use class_weight='balanced' and precision/recall, and remember that predict_proba plus a tuned threshold is often the real fix — lowering the decision threshold from 0.5 to 0.15 raised the rare-class catch rate from 59% to 75% in the same test.

3. feature_importances_ is not a causal story. It is a description of how the model used the features, computed from split structure, and it splits credit across correlated features (you watched a feature’s importance halve when duplicated). Reading it as “this feature causes the outcome” is how analysts embarrass themselves in front of domain experts. Importances rank what the model leaned on; causation requires an experiment, not a forest.

4. predict versus predict_proba. predict returns a hard label using a 0.5 threshold; predict_proba returns the underlying probabilities. Whenever you need to rank customers by risk, tune a threshold, plot an ROC curve, or handle imbalance, you want predict_proba. A surprising amount of production ML value comes not from a better model but from taking its probabilities and choosing a threshold that matches the business cost of a false positive versus a false negative — a knob predict hides from you entirely.

---

Cheat-sheet

---

Interview and exam questions

Q: Why do kNN and SVM need feature scaling but decision trees don’t? A: kNN and SVM decide by distance — kNN by straight-line distance to neighbours, SVM through a distance-based kernel — so a feature on a large scale (area in the thousands) dominates the metric and drowns out small-scale features. A decision tree instead asks “is feature > threshold,” and the answer is unchanged if you multiply the feature by 1000 or add 100 to it, so trees (and their ensembles) are scale-invariant. The practical rule: always put a StandardScaler before kNN/SVM/regularized-linear/neural-net models; never bother for any tree model.

Q: Explain L1 vs L2 regularization and when you’d choose each. A: Both add a penalty on coefficient size to fight overfitting. L2 (Ridge) penalizes the sum of squared coefficients and shrinks them all toward zero smoothly but never to zero — it keeps every feature, just tamed, and is the safe default. L1 (Lasso) penalizes the sum of absolute coefficients and drives weak ones exactly to zero, so it performs automatic feature selection and yields a sparse, interpretable model. Choose Ridge when you want to keep all features but stabilize them; choose Lasso when you want a small subset of features. ElasticNet blends both, which handles correlated feature groups better than pure Lasso.

Q: What is C in logistic regression and SVM, and which direction does it regularize? A: C is the inverse of regularization strength — the counter-intuitive part. A small C means strong regularization (small coefficients, simpler model, may underfit); a large C means weak regularization (the model fits freely, coefficients grow, may overfit). On breast cancer, raising C from 0.001 to 100 grew the total coefficient size from 1.4 to 95 and moved the model from underfit (0.881) through a sweet spot to slight overfit. It’s the mirror image of alpha in Ridge/Lasso, where higher means stronger.

Q: How does a random forest differ from gradient boosting? A: Both are ensembles of decision trees, but they build the crowd differently. A random forest trains many deep trees independently and in parallel, each on a random bootstrap sample with random feature subsets, then averages their votes — averaging cancels the individual trees’ errors, giving a robust, low-tuning model. Gradient boosting builds trees sequentially, each new tree trained to correct the previous ensemble’s residual errors, with a learning_rate controlling the step size. Boosting is usually more accurate on tabular data but needs more careful tuning and is less parallelizable. Rule of thumb: random forest for a robust default, gradient boosting (HistGB / XGBoost / LightGBM) for maximum accuracy.

Q: You have a tabular dataset. Should you start with a deep neural network? Why or why not? A: No. On ordinary tabular data, gradient-boosted trees generally match or beat deep neural networks, train far faster, and need almost no feature scaling or architecture design. In a fair comparison on a non-linear tabular set, HistGradientBoosting scored 0.828 while a neural network reached only 0.802 at three times the fit time. Neural networks earn their keep on unstructured data — images, text, audio — where features can’t be hand-engineered and deep architectures learn representations. For a spreadsheet with named columns, start with a linear baseline and gradient boosting; reach for a neural net only if the data is unstructured.

Q: A decision tree gets 100% training accuracy and 85% test accuracy. What’s wrong and how do you fix it? A: It’s overfitting — the tree grew until every training point sat in its own pure leaf, memorizing the training set (including its noise) rather than learning generalizable rules. The 15-point gap between train and test is the signature. Fix by pruning: cap max_depth, raise min_samples_leaf (so leaves can’t be carved out for single outliers), or set ccp_alpha for cost-complexity pruning. Better still, use an ensemble — a random forest or gradient boosting averages/corrects across many trees and generalizes far better than any single one.

Q: What is the curse of dimensionality, and which model here suffers most from it? A: In high-dimensional space, distances between points become nearly uniform — everything is roughly equidistant from everything, so “nearest” loses meaning. kNN suffers most, because it relies entirely on distance: in a demonstration, adding pure-noise features dropped kNN’s accuracy from 0.921 (5 features) to 0.591 (505 features), barely above chance. Regularized linear models and tree ensembles cope far better because they can learn to ignore irrelevant features, whereas every feature contributes to kNN’s distance. The remedy for kNN is dimensionality reduction or feature selection first, and few, informative, scaled features.

Q: When would you reach for Naive Bayes over the other classifiers? A: As a fast baseline — it trains in milliseconds with no iterative optimization — and especially for text classification (spam, topic, sentiment), where MultinomialNB over a bag-of-words or TF-IDF matrix is a strong, cheap default. It assumes features are independent given the class, which is usually false but often works well enough. Reach for it when you want a quick benchmark to beat, when training speed matters, or when you have very little data. It won’t usually top a tuned gradient-boosting model on structured data.

Q: Your model reports 96% accuracy but stakeholders say it’s useless for catching fraud. What happened? A: The classes are imbalanced — if fraud is 4% of transactions, a model that predicts “not fraud” for everything scores 96% accuracy while catching zero fraud. Accuracy is the wrong metric here. Look at precision and recall on the fraud class, set class_weight='balanced' to up-weight the rare class during training, and use predict_proba with a tuned threshold rather than the default 0.5 — lowering the threshold catches more fraud (higher recall) at the cost of more false alarms, and you pick the point that matches the business cost of each error type.

Q: How do you interpret a linear model versus a random forest, and what’s the trap with tree importances? A: A linear model exposes signed coefficients — magnitude (on scaled data) is strength, sign is direction — precise and directly tied to the model’s arithmetic. A random forest exposes feature_importances_, how much each feature reduced impurity across splits. The trap: importances are computed from model structure, split across correlated features, and are not causal — duplicating a strong feature roughly halves its apparent importance though nothing changed. For robust attribution use permutation importance or SHAP values; never read tree importances as causation.

Q (coding): Write a fair, leakage-proof comparison of logistic regression, SVM and gradient boosting on a dataset, scaling only where needed. A:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import HistGradientBoostingClassifier

models = {
    "logreg": make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),  # scale
    "svm":    make_pipeline(StandardScaler(), SVC()),                              # scale
    "histgb": HistGradientBoostingClassifier(),                                    # no scale
}
for name, mdl in models.items():
    print(name, round(cross_val_score(mdl, X, y, cv=5).mean(), 3))

Each model is in a Pipeline (so scaling is fit per-fold, no leakage), scaled only where the algorithm needs it, and evaluated with the same 5-fold cross-validation — a genuinely fair fight.

Q: What does the kernel trick let an SVM do, and what are its two key hyperparameters? A: The kernel trick lets an SVM draw a non-linear boundary by implicitly mapping the data into a higher-dimensional space where it becomes linearly separable — without ever computing those high-dimensional coordinates, using a kernel function as a shortcut. The rbf kernel is the flexible default. The two key knobs are C (inverse regularization / margin softness — high overfits, low underfits) and gamma (the reach of each point’s influence — high makes wiggly local boundaries that overfit, low makes smooth global ones), and they must be tuned together. The catch: SVMs scale poorly to large n (roughly quadratic-to-cubic training cost).

---

Key takeaways

• Choosing an algorithm is choosing an assumption about your data’s shape. Linear models assume a straight boundary; kNN assumes neighbours share labels; SVM assumes a wide separating margin; trees assume axis-aligned rectangles. When the assumption fits, the model is fast, accurate and often interpretable; when it doesn’t, no tuning saves it. Ask two questions of every dataset: is it roughly linear, and does the model measure distance?
• Start with a linear baseline — sometimes it wins. Logistic/linear regression is fast, interpretable, and hard to beat on near-linear data (it topped a neural network on breast cancer at 1/60th the fit time). Regularize it: Ridge (L2) shrinks all weights, Lasso (L1) zeros weak ones for automatic feature selection (10 features down to 4 as alpha rose), ElasticNet blends both. Remember C is inverse strength — small C regularizes harder.
• Scale the distance-based models or watch them break. kNN and SVM are pure distance, so an unscaled large-range feature owns the metric and accuracy collapses — always StandardScaler them. kNN’s k and SVM’s C/gamma trade bias against variance, kNN drowns under the curse of dimensionality (0.92 → 0.59 with noise features), and SVM doesn’t scale past tens of thousands of rows.
• Trees need no scaling but overfit if unpruned. A decision tree splits on thresholds (scale-invariant) and is fully interpretable, but grown unbounded it memorizes the training set — 100% train, mediocre test. Cap max_depth/min_samples_leaf. A single tree is rarely best; its real value is as an interpretable model and as the building block of ensembles.
• Ensembles dominate tabular ML, and gradient boosting usually wins. Random forests average many independent trees for a robust, low-tuning default; gradient boosting builds trees sequentially, each fixing the last’s errors, and is typically the most accurate tabular model. Start with HistGradientBoostingClassifier (sklearn-native, fast, handles NaN); reach for XGBoost/LightGBM/CatBoost to squeeze the last points. Neither needs scaling.
• A neural network usually does NOT beat boosting on tabular data. In a fair non-linear comparison, HistGradientBoosting (0.828) edged the neural net (0.802) at a third of the fit time — a robust, published finding. Neural networks win on unstructured data (images, text, audio) where features can’t be hand-crafted; on spreadsheets, boosting wins cheaper. That is the bridge to the deep-learning lessons.
• Run a fair fight, and read the model honestly. Compare on one split with each model in a Pipeline scaled only where needed, and judge by cross-validated accuracy, not a single split. Linear coefficients are the gold standard of interpretability; tree feature_importances_ are useful but split credit across correlated features and are never causal — use permutation importance or SHAP when it matters. And on imbalanced data, abandon accuracy for precision/recall, class_weight='balanced', and a tuned predict_proba threshold.

---

This is Part 2 of the scikit-learn arc. You can now match an algorithm to a dataset’s shape, know which models demand scaling and which shrug it off, tune the two or three hyperparameters that matter for each, and run a fair comparison that names a real winner instead of a favourite. Part 3 completes the arc: it takes the models you just met and the Pipeline from Part 1 and puts them inside cross-validation and hyperparameter search, so you can tune preprocessing and model together, honestly, and know that the score you report is the score you’ll get in production. For the ideas underneath these models, revisit ML fundamentals: supervised vs unsupervised; for the array math that every one of them runs on, NumPy arrays, broadcasting and vectorization; and for the complexity intuition behind “SVM doesn’t scale to large n” and “kNN is expensive at predict time,” algorithms: search, sort and complexity. The habit from Part 1 still holds every model you built here: everything inside one Pipeline, fit on train only.