Python Lesson 51 of 71

Computer Vision Part 1 — Image Processing with OpenCV & Pillow

Point a camera at the world and what reaches your program is not a picture — it is a wall of numbers. A 1920×1080 colour photo is 1920 × 1080 × 3 ≈ 6.2 million bytes, each an integer from 0 to 255, and every computer-vision technique in this lesson is just arithmetic on that block of bytes. Blur an image and you are averaging neighbouring numbers. Detect an edge and you are looking for places where the numbers jump. Find an object and you are grouping numbers that cluster. There is no magic layer underneath; there is NumPy, all the way down.

That is the single most useful idea in this whole field, and most tutorials bury it. An image is an ndarray — the exact same object you already met, with a shape, a dtype, strides and views. Grasp that, and OpenCV’s thousand functions become “operations that transform an array into another array,” and you can debug them with the array skills you already have: print the shape, print the dtype, print a corner of pixels.

This lesson gives you the working half of computer vision — the image-processing pipeline that sits in front of every model: load → preprocess → detect/measure. You will build it on a synthetic scene you can regenerate anywhere (no dataset to download), and you will run every stage headlessly — saving results to disk with imwrite, never popping a window — because that is how vision code actually runs on a server. The next lesson wires the output of this pipeline into a neural network; here we master the pixels first.


Why this matters

Every real vision system, from a barcode scanner to a self-driving car’s lane detector, is the same skeleton: load an image, clean it up, find the thing you care about, measure it, act. The middle steps — grayscale, blur, threshold, edges, morphology, contours — are so universal that they have standard names and standard functions. Learn the pipeline once and you can read 90% of the OpenCV code on the internet.

The reason a beginner stalls is almost never the algorithm. It is the plumbing: an image loads with its colours swapped and every downstream step looks subtly wrong; imshow crashes on the server with a cryptic “function not implemented”; imread silently returns None for a typo’d path and the traceback points at the wrong line; a uint8 add wraps 255 + 10 around to 9 and the brightness filter goes dark. These are not deep problems, but they cost hours because nobody warns you. This lesson front-loads every one of them.

Here is the pipeline you are about to build, drawn as the real sequence of array transformations. Read it left to right: an image enters as a BGR array (trap #1), gets normalised to grayscale or HSV and blurred, is segmented by threshold or Canny edges, cleaned with morphology, traced into contours, measured with bounding boxes, and written back to disk — because on a headless box you save, you do not display.

Computer-vision image-processing pipeline diagram: an image loaded by cv2.imread as an H×W×3 uint8 BGR array (with the BGR-not-RGB red/blue swap flagged as a trap), converted with cvtColor to grayscale or HSV and denoised with a Gaussian blur, segmented by an Otsu automatic threshold (value 79) or Canny edge detection with 50/150 hysteresis thresholds, cleaned with morphological opening and closing, traced with findContours using RETR_EXTERNAL, measured into bounding boxes and centroids for four detected shapes, and saved to disk with cv2.imwrite because imshow has no window on a headless server

The badges mark the six places beginners get hurt: the BGR≠RGB swap at load (1), choosing grayscale vs HSV (2), blurring before you threshold (3), picking segmentation thresholds (4), morphology-then-contours to detect blobs (5), and saving instead of showing on a server (6). We hit each one in order.

Pipeline stage Typical function(s) Input → output What it buys you
Load cv2.imread, Image.open file → H×W×3 uint8 (BGR!) Pixels in memory as an array
Colour convert cv2.cvtColor BGR → gray / HSV One channel for algorithms; hue for colour masks
Filter / denoise GaussianBlur, medianBlur array → smoother array Kills noise so later steps don’t misfire
Threshold / edges threshold (Otsu), Canny gray → binary mask / edge map Splits “interesting” pixels from background
Morphology erode, dilate, morphologyEx mask → cleaner mask Removes specks, fills holes
Contours findContours, boundingRect mask → list of shapes + boxes Detects and measures objects
Output cv2.imwrite, Image.save array → file Persist results (headless-safe)

An image is a NumPy array

Load an image with OpenCV and inspect it the way you would inspect any array. A colour image is three-dimensional — (height, width, channels) — and the dtype is uint8, an unsigned 8-bit integer holding exactly 0255. That range is not arbitrary: 0 is “no light in this channel,” 255 is “full brightness,” and 8 bits is what camera sensors and display hardware settled on decades ago.

import cv2
import numpy as np

# synthesize a 120x240 colour image: three vertical blocks
img = np.zeros((120, 240, 3), dtype=np.uint8)   # (H, W, C) of zeros = black
img[:, :80]    = (0, 0, 255)   # left  block
img[:, 80:160] = (0, 255, 0)   # middle block
img[:, 160:]   = (255, 0, 0)   # right block
cv2.imwrite("blocks.png", img)   # write to disk (headless-safe)

print(type(img).__name__)   # ndarray      -- it's just a NumPy array
print(img.shape)            # (120, 240, 3)  H=120 rows, W=240 cols, 3 channels
print(img.dtype)            # uint8         every pixel 0-255
print(img.min(), img.max()) # 0 255
print(img[0, 0])            # [  0   0 255]  the top-left pixel's 3 channel values

Read those attributes and you already know how to manipulate the image: it is 120 * 240 * 3 = 86,400 bytes in one contiguous buffer, indexed as img[row, col, channel]. Slicing works exactly as it does for any array — img[:, :80] is “all rows, first 80 columns,” a view onto the same buffer (the aliasing rule from the NumPy lesson applies: mutate a slice, mutate the original). A pixel is a length-3 vector; a row is a (240, 3) array; the whole thing is a stack you can broadcast over.

Attribute Colour image Grayscale image Meaning
shape (H, W, 3) (H, W) Grid size; colour has a channel axis, gray does not
ndim 3 2 Number of axes
dtype uint8 uint8 0–255 per channel
img[y, x] [b, g, r] (3 values) one scalar A single pixel
img[y, x, 0] one channel value Indexing into the channel axis
img.size H*W*3 H*W Total byte count

Grayscale drops the channel axis

Convert to grayscale and the channel axis disappears — a grayscale image is genuinely two-dimensional, (H, W), one intensity per pixel. This trips people constantly: code that indexes img[y, x, 0] explodes the moment the image is grayscale, because there is no third axis to index.

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
print(gray.shape)   # (120, 240)   NO channel axis -- it's 2-D
print(gray.ndim)    # 2
print(gray[10, 10], gray[10, 100], gray[10, 200])   # 76 150 29

Those three numbers — 76 150 29 — are the grayscale values of the red, green and blue blocks, and they reveal how grayscale is computed. It is not the average of the channels; it is a luminosity-weighted sum, Y = 0.299·R + 0.587·G + 0.114·B, because human eyes are far more sensitive to green than to blue. Pure red (255) becomes 0.299 × 255 ≈ 76, pure green becomes 0.587 × 255 ≈ 150, pure blue only 0.114 × 255 ≈ 29. Green looks brightest to us, so it dominates the gray value.

The dtype matters — a lot

Because pixels are uint8, all the dtype pitfalls from NumPy apply, and one bites immediately: uint8 arithmetic wraps around. 255 + 10 is not 265 (which doesn’t fit in 8 bits) — it wraps modulo 256 to 9. Brighten an image with plain + and the bright regions turn dark.

px = np.array([250, 100, 50], dtype=np.uint8)
print(px + np.uint8(30))          # [ 24 130  80]   250+30 wrapped to 24!
# safe options:
print(cv2.add(px, np.array([30, 30, 30], np.uint8)))   # [255 130 80]  SATURATES at 255
print(np.clip(px.astype(np.int16) + 30, 0, 255).astype(np.uint8))  # [255 130 80]

cv2.add saturates (clamps at 0 and 255) instead of wrapping, which is almost always what you want for image math. The other safe pattern is to widen to int16/float, do the arithmetic, np.clip to [0, 255], and cast back to uint8. We return to this in troubleshooting because it is the single most common “why did my filter darken the image” bug.

dtype Range Where it shows up Danger
uint8 0–255 The default for loaded images Wraps on overflow; must clip
uint16 0–65535 16-bit medical / RAW images Rarely displayed directly
float32 any After normalising to [0,1] for ML Must scale back to uint8 to save
float64 any Intermediate math (blur, warp) imwrite truncates; PIL refuses it

The BGR-vs-RGB gotcha

This one deserves its own section because it wastes more beginner hours than any algorithm. OpenCV loads and stores images in B, G, R channel order — blue first — for historical reasons (the Windows bitmap format it grew up with). Almost everything else in the Python world — Pillow, matplotlib, imageio, PyTorch, the entire web — uses R, G, B. So the instant you move an image between OpenCV and any other library without converting, red and blue swap: skies go orange, faces go blue, and no error is raised because the array shape is identical.

Watch the swap happen with real pixel values. The same red pixel is [0, 0, 255] in OpenCV’s BGR and [255, 0, 0] in RGB — the channels are literally reversed:

loaded = cv2.imread("blocks.png")     # OpenCV gives you BGR
print(loaded[10, 10])                  # [  0   0 255]   a RED pixel, stored B=0 G=0 R=255

rgb = cv2.cvtColor(loaded, cv2.COLOR_BGR2RGB)   # THE FIX
print(rgb[10, 10])                     # [255   0   0]   same pixel, now R=255 G=0 B=0
print(np.array_equal(loaded[..., ::-1], rgb))   # True   BGR->RGB is just reversing channels

cv2.cvtColor(img, cv2.COLOR_BGR2RGB) is the fix, and since the conversion is just a channel reversal you will also see the NumPy shortcut img[..., ::-1]. The rule to internalise: convert at the boundary. Keep images in BGR while you are inside OpenCV, and convert to RGB only when you hand the array to Pillow, matplotlib, or a model. If you are using matplotlib to display an image, plt.imshow expects RGB — feed it a raw cv2 array and you get the blue-face bug in its purest form.

import matplotlib
matplotlib.use("Agg")            # headless backend: render to file, no window
import matplotlib.pyplot as plt

fig, ax = plt.subplots(1, 2, figsize=(6, 2))
ax[0].imshow(loaded); ax[0].set_title("cv2 array, WRONG")   # blue<->red swapped
ax[1].imshow(rgb);    ax[1].set_title("cvtColor, right")
plt.savefig("bgr_vs_rgb.png")    # save, don't plt.show() on a server
Library Channel order Array shape Notes
OpenCV (cv2) BGR (H, W, 3) The odd one out; convert on the way out
Pillow (PIL) RGB via np.asarray(H, W, 3) .size is (W, H) — transposed!
matplotlib RGB expects (H, W, 3) or (H, W) imshow shows blue faces if fed BGR
PyTorch / TF RGB often (C, H, W) Also transposes the channel axis to the front
NumPy raw whatever you put (H, W, C) img[..., ::-1] flips channel order

Pillow vs OpenCV: two libraries, two jobs

Python has two dominant image libraries and they are good at different things. Pillow (PIL) is the friendly, Pythonic one: open a file, resize it, crop it, convert its format, save it. It thinks in Image objects and (width, height). OpenCV (cv2) is the computer-vision workhorse: thresholding, edges, contours, camera calibration, optical flow, and thousands of algorithms Pillow has never heard of. It thinks in NumPy arrays and (height, width). Most real projects use both — Pillow for I/O and simple transforms, OpenCV for the vision.

Pillow (PIL) OpenCV (cv2)
Mental model Image object NumPy ndarray
Dimension order img.size = (W, H) img.shape = (H, W, C)
Channel order RGB BGR
Strengths Open/save, format conversion, resize, crop, rotate, thumbnails, EXIF, text Filtering, thresholding, edges, contours, morphology, video, ML preprocessing
Weaknesses No CV algorithms Clunky I/O, no format niceties
Install pip install pillow pip install opencv-python-headless
Grayscale .convert("L") → 2-D cvtColor(..., BGR2GRAY) → 2-D
Typical role Load / save / simple edits The vision pipeline

Pillow: the I/O and simple-edits toolkit

Pillow’s API reads like plain English. Image.open is lazy — it reads the header (size, format) but not the pixels until you actually need them, which makes opening a directory of thumbnails cheap.

from PIL import Image, ImageOps

im = Image.open("blocks.png")
print(im.size, im.mode, im.format)   # (240, 120) RGB PNG   <- size is (W, H)!

im.resize((200, 150))                # (W, H) -- note the order
im.crop((40, 40, 140, 160))          # box = (left, top, right, bottom) -> 100x120
im.rotate(30, expand=True)           # rotate CCW; expand grows the canvas to fit
thumb = im.copy(); thumb.thumbnail((100, 100))   # IN PLACE, preserves aspect ratio
print(thumb.size)                    # (100, 50)   fit inside 100x100, aspect kept
im.convert("L").save("gray.png")     # to grayscale, then save
im.save("out.jpg", quality=85)       # format inferred from extension

The (W, H) versus (H, W) disagreement between Pillow and NumPy is a genuine trap: im.size is (width, height) but np.asarray(im).shape is (height, width, channels). When you convert between them, the two numbers swap places.

Pillow method Does Argument order
Image.open(path) Lazily open (reads header only)
im.size (width, height) note: not (H, W)
im.resize((w, h)) Resize to exact size (W, H)
im.crop((l, t, r, b)) Crop to a box left, top, right, bottom
im.rotate(deg, expand=) Rotate CCW; expand fits canvas degrees
im.thumbnail((w, h)) Shrink in place, keep aspect max box
im.convert("L") / "RGB" / "RGBA" Change mode/channels mode string
im.transpose(Image.Transpose.FLIP_LEFT_RIGHT) Flip / rotate by 90° steps enum constant
im.save(path, quality=) Save; format from extension
ImageOps.exif_transpose(im) Apply the EXIF orientation tag

EXIF orientation is worth a warning. Phone cameras almost always store pixels in one physical orientation and add an EXIF tag saying “rotate this 90° when displaying.” If you read the raw pixels and ignore the tag, portrait photos come out sideways. ImageOps.exif_transpose(im) bakes the tag into the pixels so the array matches what a human sees — call it right after Image.open on any photo from a real camera.

Formats and modes: lossy, lossless, alpha

A big part of Pillow’s job is format conversion, and the format you save to is a real decision, not a cosmetic one. PNG is lossless and keeps an alpha (transparency) channel — the right choice for masks, screenshots, line art, and anything with sharp edges or text. JPEG is lossy and has no alpha — right for photographs, wrong for masks (its block compression smears hard edges, and it will flatly refuse an RGBA image). WebP does both lossy and lossless with alpha at smaller sizes, and is the sensible modern default for the web. Saving is just choosing an extension; Pillow infers the encoder.

im.save("out.png")                  # lossless, keeps alpha
im.save("out.jpg", quality=85)      # lossy; quality 1-95; NO alpha channel
im.convert("RGB").save("out.webp")  # modern: small files, optional alpha
im.convert("RGBA").save("bad.jpg")  # OSError: cannot write mode RGBA as JPEG

That last line is a real traceback (OSError: cannot write mode RGBA as JPEG) and a common trip-up: convert to RGB before saving a transparent image as JPEG, or it errors.

Format Compression Alpha Best for Avoid for
PNG Lossless Yes Masks, screenshots, text, line art Huge photos (large files)
JPEG Lossy No Photographs Masks, sharp edges, transparency
WebP Lossy or lossless Yes Web images (small + alpha) Very old viewers
BMP None No Quick debugging dumps Anything shipped (uncompressed)
TIFF Lossless/none Yes Scientific, archival, 16-bit The web

Pillow tracks channel layout with a mode string, and convert() moves between modes. The mode decides the shape you get back from np.asarray, so it is the first thing to check when an array’s shape surprises you.

PIL mode Meaning np.asarray shape
"L" 8-bit grayscale (H, W)
"RGB" 3-channel colour (H, W, 3)
"RGBA" Colour + alpha (H, W, 4)
"P" Palette (indexed colour) (H, W) of indices
"1" 1-bit bilevel (black/white) (H, W)

Reading, writing, and the headless imshow reality

OpenCV reads and writes with imread/imwrite, and both have sharp edges. imread returns a BGR array; you can force grayscale at load time with a flag. Crucially, imread does not raise on a bad path — it returns None, and the crash comes later when you use the None:

img = cv2.imread("typo.png")     # file doesn't exist
print(img is None)               # True   -- NO exception here
print(img.shape)                 # AttributeError: 'NoneType' object has no attribute 'shape'

Always guard imread with if img is None: raise FileNotFoundError(path). This is the number-one “why is my traceback pointing at the wrong line” bug in OpenCV.

Call Returns Note
cv2.imread(path) BGR (H,W,3) uint8, or None Default IMREAD_COLOR; drops alpha
cv2.imread(path, cv2.IMREAD_GRAYSCALE) (H,W) uint8 Load straight to gray
cv2.imread(path, cv2.IMREAD_UNCHANGED) Keeps alpha → (H,W,4) For PNGs with transparency
cv2.imwrite(path, img) True/False Format from extension; needs uint8
Image.open(path) PIL Image (lazy, RGB) .load() to force-read
im.save(path) Format from extension

Now the part that surprises everyone deploying to production: cv2.imshow needs a graphical window server, and servers don’t have one. On your laptop imshow pops a window; in Docker, CI, or a cloud VM there is no display, and the headless OpenCV build (the opencv-python-headless package, which you should install on servers) raises:

cv2.error: OpenCV(4.x) .../highgui/src/window.cpp: error: (-2:Unspecified error)
The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or
Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and
pkg-config, then re-run cmake ... in function 'cvShowImage'

The fix is not to install a GUI — it is to stop trying to display and start saving. Every debugging step becomes cv2.imwrite("stage_03_edges.png", edges), and you open the files. That is why this entire lesson is headless: it is how vision code runs where it matters. (On some desktop builds imshow may silently pop a window even from the “headless” wheel — but never rely on it; write your code to save, and it runs everywhere.)

⚠️ Install opencv-python-headless, not opencv-python, on any server or CI runner. The two packages conflict if both are installed; pick one. The headless build is smaller, has no GUI dependencies, and forces the save-don’t-show discipline you want anyway.

Moving arrays between the libraries

Because both libraries ultimately speak NumPy, you convert freely — but you must fix the channel order at every crossing. np.asarray(pil_image) gives an RGB array; Image.fromarray(array) expects RGB. So the round-trip from OpenCV to Pillow and back is always a cvtColor sandwich:

cv = cv2.imread("blocks.png")                       # BGR array
pil = Image.fromarray(cv2.cvtColor(cv, cv2.COLOR_BGR2RGB))   # -> RGB for PIL
back = cv2.cvtColor(np.asarray(pil), cv2.COLOR_RGB2BGR)      # -> BGR for cv2

# proof the same red pixel differs by channel order:
print(pil.getpixel((10, 10)))   # (255, 0, 0)    PIL: R, G, B
print(cv[10, 10])               # [  0   0 255]   cv2: B, G, R  (reversed)

Geometric operations

Geometry — resize, crop, flip, rotate, warp — is where you prepare an image for a model (which wants a fixed input size) or correct for how it was captured. Two ideas carry all of it: crop is just NumPy slicing, and resize forces you to choose an interpolation — the rule for inventing pixel values that didn’t exist before.

Resize and the interpolation choice

When you shrink or grow an image, the new pixel grid doesn’t line up with the old one, so OpenCV must interpolate. The flag you pass changes both speed and quality, and — critically — the right choice differs for shrinking versus growing.

img = cv2.imread("blocks.png")
print(img.shape)                                        # (120, 240, 3)

big  = cv2.resize(img, (480, 240), interpolation=cv2.INTER_CUBIC)   # (W, H)!
small = cv2.resize(img, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
print(big.shape, small.shape)                           # (240, 480, 3) (60, 120, 3)

Note the trap already lurking: cv2.resize takes the target size as (width, height), the opposite of shape’s (height, width). Passing img.shape[:2] directly gives you a transposed image.

Interpolation flag Speed Quality Use for
INTER_NEAREST Fastest Blocky Masks/labels (must not blend classes)
INTER_LINEAR Fast Good The default; general resizing
INTER_AREA Medium Best when shrinking Downscaling (avoids moiré/aliasing)
INTER_CUBIC Slow Smooth when growing Upscaling, 4×4 neighbourhood
INTER_LANCZOS4 Slowest Sharpest High-quality upscaling, 8×8

You can see the difference by upscaling a tiny image and counting how many distinct gray levels appear. INTER_NEAREST just copies existing values (few levels, blocky); the smooth methods invent in-between values (many levels, gradients):

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
tiny = cv2.resize(gray, (20, 15), interpolation=cv2.INTER_AREA)
for name, flag in [("NEAREST", cv2.INTER_NEAREST), ("LINEAR", cv2.INTER_LINEAR),
                   ("CUBIC", cv2.INTER_CUBIC), ("LANCZOS4", cv2.INTER_LANCZOS4)]:
    up = cv2.resize(tiny, (400, 300), interpolation=flag)
    print(f"{name:9s} distinct gray levels: {len(np.unique(up))}")
# NEAREST   distinct gray levels: 39
# LINEAR    distinct gray levels: 129
# CUBIC     distinct gray levels: 160
# LANCZOS4  distinct gray levels: 164

One hard rule: for a label/segmentation mask, always use INTER_NEAREST. A mask’s pixel values are class IDs (0=background, 1=cat, 2=dog); blending them with INTER_LINEAR produces a meaningless 1.5 that belongs to no class.

Crop, flip, rotate, and affine/perspective warps

Cropping needs no special function — it is a slice, img[y0:y1, x0:x1], and (being a basic slice) it returns a view, so .copy() it if you plan to draw on the crop without touching the original. Flipping mirrors along an axis. Rotation and general warps go through warpAffine (which takes a 2×3 matrix) and warpPerspective (a 3×3 matrix), because any rotate/scale/shear/translate is one matrix multiply per pixel coordinate.

crop  = img[20:100, 40:200]           # a VIEW: rows 20-99, cols 40-199 -> (80, 160, 3)
flip_h = cv2.flip(img, 1)             # 1=horizontal, 0=vertical, -1=both
flip_v = cv2.flip(img, 0)

h, w = img.shape[:2]
M = cv2.getRotationMatrix2D((w/2, h/2), 30, 1.0)   # centre, 30 deg CCW, scale 1.0
rotated = cv2.warpAffine(img, M, (w, h))           # M is 2x3
print(M.shape, rotated.shape)                      # (2, 3) (120, 240, 3)

# perspective: map 4 source corners to 4 destination corners (a 3x3 matrix)
src = np.float32([[0,0],[w,0],[0,h],[w,h]])
dst = np.float32([[20,10],[w-10,0],[0,h-5],[w-30,h-20]])
P = cv2.getPerspectiveTransform(src, dst)
warped = cv2.warpPerspective(img, P, (w, h))
print(P.shape)                                     # (3, 3)

Perspective warps are how you “de-skew” a photographed document: pick its four corners as src, map them to a clean rectangle as dst, and warpPerspective flattens it.

Both warps rest on one idea: a transformation matrix maps each output pixel’s coordinate back to a source coordinate, and OpenCV interpolates the colour there. An affine 2×3 matrix can translate, scale, rotate and shear, and it always keeps parallel lines parallel; a perspective 3×3 matrix additionally lets parallel lines converge, which is exactly what creates the illusion of depth (railway tracks meeting at the horizon). You almost never build these matrices by hand — helpers like getRotationMatrix2D and getPerspectiveTransform construct them — but knowing which one you need tells you which warp* function to call.

Transform Matrix Keeps parallel lines? Built by
Translate 2×3 Yes Manual [[1,0,tx],[0,1,ty]]
Scale 2×3 Yes [[sx,0,0],[0,sy,0]]
Rotate (about a point) 2×3 Yes getRotationMatrix2D(c, deg, scale)
General affine 2×3 Yes getAffineTransform(3 pts → 3 pts)
Perspective 3×3 No (they converge) getPerspectiveTransform(4 pts → 4 pts)
Operation Function Key parameter
Crop img[y0:y1, x0:x1] (slice) Returns a view
Resize cv2.resize(img, (w, h), interpolation=) Size is (W, H)
Flip cv2.flip(img, code) 1=horiz, 0=vert, -1=both
Rotate 90° cv2.rotate(img, cv2.ROTATE_90_CLOCKWISE) Lossless, no interpolation
Rotate any angle warpAffine(img, getRotationMatrix2D(c, a, s), (w,h)) 2×3 matrix
Translate/shear/scale warpAffine(img, M, (w,h)) Any 2×3 affine M
Perspective warpPerspective(img, getPerspectiveTransform(src,dst), (w,h)) 3×3 matrix

Colour spaces and thresholding

To find things by colour, raw BGR is a poor coordinate system: change the lighting and all three channels move together, so “green” spans a huge, awkward BGR region. HSV — Hue, Saturation, Value — fixes this by putting colour on one axis (hue) and brightness on another (value). “All the green pixels, whatever the lighting” becomes a simple hue range, which is why HSV is the standard space for colour-based masking.

hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
print(hsv[10, 10], hsv[10, 100], hsv[10, 200])   # [0 255 255] [60 255 255] [120 255 255]

Mind OpenCV’s HSV ranges, which are non-obvious because they are squeezed into uint8: Hue is 0179 (degrees halved, since 360 won’t fit in a byte), while Saturation and Value are 0255. Red sits at hue 0, green at 60, blue at 120 — half their usual degree values.

cv2.inRange builds a binary mask (255 where the pixel falls inside the HSV bounds, 0 elsewhere), and that mask is how you select or count colour:

lower = np.array([40, 50, 50])     # low green: hue 40, some saturation/brightness
upper = np.array([80, 255, 255])   # high green: hue 80
mask = cv2.inRange(hsv, lower, upper)
print(mask.shape, np.unique(mask))          # (120, 240) [  0 255]   a 2-D binary mask
print(int((mask > 0).sum()), "green px")    # 9600 green px
# apply the mask to keep only the green region:
green_only = cv2.bitwise_and(img, img, mask=mask)

One colour breaks the “single range” convenience: red wraps around the hue circle. Red sits at hue 0, but because the circle joins 179 back to 0, red pixels land at both ends — roughly hue 010 and 170179. A single inRange can’t span that gap, so to mask red you build two masks and OR them: cv2.inRange(hsv,(0,120,70),(10,255,255)) | cv2.inRange(hsv,(170,120,70),(179,255,255)). Every other colour is one contiguous band; only red needs the two-piece treatment.

Colour space cvtColor code Axes Best for
Grayscale COLOR_BGR2GRAY intensity Most algorithms (edges, threshold)
RGB COLOR_BGR2RGB R, G, B Handing off to PIL/matplotlib/ML
HSV COLOR_BGR2HSV H 0-179, S 0-255, V 0-255 Colour-based masking
LAB COLOR_BGR2LAB L, a, b Perceptual colour distance
HLS COLOR_BGR2HLS H, L, S Alternative to HSV

Thresholding: binary, Otsu, adaptive

Thresholding turns a grayscale image into a black-and-white mask: pixels above a cutoff become 255, the rest 0. The question is what cutoff, and there are three answers. A fixed threshold is a number you pick. Otsu picks the number for you by finding the split that best separates the histogram’s two peaks — ideal when foreground and background are distinct. Adaptive computes a different threshold for each small region, which is the only thing that works under uneven lighting.

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# fixed: everything brighter than 100 -> white
t1, binary = cv2.threshold(gray, 100, 255, cv2.THRESH_BINARY)
print(t1, np.unique(binary))          # 100.0 [  0 255]

# Otsu: pass 0 as the threshold and add the flag; it RETURNS the value it chose
t2, otsu = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
print("Otsu picked:", t2)             # Otsu picked: 76.0

# adaptive: per-region threshold, for uneven illumination
adaptive = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                 cv2.THRESH_BINARY, blockSize=11, C=2)

The most useful is Otsu: give it 0 for the threshold, add THRESH_OTSU, and it returns the cutoff it computed from the data — no magic number to tune. It assumes a bimodal histogram (two clear peaks), so it shines on “dark objects on a light background” and struggles on smooth gradients.

Method Call Threshold source Use when
Fixed binary threshold(g, T, 255, THRESH_BINARY) You pick T You know the cutoff
Inverse THRESH_BINARY_INV You pick T Dark objects, light background
Otsu threshold(g, 0, 255, THRESH_BINARY+THRESH_OTSU) Auto (from histogram) Bimodal, even lighting
Adaptive adaptiveThreshold(g, 255, ...) Per-region Uneven lighting, documents
To-zero THRESH_TOZERO You pick T Keep bright values, zero the rest

Filtering, convolution, and the bridge to CNNs

Filtering is how you smooth, sharpen, or extract features, and every filter here is a convolution: slide a small grid of weights (a kernel) over the image, and at each position compute the weighted sum of the neighbourhood. Change the weights and you change the effect. This is not an analogy for what a convolutional neural network does — it is literally the same operation. A CNN’s “learned filters” are kernels exactly like these, except the network discovers the weights instead of you writing them. Master hand-built kernels here and the next step into deep learning is a short one.

Blur: Gaussian for general noise, median for salt-and-pepper

A Gaussian blur replaces each pixel with a weighted average of its neighbours (nearby pixels count more), smoothing out fine noise. It is the standard “denoise before you threshold or detect edges” step. But for salt-and-pepper noise — random pure-white and pure-black specks — a Gaussian smears the specks instead of removing them; you want a median blur, which replaces each pixel with the median of its neighbourhood, and a lone white speck is never the median, so it vanishes.

# a clean scene, then salt & pepper noise added
scene = cv2.imread("scene.png")            # (built in the lab below)
noisy = scene.copy()
# ... sprinkle 2000 white + 2000 black pixels ...

median = cv2.medianBlur(noisy, 5)          # ksize must be ODD
gauss  = cv2.GaussianBlur(noisy, (5, 5), 0)  # (kernel W,H) odd; sigma 0 = auto

# mean abs difference from the clean original (lower = better denoise):
def mad(a, b): return float(np.abs(a.astype(int) - b.astype(int)).mean())
print("noisy :", round(mad(noisy,  scene), 2))    # 4.17
print("median:", round(mad(median, scene), 2))    # 0.04   <- median crushes s&p noise
print("gauss :", round(mad(gauss,  scene), 2))    # 4.62   <- barely helps here

The numbers are decisive: median blur drops the error from 4.17 to 0.04 on salt-and-pepper noise, while Gaussian barely moves it. Matching the filter to the noise is the whole skill.

Two knobs control every blur. Kernel size must be odd (3, 5, 7, …) so it has a well-defined centre pixel; a bigger kernel averages more neighbours and smooths harder, at the cost of detail and speed. For GaussianBlur, a sigma of 0 tells OpenCV to derive the spread from the kernel size. And when you need to remove noise but keep edges crisp — smoothing a portrait’s skin without blurring the eyes — reach for cv2.bilateralFilter, which weights neighbours by both distance and colour similarity, so it refuses to average across a strong edge. It is slower than a Gaussian but the only one-shot filter that denoises without softening boundaries.

Filter Function Kernel Best for
Gaussian blur GaussianBlur(img, (k,k), sigma) Bell-shaped weights General noise; pre-edge smoothing
Median blur medianBlur(img, k) Median of neighbourhood Salt-and-pepper noise
Box/mean blur blur(img, (k,k)) Uniform average Fast, crude smoothing
Bilateral bilateralFilter(img, d, sc, ss) Edge-aware Denoise while keeping edges
Custom filter2D(img, -1, kernel) Any weights you supply Sharpen, emboss, feature detect

A kernel is just a weight matrix (this is a conv layer)

cv2.filter2D lets you supply any kernel, which makes the CNN connection concrete. A 3×3 box of 1/9 averages (blur); a centre-heavy kernel sharpens; a left-minus-right kernel detects vertical edges. Prove to yourself that convolution is exactly a sliding weighted sum:

box     = np.ones((3, 3), np.float32) / 9.0            # average of 9 -> blur
sharpen = np.array([[0,-1,0],[-1,5,-1],[0,-1,0]], np.float32)   # centre-surround
sobel_x = np.array([[-1,0,1],[-2,0,2],[-1,0,1]], np.float32)    # vertical-edge detector

blurred = cv2.filter2D(gray, -1, box)
edges_x = cv2.filter2D(gray, -1, sobel_x)

# the box filter at a pixel == the plain mean of its 3x3 neighbourhood:
y, x = 100, 90
print("manual 3x3 mean:", round(float(gray[y-1:y+2, x-1:x+2].mean()), 1))  # 135.0
print("filter2D value :", int(blurred[y, x]))                              # 135

The manual mean and filter2D agree exactly (135), because that is all convolution is: a weighted neighbourhood sum, one per pixel. A CNN stacks thousands of these and learns the weights — sobel_x is a hand-designed edge detector; a trained network derives similar filters on its own in its first layer.

Edge detection with Canny

Edges — sharp intensity changes — are where objects meet background, and Canny is the classic detector. It takes two thresholds and uses hysteresis: pixels with gradient above the high threshold are definite edges; pixels above the low threshold are edges only if they connect to a definite one. This two-level scheme traces continuous contours while rejecting isolated noise.

blur = cv2.GaussianBlur(gray, (7, 7), 0)   # ALWAYS blur first
edges = cv2.Canny(blur, 50, 150)           # (low, high) thresholds
print(edges.shape, np.unique(edges), int((edges>0).sum()))  # (120,240) [0 255] ...

Getting the two thresholds wrong is the usual Canny failure, in both directions. Too high and real edges vanish; too low (especially without blurring first) and noise floods the output with spurious edges:

# too high -> misses edges;  too low on a noisy image -> noise storm
print(int((cv2.Canny(blur, 240, 250) > 0).sum()))   # 496    (too high: barely any edges)
noisy_gray = np.clip(gray + rng.normal(0, 25, gray.shape), 0, 255).astype(np.uint8)
print(int((cv2.Canny(noisy_gray, 10, 30) > 0).sum()))  # 44194  (too low: noise flood)

Canny’s own advice is a high:low ratio of 2:1 or 3:1, and to always blur first. Start with something like Canny(blur, 50, 150) and adjust.

Canny knob Effect Symptom if wrong
Low threshold Weak-edge floor (hysteresis) Too low → noisy edges everywhere
High threshold Strong-edge seed Too high → broken/missing edges
Blur first Removes noise before gradients Skip it → snowstorm of false edges
apertureSize Sobel kernel size (3/5/7) Larger = thicker gradient response

Morphology and contours

After thresholding you have a binary mask, but it is rarely clean: a few stray white specks, a few black pinholes inside solid shapes. Morphology fixes this with two primitives operating on white regions — erode (shrink white) and dilate (grow white) — and their two useful combinations. Opening (erode then dilate) removes small white specks without shrinking big shapes; closing (dilate then erode) fills small black holes without growing shapes.

kernel = np.ones((5, 5), np.uint8)          # the structuring element
eroded  = cv2.erode(binary, kernel, iterations=1)
dilated = cv2.dilate(binary, kernel, iterations=1)
opened  = cv2.morphologyEx(binary, cv2.MORPH_OPEN,  kernel)   # de-speckle
closed  = cv2.morphologyEx(binary, cv2.MORPH_CLOSE, kernel)   # fill holes

print("white px  binary:", int((binary>0).sum()))    # 34324
print("white px  erode :", int((eroded>0).sum()))     # 31268  (shrank)
print("white px  dilate:", int((dilated>0).sum()))    # 37508  (grew)

The pixel counts confirm the intuition: erode drops white pixels (34324 → 31268), dilate adds them (→ 37508). Opening’s power shows on a speckled mask — sprinkle 194 tiny white blobs and opening deletes essentially all of them while leaving the real shapes:

# speckled mask has 194 connected components; opening removes the specks
n_before, _ = cv2.connectedComponents(speckled)
n_after,  _ = cv2.connectedComponents(cv2.morphologyEx(speckled, cv2.MORPH_OPEN, kernel))
print(n_before - 1, "->", n_after - 1)   # 194 -> 4   (specks gone, 4 shapes remain)
Morphology op morphologyEx flag Effect Use for
Erode cv2.erode Shrinks white regions Separate touching blobs, thin
Dilate cv2.dilate Grows white regions Connect gaps, thicken
Opening MORPH_OPEN (erode→dilate) Removes small white specks De-noise a mask
Closing MORPH_CLOSE (dilate→erode) Fills small black holes Solidify shapes
Gradient MORPH_GRADIENT Dilate − erode Outline / edge of blobs
Top hat MORPH_TOPHAT Image − opening Bright detail on dark bg

Contours: find, measure, and box the shapes

A contour is the boundary curve of a connected white region. cv2.findContours walks your cleaned mask and returns a list of these curves; each one is an object you can measure. RETR_EXTERNAL keeps only outermost contours (ignore holes); CHAIN_APPROX_SIMPLE compresses straight runs to their endpoints to save memory.

contours, hierarchy = cv2.findContours(closed, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
print(len(contours), "shapes found")        # 4 shapes found

for c in sorted(contours, key=cv2.contourArea, reverse=True):
    area = cv2.contourArea(c)               # pixels enclosed
    x, y, w, h = cv2.boundingRect(c)        # upright box (x, y, width, height)
    M = cv2.moments(c)                      # spatial moments
    cx, cy = int(M["m10"]/M["m00"]), int(M["m01"]/M["m00"])   # centroid
    print(f"area={int(area):6d}  bbox=({x},{y},{w},{h})  centroid=({cx},{cy})")
# area= 11998  bbox=(40,40,101,121)   centroid=(90,100)
# area=  9452  bbox=(245,35,111,111)  centroid=(300,90)
# area=  7342  bbox=(251,201,109,69)  centroid=(305,235)
# area=  4868  bbox=(71,201,79,79)    centroid=(110,240)

The centroid comes from image moments: m00 is the area, and (m10/m00, m01/m00) is the centre of mass. That is genuinely all “object detection” is at this classical level — threshold, clean, contour, measure. A modern neural detector replaces these four steps with one learned model, but the outputs (a box and a centre per object) are identical, which is why understanding this makes the ML version legible.

Contour tool Call Returns
Find contours findContours(mask, mode, method) (contours, hierarchy)
Retrieval mode RETR_EXTERNAL / RETR_TREE / RETR_LIST Outer only / nested / flat
Area cv2.contourArea(c) Enclosed pixel count
Perimeter cv2.arcLength(c, closed=True) Boundary length
Upright box cv2.boundingRect(c) (x, y, w, h)
Rotated box cv2.minAreaRect(c) ((cx,cy),(w,h),angle)
Centroid cv2.moments(c)m10/m00, m01/m00 Centre of mass
Convex hull cv2.convexHull(c) Smallest enclosing polygon
Draw cv2.drawContours(img, contours, -1, colour, t) In place

The two flags to findContours decide which contours come back and how densely each is stored. Pick the wrong retrieval mode and holes inside a shape return as separate objects (or vanish); pick CHAIN_APPROX_NONE and you store every boundary pixel instead of just the corners.

Flag Category Meaning
RETR_EXTERNAL Retrieval Outermost contours only (ignore holes)
RETR_LIST Retrieval All contours, flat (no hierarchy)
RETR_CCOMP Retrieval Two levels: outer boundaries + holes
RETR_TREE Retrieval Full nested parent/child hierarchy
CHAIN_APPROX_NONE Approximation Every boundary point (memory-heavy)
CHAIN_APPROX_SIMPLE Approximation Compress straight runs to endpoints

Drawing: annotate what you found

To visualise detections you draw straight onto the array. All the drawing functions modify the image in place, take BGR colours, and use (x, y) (column, row) coordinates — the opposite order to array indexing, another easy slip.

annot = scene.copy()
for c in contours:
    x, y, w, h = cv2.boundingRect(c)
    cv2.rectangle(annot, (x, y), (x+w, y+h), (0, 255, 255), 2)   # yellow box, thickness 2
    cv2.circle(annot, (x+w//2, y+h//2), 4, (255, 255, 255), -1)  # -1 = filled
    cv2.putText(annot, "shape", (x, y-6), cv2.FONT_HERSHEY_SIMPLEX,
                0.5, (255, 255, 255), 1, cv2.LINE_AA)
cv2.imwrite("annotated.png", annot)        # save, don't show

Histograms and contrast

A histogram counts how many pixels sit at each intensity — the shape of an image’s tonal distribution. A low-contrast image has all its counts bunched in a narrow band; histogram equalization stretches that band across the full 0–255 range, boosting contrast automatically.

lowc = (gray * 0.4 + 100).clip(0, 255).astype(np.uint8)   # squash into 112-163
eq   = cv2.equalizeHist(lowc)                              # stretch to full range
print("low-contrast range:", lowc.min(), "-", lowc.max(), " std", round(float(lowc.std()),1))
print("equalized    range:", eq.min(),   "-", eq.max(),   " std", round(float(eq.std()),1))
# low-contrast range: 112 - 163  std 18.7
# equalized    range: 0 - 255    std 83.9
hist = cv2.calcHist([lowc], [0], None, [256], [0, 256])   # 256-bin histogram

The standard deviation jumps from 18.7 to 83.9 — that is the contrast you gained, quantified. (For colour images, equalize the V channel in HSV, or use cv2.createCLAHE() for a local, less harsh version.)

From hand-built pipelines to learned ones

Step back and notice what you have been doing: at every stage you chose the operation — this blur, that Otsu threshold, these 5×5 morphology kernels. That is classical computer vision, and it is genuinely powerful: fast, interpretable, needs no training data, and runs on a Raspberry Pi. Its ceiling is that a human has to design each feature. The moment “find the defective solder joint” depends on a texture too subtle to write a kernel for, hand-tuning collapses.

Deep learning flips the authorship. A convolutional neural network is a stack of the very same filter2D convolutions you used above — but the kernel weights are learned from labelled examples instead of hand-set, so the network discovers the right filters (crude edge detectors in its first layer, object parts deeper in) rather than you writing them. You already built a Sobel edge detector by hand; a CNN derives similar filters on its own, then thousands more you would never think of. The trade is real: it needs data, it needs a GPU to train, and it gives up interpretability. Neither approach wins outright — production systems routinely use classical CV for the plumbing (load, denoise, crop, colour-normalise) and hand the hard perception in the middle to a network. Everything in this lesson is that plumbing; the next step into machine learning is the network that sits in the gap.

Aspect Classical CV (this lesson) Deep learning (next lessons)
Who designs the features You — kernels, thresholds, morphology The network learns them from data
Training data None needed Many labelled examples
Compute Runs on a CPU / edge device A GPU to train, ideally
Interpretability High — every step is explicit Low — millions of opaque weights
Best at Well-defined, geometric tasks Messy, subtle, high-variety perception
Core operation cv2.filter2D with fixed kernels Conv layers with learned kernels

Hands-on lab

You will build the entire pipeline end to end on a synthesized scene, saving every stage to disk so you can open the PNGs and see what each step did. No dataset needed — the scene is generated in code, so this runs identically on your laptop or a bare server.

⚠️ Use a Python 3.12+ interpreter (macOS system Python is 3.9). Work in a virtual environment and install the headless OpenCV build:

python3.12 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install opencv-python-headless pillow numpy
python -c "import cv2, numpy; print('cv2', cv2.__version__, '| numpy', numpy.__version__)"
# cv2 5.0.0 | numpy 2.5.1   (versions may differ)

Create cv_lab.py and add each step, running python cv_lab.py as you go. Every non-timing output below is exact.

Step 1 — Synthesize a scene and read its header.

import cv2, numpy as np, time
rng = np.random.default_rng(0)

scene = np.full((300, 400, 3), 30, dtype=np.uint8)              # dark gray background
cv2.rectangle(scene, (40, 40), (140, 160), (60, 180, 75), -1)   # green rectangle (BGR)
cv2.circle(scene, (300, 90), 55, (0, 140, 255), -1)            # orange circle
cv2.circle(scene, (110, 240), 40, (200, 60, 200), -1)          # purple circle
cv2.rectangle(scene, (250, 200), (360, 270), (40, 40, 230), -1) # red rectangle
cv2.imwrite("stage0_scene.png", scene)
print("scene:", scene.shape, scene.dtype)   # scene: (300, 400, 3) uint8

What just happened: you built a (300, 400, 3) uint8 array with four coloured shapes on a dark background — a controlled test image. The -1 thickness fills each shape. This is your ground truth: four objects to find.

Step 2 — Prove the BGR→RGB swap.

print("green pixel BGR (cv2):", scene[100, 90])                      # [ 60 180  75]
rgb = cv2.cvtColor(scene, cv2.COLOR_BGR2RGB)
print("green pixel RGB       :", rgb[100, 90])                       # [ 75 180  60]
print("just a channel flip? :", np.array_equal(scene[..., ::-1], rgb))  # True
cv2.imwrite("stage2_rgb_wrong.png", rgb)   # saving RGB as if BGR -> looks swapped

What just happened: the same green pixel is [60,180,75] in cv2’s BGR and [75,180,60] in RGB — red and blue traded places. stage2_rgb_wrong.png deliberately saves the RGB array back through cv2 (which assumes BGR), so it looks colour-swapped: proof of the bug, on disk.

Step 3 — Grayscale, then Gaussian-blur.

gray = cv2.cvtColor(scene, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (7, 7), 0)
print("gray shape:", gray.shape, "(2-D, no channel axis)")   # (300, 400)
cv2.imwrite("stage3_gray.png", gray)
cv2.imwrite("stage3_blur.png", blur)

What just happened: grayscale collapsed the 3 channels into one (300, 400) plane; the 7×7 Gaussian smoothed it so the next steps won’t fire on stray pixels. Blurring before thresholding/edges is the habit that prevents most noise bugs.

Step 4 — Otsu threshold to a binary mask.

t, binary = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
print("Otsu chose threshold:", t)                 # Otsu chose threshold: 79.0
print("mask values:", np.unique(binary), "white px:", int((binary>0).sum()))
cv2.imwrite("stage4_binary.png", binary)          # mask values: [0 255] white px: 34324

What just happened: Otsu inspected the histogram and picked 79 on its own, producing a black-and-white mask where the four shapes are white. No hand-tuned magic number — the data chose the split.

Step 5 — Morphology to clean the mask.

kernel = np.ones((5, 5), np.uint8)
clean = cv2.morphologyEx(binary, cv2.MORPH_OPEN,  kernel)   # remove specks
clean = cv2.morphologyEx(clean,  cv2.MORPH_CLOSE, kernel)   # fill holes
cv2.imwrite("stage5_clean.png", clean)
print("white px after morphology:", int((clean>0).sum()))   # 34324

What just happened: opening then closing scrubbed the mask — any speckle gone, any pinhole filled — so the contour step traces four solid blobs instead of a ragged edge.

Step 6 — Find contours, measure and box each shape.

contours, _ = cv2.findContours(clean, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
print("shapes detected:", len(contours))          # shapes detected: 4
annot = scene.copy()
for i, c in enumerate(sorted(contours, key=cv2.contourArea, reverse=True)):
    area = cv2.contourArea(c)
    x, y, w, h = cv2.boundingRect(c)
    M = cv2.moments(c); cx, cy = int(M["m10"]/M["m00"]), int(M["m01"]/M["m00"])
    cv2.rectangle(annot, (x, y), (x+w, y+h), (0, 255, 255), 2)
    cv2.circle(annot, (cx, cy), 4, (255, 255, 255), -1)
    cv2.putText(annot, f"#{i} A={int(area)}", (x, y-6),
                cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255,255,255), 1, cv2.LINE_AA)
    print(f"  #{i}: area={int(area):6d}  bbox=({x},{y},{w},{h})  centroid=({cx},{cy})")
cv2.imwrite("stage6_annotated.png", annot)
#   #0: area= 11998  bbox=(40,40,101,121)   centroid=(90,100)
#   #1: area=  9452  bbox=(245,35,111,111)  centroid=(300,90)
#   #2: area=  7342  bbox=(251,201,109,69)  centroid=(305,235)
#   #3: area=  4868  bbox=(71,201,79,79)    centroid=(110,240)

What just happened: all four shapes detected, each with a real area, bounding box, and centroid drawn onto stage6_annotated.png. The centroids (90,100), (300,90), (305,235), (110,240) match exactly where you drew the shapes. That is a complete classical object-detection pipeline.

Step 7 — HSV-mask a single colour.

hsv = cv2.cvtColor(scene, cv2.COLOR_BGR2HSV)
green_mask = cv2.inRange(hsv, np.array([40,50,50]), np.array([80,255,255]))
green_only = cv2.bitwise_and(scene, scene, mask=green_mask)
print("green pixels:", int((green_mask>0).sum()))   # green pixels: 12221
cv2.imwrite("stage7_green.png", green_only)

What just happened: HSV isolated just the green rectangle by hue range, ignoring the other three shapes — the colour-selection tool you reach for when grayscale thresholding can’t tell objects apart (a dark-red shape on a dark background, say, is invisible to Otsu but obvious in HSV).

Step 8 — MEASURE vectorized vs a per-pixel Python loop.

small = cv2.resize(scene, (200, 150))       # 30,000 pixels
H, W = small.shape[:2]

t0 = time.perf_counter()                    # hand-rolled per-pixel grayscale (SLOW)
out_loop = np.empty((H, W), dtype=np.uint8)
for yy in range(H):
    for xx in range(W):
        b, g, r = small[yy, xx]
        out_loop[yy, xx] = int(0.114*b + 0.587*g + 0.299*r)
t_loop = time.perf_counter() - t0

t0 = time.perf_counter()                    # vectorized: whole-array math (FAST)
bb, gg, rr = small[...,0], small[...,1], small[...,2]
out_vec = (0.114*bb + 0.587*gg + 0.299*rr).astype(np.uint8)
t_vec = time.perf_counter() - t0

t0 = time.perf_counter(); out_cv = cv2.cvtColor(small, cv2.COLOR_BGR2GRAY)
t_cv = time.perf_counter() - t0

print(f"python loop : {t_loop*1000:8.2f} ms")           # python loop :    83.85 ms
print(f"numpy vec   : {t_vec*1000:8.3f} ms -> {t_loop/t_vec:5.0f}x")  # ->   646x
print(f"cv2.cvtColor: {t_cv*1000:8.3f} ms -> {t_loop/t_cv:5.0f}x")    # ->  4264x
print("loop vs vec identical:", int(np.abs(out_loop.astype(int)-out_vec.astype(int)).max()) == 0)

What just happened: the exact same grayscale conversion ran ~650× faster vectorized and thousands of times faster in OpenCV’s C code — on just 30k pixels; a full-HD frame has 60× more. The per-pixel Python loop pays the interpreter tax two million times; NumPy and OpenCV pay it never. This is the vectorization lesson made visible on an image: never loop over pixels in Python — there is an array operation or an OpenCV function that does it in C. (Timings vary with your machine; the order of magnitude does not.)

You have now loaded, colour-converted, blurred, thresholded, cleaned, detected, measured, colour-masked, and benchmarked — the whole pipeline, every stage on disk.


Common mistakes and troubleshooting

Symptom / traceback Cause Fix
Skies orange, faces blue, colours “off” Fed a cv2 BGR array to PIL/matplotlib/ML (they want RGB) cv2.cvtColor(img, cv2.COLOR_BGR2RGB) at the boundary
AttributeError: 'NoneType' object has no attribute 'shape' cv2.imread got a bad path → returned None (no exception) Check if img is None: raise FileNotFoundError(path)
cv2.error: ... The function is not implemented. Rebuild ... with ... support imshow on a headless server (no window system) Save with cv2.imwrite instead; use opencv-python-headless
Brightening img + 50 makes bright areas dark uint8 overflow — 255+50 wraps to 49 cv2.add(img, v) (saturates) or np.clip(img.astype(int)+v,0,255).astype(np.uint8)
error: (-215) ... 'scn' is 1 on cvtColor(gray, COLOR_BGR2GRAY) Converting an image that is already grayscale (1 channel) Check img.ndim; only convert 3-channel images
IndexError/error indexing img[y,x,0] Image is grayscale (H,W)no channel axis Grayscale is 2-D; index img[y,x], or convert to colour first
findContours: error: ... support only CV_8UC1 images Passed a colour (or non-binary) image to findContours Threshold to a single-channel uint8 binary mask first
Resized image is transposed / wrong shape cv2.resize takes (W, H); you passed img.shape[:2] = (H, W) Pass (width, height); or shape[1::-1]
Resized mask has fuzzy 1.5-class edges Used INTER_LINEAR on a label mask Use INTER_NEAREST for masks/labels
Canny returns nothing or a snowstorm Thresholds too high (no edges) / too low + no blur (noise) Blur first; start Canny(blur, 50, 150), high:low ≈ 2–3:1
TypeError: Cannot handle this data type: (1,1,3), <f8 from Image.fromarray Array is float64, not uint8 arr.clip(0,255).astype(np.uint8) before fromarray/imwrite
Saved image is all black / washed out Forgot to scale a float [0,1] image back to uint8 [0,255] (arr*255).astype(np.uint8) before saving
Contours found on the wrong regions Objects are dark on light bg; THRESH_BINARY kept the background Use THRESH_BINARY_INV, or invert the mask

Three of these eat the most hours, so they get extra words.

1. The BGR/RGB swap (blue faces). The tell is that colours look wrong but plausible — an image that is clearly displaying, just with red and blue exchanged. It happens at every crossing between OpenCV and the RGB world: plt.imshow(cv2_img), Image.fromarray(cv2_img), feeding a cv2 frame to a model trained on RGB. There is no error because the array is structurally fine. The discipline that eliminates it: stay BGR inside OpenCV, cvtColor to RGB exactly once, at the moment you hand the array to something else. When you see swapped colours, search your code for the last non-OpenCV consumer and add the conversion just before it.

2. imread returns None, silently. Unlike open(), which raises FileNotFoundError, cv2.imread on a missing or unreadable file returns None and prints only a stderr warning you will miss. Your program marches on and dies several lines later on None.shape or None.astype, with a traceback that fingers the wrong statement. Because image paths are so often wrong (relative-path confusion, a typo, a missing file in the container), guard every imread: img = cv2.imread(p); assert img is not None, f"could not read {p}". This single habit turns a baffling NoneType error into a clear “file not found.”

3. uint8 overflow darkens instead of brightens. Add 100 to a pixel already at 200 and uint8 wraps 300 to 44 — the bright spot goes dark, the opposite of what you intended, and no warning fires. It is the integer-overflow pitfall from NumPy wearing an image costume. Two fixes: use OpenCV’s saturating arithmetic (cv2.add, cv2.subtract, cv2.addWeighted), which clamp at 0 and 255; or do the math in a wider dtype and clip — np.clip(img.astype(np.int16) + 100, 0, 255).astype(np.uint8). The general rule mirrors NumPy’s: compute in int16/float, clip, and narrow back to uint8 only for saving.


Cheat-sheet

Task Code
Read image (BGR) img = cv2.imread("x.png") — check is None!
Read grayscale cv2.imread("x.png", cv2.IMREAD_GRAYSCALE)
Write image cv2.imwrite("out.png", img)
Open/save with PIL Image.open(p) / im.save(p, quality=85)
Shape / dtype img.shape (H,W,C); img.dtype uint8
BGR → RGB cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
BGR → grayscale cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)(H,W)
BGR → HSV cv2.cvtColor(img, cv2.COLOR_BGR2HSV) (H 0-179)
PIL ↔ NumPy np.asarray(im) / Image.fromarray(arr) (RGB)
Resize cv2.resize(img, (W,H), interpolation=cv2.INTER_AREA)
Crop img[y0:y1, x0:x1] (a view — .copy() to detach)
Flip cv2.flip(img, 1) (horiz) 0 (vert)
Rotate warpAffine(img, getRotationMatrix2D(c, deg, 1), (w,h))
Brighten (safe) cv2.add(img, np.full_like(img, 30))
Gaussian blur cv2.GaussianBlur(img, (5,5), 0) (odd kernel)
Median blur (s&p) cv2.medianBlur(img, 5) (odd ksize)
Custom kernel cv2.filter2D(img, -1, kernel)
Fixed threshold cv2.threshold(g, 127, 255, cv2.THRESH_BINARY)
Otsu threshold cv2.threshold(g, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)
Adaptive threshold cv2.adaptiveThreshold(g, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 11, 2)
Colour mask (HSV) cv2.inRange(hsv, lo, hi)
Canny edges cv2.Canny(blur, 50, 150) (low, high)
Erode / dilate cv2.erode(m, k) / cv2.dilate(m, k)
Open / close cv2.morphologyEx(m, cv2.MORPH_OPEN/CLOSE, k)
Find contours cv2.findContours(m, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
Area / box / centroid contourArea(c) / boundingRect(c) / moments(c)
Draw box / text cv2.rectangle(...) / cv2.putText(...) (BGR, in place)
Equalize contrast cv2.equalizeHist(gray)
Histogram cv2.calcHist([g], [0], None, [256], [0,256])

Interview and exam questions

Q: In one sentence, what is an image to a Python program? A: A NumPy ndarray — for colour, a (height, width, 3) array of uint8 values 0255; for grayscale, a 2-D (height, width) array. Every CV operation is arithmetic on that array, which is why NumPy skills transfer directly.

Q: Why do colours look wrong when you move an image from OpenCV to matplotlib, and how do you fix it? A: OpenCV stores channels as BGR while matplotlib (and PIL, and most of the world) expects RGB, so red and blue are swapped — no error, just wrong colours. Convert at the boundary with cv2.cvtColor(img, cv2.COLOR_BGR2RGB) before handing the array off. The conversion is just a channel reversal, equivalent to img[..., ::-1].

Q: cv2.imread("photo.png") and the next line crashes with NoneType has no attribute 'shape'. What happened? A: imread couldn’t read the file (wrong path, missing, unreadable) and returned None instead of raising. The crash is on the next use of that None. Always guard: img = cv2.imread(p); assert img is not None, p. This is why the traceback appears to point at the wrong line.

Q: Your brightness filter img + 50 makes bright regions darker. Why, and what’s the fix? A: Pixels are uint8, so 220 + 50 = 270 overflows and wraps modulo 256 to 14 — bright becomes dark, silently. Use saturating arithmetic cv2.add(img, 50) (clamps at 255), or widen and clip: np.clip(img.astype(np.int16)+50, 0, 255).astype(np.uint8).

Q: When would you use HSV instead of BGR/RGB? A: For colour-based selection. In HSV the colour lives on one axis (hue) separate from brightness (value), so “select all the green, whatever the lighting” is a simple hue range with cv2.inRange — robust in a way raw BGR isn’t, because in BGR a lighting change moves all three channels at once. Note OpenCV’s hue is 0179, not 0359.

Q: Explain Otsu thresholding and when it fails. A: Otsu automatically picks the threshold that best separates a bimodal histogram into two classes (foreground/background) by minimising within-class variance — you pass 0 as the threshold with the THRESH_OTSU flag and it returns the value it chose. It fails when the histogram isn’t bimodal (smooth gradients) or lighting is uneven; there you use adaptive thresholding, which computes a local threshold per region.

Q: What is a convolution kernel, and how does it connect to CNNs? A: A kernel is a small matrix of weights slid over the image; at each position the output pixel is the weighted sum of the neighbourhood (cv2.filter2D). A box kernel blurs, a centre-surround kernel sharpens, a Sobel kernel detects edges. A CNN does the identical operation, except it learns the kernel weights from data instead of you hand-designing them — hand-built filters here are literally single, fixed conv layers.

Q: Why does Canny take two thresholds? A: Hysteresis. Pixels with gradient above the high threshold are seeded as definite edges; pixels above the low threshold are kept only if they connect to a definite edge. This traces continuous contours while rejecting isolated noise. Rule of thumb: blur first, high:low ratio around 2:1–3:1. Too high → missing edges; too low → noise flood.

Q: What do opening and closing do, and in what order do their primitives run? A: Opening = erode then dilate; it removes small white specks without shrinking real shapes. Closing = dilate then erode; it fills small black holes without growing shapes. Erode shrinks white regions, dilate grows them; the two-step combos undo the size change while keeping the cleanup.

Q: You have a binary mask of some blobs. How do you get a bounding box and centre for each? A: contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE); then per contour cv2.boundingRect(c) gives (x,y,w,h), cv2.contourArea(c) the size, and cv2.moments(c) the centroid as (m10/m00, m01/m00). That four-step threshold→clean→contour→measure sequence is classical object detection.

Q (practical): Why is opencv-python-headless the right install on a server, and how do you inspect images without a display? A: The headless build has no GUI dependencies, so it installs cleanly in Docker/CI and never tries to open a window; imshow would raise “function not implemented” there anyway. You inspect by saving: cv2.imwrite("stage_N.png", stage) at each step and open the files. Never call imshow/waitKey in server code.

Q (coding): Load img.png, count how many red objects it contains, and print each one’s area. A: Convert to HSV, inRange the red hue band (red wraps around 0/179, so often two ranges OR’d together), morphologically open the mask to de-speckle, findContours, filter tiny contours by contourArea, and print the areas: hsv=cv2.cvtColor(img,cv2.COLOR_BGR2HSV); mask=cv2.inRange(hsv,(0,120,70),(10,255,255)); mask=cv2.morphologyEx(mask,cv2.MORPH_OPEN,np.ones((5,5),np.uint8)); cnts,_=cv2.findContours(mask,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE); [print(cv2.contourArea(c)) for c in cnts if cv2.contourArea(c)>50].


Key takeaways

pythonopencvcv2pillowpilcomputer-visionimage-processingbgr-rgbthresholdingotsucannycontoursmorphologyhsvnumpy
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments