← Notes

Notes

Interactive

Image Processing Laboratory

14 min read computer-visionsignal-processinginteractive

Classical image processing is a small vocabulary of operations that compose. The hard part was never any single filter — it is knowing what each one does to the thing the next one receives. A median filter before an edge detector is a denoiser; after it, it is a line thinner. Same operation, different job, and nothing in the code tells you which.

So this bench shows the intermediates. The first bay lets you assemble a pipeline and see the image after every stage; the rest take one family of operations each and open it up. Everything runs in this tab, on the built-in targets or on any image you drop in.

Processing bench

Every stage, and what it hands to the next one

09 bays
live · client-side
f the input — and outside its bounds, whatever the padding rule invents
k the kernel. Its sum decides whether brightness survives; its symmetry decides everything else
g the output, which becomes the next stage's f
x − i the minus sign: convolution flips the kernel. Correlation does not, and for Sobel that is a sign flip
[ 01 ]

Build a pipeline, see every stage

live

Add operations, reorder them, tune their parameters. Each card shows the image as it leaves that stage, its cost in milliseconds, and one sentence on what it just did. Order matters enormously and is the easiest thing to get wrong — try moving the median filter after the edge detector and watch what it becomes.

Add

Appends an operation to the end of the chain. Reorder or remove it from its own card once it is there.

Presets

Ready-made chains worth stepping through — each is a shape that shows up in real pipelines.

Boundaryoutside the image

Every filter needs samples that do not exist. Set this to Zero and watch a dark rim appear — the filter is averaging in blackness that was never in the picture.

stages—
total time—

Imageor drop your own

  • Load the default Denoise → edges chain. Now delete the median filter: the salt-and-pepper noise survives into the Sobel stage and the edge map fills with dots. That one card is the entire argument for denoising first.
  • Move Closing above Threshold. It stops doing anything useful — morphology on a greyscale image is a min/max filter, not a shape operation.
  • Set the boundary to Circular on the Shapes target. Detail from the opposite edge bleeds in, because circular padding asserts the image tiles the plane. The FFT in bay 05 assumes exactly this.
Takeaway A pipeline is not a list of operations, it is a sequence of representations. After a threshold you no longer have an image, you have a set; after an edge detector you have a measurement, not a picture. Most pipeline bugs are one stage receiving a kind of thing the next one was not written for.
[ 02 ]

Four ways to blur, and only two are linear

live

Box and Gaussian are convolutions: they have a kernel, and that kernel is shown on the right. Median and bilateral are not — no fixed kernel exists for them, which is precisely why they can preserve an edge that no linear filter can.

filtered
the kernel

Filter

Box and Gaussian are convolutions and have a kernel you can look at. Median and bilateral are not, and do not.

Parameters

σ
size
radius
σ space
σ range

Only the parameters belonging to the selected filter are shown.

Boundary

What the filter reads beyond the edge of the image. Every mode is identical inside it and they differ only in the border.

kernel—
Σ weights—
time—
PSNR vs input—

Imageor drop your own

  • Compare Box and Gaussian at similar strength on the Chirp target. The box filter's response crosses zero and comes back — some frequencies survive that shouldn't, and the fine end of the chirp shows ghost bands the Gaussian never produces.
  • Switch to Bilateral and raise σ range past about 60. It degenerates into an ordinary Gaussian, because at that point no intensity difference is large enough to matter.
  • Note the Σ weights readout: 1.0000 for both linear blurs. A kernel summing to anything else changes overall brightness — which is a bug unless you meant it.
Takeaway Linear filters are completely described by their kernel, which means they are also completely limited by it: the same weights are applied at an edge and in a flat region, because the filter cannot tell the difference. Every edge-preserving method is non-linear, and pays for it in speed.
[ 03 ]

Nine numbers

live

Type any nine numbers and see what they do. Two properties predict almost everything: the sum decides whether average brightness survives, and the symmetry decides whether you get a blur, a derivative or an embossing. Everything else is detail.

Kernel3 × 3

Type any nine numbers. Convolution flips the kernel before applying it, so an antisymmetric one changes sign.

Load

Standard kernels to start from — edit the cells afterwards and watch what breaks.

Σ weights—
behaviour—

Imageor drop your own

  • Load Sharpen — it sums to 1, so brightness is preserved. Now change the centre from 5 to 6 and the whole image lifts: the sum is 2, and every pixel is being doubled.
  • Load Sobel X. It sums to zero, so flat regions map to zero — mid grey once shifted for display. A zero-sum kernel measures change and is blind to level.
  • Make a kernel that is antisymmetric about the diagonal (like Emboss). You get a lighting effect, because it responds to slope in one direction only.
Takeaway Every linear shift-invariant operation on an image — every one — is a convolution with some kernel. Blur, sharpen, derivative, embossing, and the first layer of a convolutional network are the same operation with different numbers in the box.
[ 04 ]

The same image, written in waves

live

Every image is a sum of 2D sinusoids, and the spectrum says how much of each. Low frequencies sit at the centre, high ones at the edges. Some defects that are hopeless to describe spatially — a repeating interference pattern, say — are two isolated dots here, and deleting two dots is easy.

spectrum
reconstruction

Filter

cutoff
outer band only
edge softness

Box and Gaussian are convolutions and have a kernel you can look at. Median and bilateral are not, and do not.

Show

Magnitude is what most people mean by \u201cthe spectrum\u201d. Phase looks like noise and is where nearly all the structure lives.

Rebuild from

Discard the phase and the image becomes texture with no structure. Discard the magnitude instead — keep only the phase — and the picture is still recognisable. Almost everything you would call "the content" lives in the phase.

energy kept—
PSNR vs original—

Imageor drop your own

the spectrum is log-scaled and centred; dimmed areas are what the mask rejects256 × 256, radix-2 FFT

  • The default Moiré target carries a sinusoidal interference pattern. Find its two bright dots off-centre in the magnitude, then use Band to cut just that ring — the pattern goes and the picture survives.
  • Set edge softness to its minimum with a low-pass. Ringing appears around every edge: a hard-edged mask in frequency is a sinc in space, and a sinc has ripples that go on forever.
  • Switch Rebuild from to Phase. The image is still legible with every amplitude thrown away. Then try Magnitude — unrecognisable.
Takeaway The Fourier transform loses nothing — it is a change of basis, and it inverts exactly. What it buys is a domain where "smooth", "noisy" and "periodic" become positions rather than adjectives. What it costs is locality: every coefficient describes the entire image at once, which is why a single bright dot in the spectrum is a pattern spanning every pixel.
[ 05 ]

Two routes to the same answer

live

Convolution in space is multiplication in frequency. Not approximately — exactly, provided the spatial version wraps around, because that is the boundary condition the DFT assumes whether you want it or not. The third panel is the difference between the two routes.

Kernel

σ Gaussian only

Type any nine numbers. Convolution flips the kernel before applying it, so an antisymmetric one changes sign.

Spatial boundary

Only Circular matches what the FFT does. Any other choice and the two disagree — at the border only, which is exactly where people forget to look.

spatial—
via FFT—
largest difference—
verdict—

Imageor drop your own

  • With Circular padding the difference is at the level of floating-point noise — around 10⁻⁵ — for every kernel. That is the convolution theorem, verified on your own screen.
  • Switch to Reflect. The interior stays identical and a bright frame appears in the difference panel. Nothing is wrong; the two methods simply answer different questions about the border.
  • Compare the timings. At σ = 0.5 the direct route wins easily; push σ up and the kernel grows as σ², while the FFT does not care at all.
Takeaway Direct convolution costs O(k²) per pixel; the FFT route costs O(log n) per pixel regardless of kernel size. The crossover is usually somewhere around a 9×9 to 15×15 kernel — which is why small blurs are done directly, large ones are done in frequency, and separable kernels like the Gaussian cheat their way out of the choice entirely.
[ 06 ]

An edge detector has parts

live

"Sobel" is usually shown as one grey edge map, which hides that it is two oriented filters and a conversion to polar form. Here all four are visible: the horizontal derivative, the vertical one, their magnitude, and their orientation as hue. Switch to Canny and the four panels become its four stages instead.

Operator

Sobel, Scharr and Prewitt differ only in their weights. Canny is not an operator at all — it is a four-stage chain.

Canny

blur σ
low
high

The two thresholds must differ for hysteresis to have anything to do. Set them equal and it degenerates to a single threshold.

edge pixels—
note—

Imageor drop your own

orientation panel: hue = gradient direction, saturation = magnitude gx and gy are signed — mid grey is zero
  • On Shapes, look at gx and gy separately. The square's vertical sides appear only in gx and its horizontal sides only in gy. Each is blind to edges parallel to its own direction.
  • Look at the orientation panel around the disc. Hue sweeps smoothly through the whole colour wheel, because the gradient direction rotates continuously around a circular boundary.
  • Switch to Canny and set low and high equal. Hysteresis stops helping and the edges fragment — that gap between the two thresholds is the entire trick.
Takeaway Gradient magnitude is a lossy summary: it throws away the direction, which is the more useful half. HOG, SIFT and every classical descriptor built on gradients uses the orientation histogram, not the magnitude image — magnitude is what you show a human, orientation is what you give an algorithm.
[ 07 ]

Corners, and the scale you find them at

live

An edge is ambiguous along its own direction — slide a window along it and nothing changes. A corner is not, which is what makes it trackable. Harris measures this directly from the structure tensor: both of its eigenvalues large means the gradient turns.

Harris corners
difference of Gaussians, by scale

Harris

pre-smooth σ
window σ
k
threshold

Pre-smoothing suppresses single-pixel noise before differentiating. The window σ sets the scale at which something counts as a corner.

corners found—
peak response—

Imageor drop your own

  • On Shapes, the square's four corners score highest while the disc's smooth boundary scores nothing — a circle has no dominant orientation anywhere along it, so the response never peaks.
  • Drop pre-smooth σ to zero. Detections explode across the speckle: an isolated single pixel is a perfect corner, high response in every direction. This is why every real detector blurs before it differentiates.
  • Raise the window σ. Corners merge and then vanish: the window becomes larger than the feature, and at that scale the corner is no longer a corner.
  • In the scale panel, watch which structures survive at each σ. Fine texture lives only at σ = 0.8; the large shapes are still there at σ = 6.4. Scale is not a nuisance parameter — it is part of what a feature is.
Takeaway There is no such thing as a feature without a scale. This is why SIFT searches scale-space for extrema rather than picking one blur and hoping, and why a detector tuned on 4K images finds nothing on the same scene at 480p.
[ 08 ]

The same picture at every size

live

Blur, halve, repeat. The Gaussian pyramid is the cheapest way to look for something at an unknown size; the Laplacian pyramid stores the difference between neighbouring levels, so each level holds exactly the detail the next-coarser one could not represent — and it reconstructs the original exactly.

Pyramid

levels

Turn the prefilter off on the Chirp target. The fine end does not get smaller, it turns into a completely different, coarser pattern — the alias. Once it is there, no amount of later processing can remove it.

levels—
total storage—

Imageor drop your own

  • The storage readout settles near 133%. A pyramid costs only a third more than the original, because ¼ + ¹⁄₁₆ + … converges to ⅓. That is why it is affordable to keep one around.
  • Switch to Laplacian. Every level is mid grey almost everywhere — the differences are small and sparse, which is precisely why this representation compresses well.
  • Uncheck the prefilter with the Chirp. Aliasing is not blur and it is not noise: it is a wrong answer that looks like a perfectly plausible right one.
Takeaway Downsampling without blurring first is the single most common signal-processing bug in graphics code, and it is invisible until the content happens to contain fine detail. The blur is not there to make things look nice; it is there because sampling below Nyquist folds high frequencies down onto low ones irreversibly.
[ 09 ]

Processing shapes instead of intensities

live

Morphology asks a geometric question at every pixel: does the structuring element fit here? Erosion keeps the pixels where it fits entirely, dilation keeps those where it touches at all, and everything else is built from those two. The Analysis view then measures the shapes that result.

View

Operations shows the six morphological primitives; Analysis measures the shapes they produce.

Structuring element

radius

The shape is not cosmetic. A cross preserves thin horizontal and vertical strokes that a disc of the same radius erases completely.

components—
deepest point—

Imageor drop your own

  • On Shapes, watch opening delete the scattered specks while leaving the disc and square at exactly their original size. Erosion alone would shrink everything; opening restores what survived.
  • Now closing: the deliberate gap in the horizontal bar seals up. Increase the radius until it does, and you have measured the gap.
  • Switch to Analysis on Strokes. The distance transform's bright ridges run down the middle of every stroke — that ridge is the medial axis, and the skeleton is it, made one pixel wide.
Takeaway Opening and closing are idempotent: applying either twice changes nothing the second time. That makes them filters in a strict sense — they select a class of shapes and are stable once applied — which is why morphology is the standard cleanup stage after thresholding, and why applying it repeatedly in a loop is always a misunderstanding.

— end of bench —

What is actually running here

dsp-sci.js holds the algorithms: convolution with four boundary rules, separable Gaussians, median and bilateral filters, Sobel/Scharr/Prewitt gradients, the full Canny chain, a radix-2 FFT with radial masks, Harris, difference of Gaussians, Gaussian and Laplacian pyramids, greyscale morphology, an exact Euclidean distance transform, connected components, and Zhang–Suen thinning. No DOM, no dependencies.

Its test suite runs 112 assertions. The ones worth knowing about are the structural invariants rather than the spot checks: the separable Gaussian must equal the full 2D one; convolution must equal correlation with a flipped kernel and differ in sign for Sobel; Parseval must hold; the Laplacian pyramid must reconstruct the original exactly; opening and closing must be idempotent and must bracket the original; and spatial convolution under circular padding must equal multiplication in the frequency domain — which is bay 05’s whole claim, verified rather than asserted.

Three tests caught mistakes of mine while building this, and all three were in the tests, not the code:

  • I expected a Sobel ramp response of 16 and got 32. The column weights sum to 4 and the operator straddles two pixels, so it is 4 × (f(x+1) − f(x−1)). My arithmetic, not the filter’s.
  • Otsu returned a threshold sitting exactly on the lower mode. With two isolated peaks and a gap between them, every threshold inside the gap scores identically — an optimum plateau, not a point. It now returns the plateau’s midpoint, which is the conventional choice and the useful one.
  • I asserted that skipping the anti-aliasing blur makes a downsampled image noisier. It does not. A pattern at the Nyquist limit aliases to DC, so a black-and-white stripe pattern sampled every other column comes out uniformly black — confidently wrong rather than visibly broken, which is worse. The test now asserts that.

Everything runs on your machine; images you drop in are decoded in this tab and never leave it.

This is the fourth of these benches, after the Camera Geometry Playground, the Colour Science Playground and the Compression Laboratory. All four share the same instrument chrome.