Skip to main content

Optical Flow and Visual Correspondence

Follow image points between frames, distinguish forward flow from backward sampling, and understand brightness constancy, occlusion and the aperture problem.

An image records where things appear at one instant. Two frames add a new question: where did the same visible pattern go? Visual correspondence pairs locations across images. Optical flow describes these pairings between video frames as a displacement field, giving a direction and distance for each selected image location.

A moving object can create flow, but so can a moving camera looking at a stationary scene. Flow is measured in the image plane. It does not directly tell us an object's 3D speed or separate camera motion from object motion. OpenCV's optical-flow tutorial introduces this apparent-motion interpretation. Camera geometry provides optional context for how 3D motion changes image coordinates; the correspondence problem itself can be understood entirely in 2D.

A vector belongs to a source pixel

Let I0I_0 and I1I_1 denote two frames. A source location p=(x,y)p=(x,y) has forward flow F01(p)=(dx,dy)F_{01}(p)=(d_x,d_y) when its corresponding location in frame 1 is:

q=p+F01(p)=(x+dx,  y+dy).q=p+F_{01}(p)=(x+d_x,\;y+d_y).

Here x increases to the right, y increases downward, and both displacement components use image pixels. A sample at (4, 4) with flow (3, −1) reaches (7, 3): three pixels right and one up. This is displacement between these two frames. Dividing by their time interval would give image velocity in pixels per second, still not meters per second.

Compare the object and background samples, then switch to the camera case. With the object selected, set horizontal displacement to 3 and vertical displacement to 0, and choose Occluder to hide its destination. Increase horizontal displacement to 8 to move it outside the image.

Follow the same surface sample
Scene
Source pixel
+3 px
0 px

Frame 0 · source

Frame 1 · target

Visible correspondence. The gather sample matches the source intensity.

The teal object moves; the background stays fixed. A moving object can also cover a background sample. Image x increases rightward and y downward. The frame-1 arrow starts at the old image coordinate and ends at the projected location.

Moving subjectBackgroundSelection / foreground strip

Authored 12 × 8 frames and exact integer correspondences, not estimated or measured flow. Colors identify surfaces; the readout reports their assigned grayscale intensities.

The camera case deliberately gives every surface the same image translation, as a flat equal-depth scene could under a suitable camera translation. Real camera translation generally produces different displacements at different depths; camera rotation and perspective introduce further variation. A uniform shift is a teaching case, not a general camera-motion model.

Missing correspondence is different from no motion

A stationary visible sample has valid flow (0, 0). An occluded sample moves behind another surface: its projected location may be inside frame 1, but the pixel there records the foreground surface. An out-of-frame sample projects beyond the captured image. Neither case means that the original sample stopped moving.

The explorer knows these motions because its scene was authored. An estimator given only the rendered frames would have to infer them, and an occluded sample supplies no direct visible match. Keep a visibility or validity mask alongside flow when reconstructing images or comparing corresponding samples. Conversely, a newly revealed region in frame 1 may have no visible source in frame 0.

Flow direction and sampling direction are separate choices

Forward flow is indexed on frame 0. Backward flow F10F_{10} is indexed on frame 1. For a mutually visible correspondence, their relationship is:

F10(p+F01(p))=−F01(p).F_{10}\bigl(p+F_{01}(p)\bigr)=-F_{01}(p).

The reverse vector belongs at the corresponding location qq, not generally at the original array index pp. With the object moving three pixels right, forward flow at (4, 4) is (3, 0), while backward flow at (7, 4) is (−3, 0). Forward flow at (7, 4) can describe stationary background instead. Negating a forward-flow array at the same indices does not construct the backward field.

To reconstruct frame 0 on its own grid, visit each source position and gather a sample from frame 1:

I^0(p)=I1(p+F01(p)).\widehat I_0(p)=I_1\bigl(p+F_{01}(p)\bigr).

This is a backward, or gather, resampling operation using a forward flow field. Each output position asks where to read its input. Fractional locations require interpolation; the explorer uses integer positions so no interpolation is needed. Occluded and out-of-frame samples are masked.

A forward splat instead visits frame-0 samples and deposits their values at target positions to construct frame 1. Multiple samples may land at one destination, while other destinations receive none. Resolving overlaps and holes is a separate problem from gathering one input value per output location. Always specify which frame is the output grid and which field supplies coordinates.

Brightness constancy provides evidence, not certainty

One useful assumption is that a visible pattern keeps its brightness as it moves:

I1(p+F01(p))≈I0(p).I_1\bigl(p+F_{01}(p)\bigr)\approx I_0(p).

The difference between the two sides is a photometric residual. In the authored explorer, corresponding visible samples have exactly equal assigned intensities and a zero residual. That equality was built into the scene. It does not demonstrate an estimator recovering motion.

Real images violate brightness constancy through changing illumination, shadows, reflections, exposure changes, and motion blur. Repeated texture creates a different difficulty: several incorrect matches can have similar brightness. A small residual therefore supports a proposed match under assumptions; it does not prove that the match follows the same surface. Occlusion requires excluding the comparison even if two unrelated surfaces happen to share an intensity.

Why an edge cannot determine both components

For a sufficiently small displacement, a first-order expansion of brightness constancy gives one linear constraint:

Ixdx+Iydy+ΔI≈0,ΔI=I1(p)−I0(p).I_x d_x + I_y d_y + \Delta I \approx 0, \qquad \Delta I=I_1(p)-I_0(p).

IxI_x and IyI_y are spatial brightness derivatives, like the signed gradients in edge detection. The equation has two unknown displacement components. One pixel does not supply enough independent information to determine both.

For an illustrative vertical edge, take Ix=2I_x=2, Iy=0I_y=0, and ΔI=−6\Delta I=-6. The constraint becomes 2dx−6=02d_x-6=0. Displacements (3, 0), (3, 2) and (3, −4) all produce zero linear residual. Motion across the edge is constrained; motion along it is invisible to this local measurement. This is the aperture problem. In a completely flat patch, neither component is constrained.

A neighborhood containing gradients in different directions can resolve the ambiguity if its pixels share a motion. Adding the independent constraint 3dy−6=03d_y-6=0 identifies (3, 2) in this exact example. Noise, near-parallel gradients, large displacement and motion boundaries make practical estimation harder. The linear equation is a local approximation, not an exact test for arbitrary large motion.

Different estimators add different information

The Lucas–Kanade method fits a shared displacement to a small neighborhood using image gradients. Corners with varied gradient directions provide better constraints than straight edges or flat patches. Image pyramids help bring larger motions into a range where local refinement is useful.

Horn and Schunck combine brightness evidence with a preference for spatially smooth flow. Neighboring estimates then constrain each other, although smoothness can be inappropriate across object boundaries. Both approaches supply information beyond a single pixel's brightness equation.

Sparse flow tracks selected locations, often well-conditioned feature points. Dense flow assigns a vector across the image grid, including regions with weak direct evidence. Density describes coverage, not reliability. OpenCV's tutorial demonstrates sparse pyramidal Lucas–Kanade and dense Farnebäck flow. RAFT is a learned dense estimator: it compares per-pixel features through all-pairs correlation volumes and recurrently refines a flow field. None of these estimators runs in the explorer.

From correspondence to video understanding

Flow can align neighboring frames for stabilization, propagate information through a video, or expose motion patterns. These uses depend on direction conventions, visibility and estimation quality. Propagating a label through an occlusion needs additional reasoning because a missing match cannot safely carry that label.

Video representations answer a broader question: what features describe the evolving scene? The V-JEPA paper explores prediction in a learned representation space. Such video features can encode temporal information without being a pixelwise flow field. Correspondence is a geometric description of where image evidence moves; a video embedding serves a different representation objective.

If you found this explanation helpful, consider sharing it with others.

Mastodon