An image records where things appear at one instant. Two frames add a new question: where did the same visible pattern go? Visual correspondence pairs locations across images. Optical flow describes these pairings between video frames as a displacement field, giving a direction and distance for each selected image location.
A moving object can create flow, but so can a moving camera looking at a stationary scene. Flow is measured in the image plane. It does not directly tell us an object's 3D speed or separate camera motion from object motion. OpenCV's optical-flow tutorial introduces this apparent-motion interpretation. Camera geometry provides optional context for how 3D motion changes image coordinates; the correspondence problem itself can be understood entirely in 2D.
A vector belongs to a source pixel
Let and denote two frames. A source location has forward flow when its corresponding location in frame 1 is:
Here x increases to the right, y increases downward, and both displacement components use image pixels. A sample at (4, 4) with flow (3, −1) reaches (7, 3): three pixels right and one up. This is displacement between these two frames. Dividing by their time interval would give image velocity in pixels per second, still not meters per second.
Compare the object and background samples, then switch to the camera case. With the object selected, set horizontal displacement to 3 and vertical displacement to 0, and choose Occluder to hide its destination. Increase horizontal displacement to 8 to move it outside the image.
Frame 0 · source
Frame 1 · target
Visible correspondence. The gather sample matches the source intensity.
The teal object moves; the background stays fixed. A moving object can also cover a background sample. Image x increases rightward and y downward. The frame-1 arrow starts at the old image coordinate and ends at the projected location.
Authored 12 × 8 frames and exact integer correspondences, not estimated or measured flow. Colors identify surfaces; the readout reports their assigned grayscale intensities.
The camera case deliberately gives every surface the same image translation, as a flat equal-depth scene could under a suitable camera translation. Real camera translation generally produces different displacements at different depths; camera rotation and perspective introduce further variation. A uniform shift is a teaching case, not a general camera-motion model.
Missing correspondence is different from no motion
A stationary visible sample has valid flow (0, 0). An occluded sample moves behind another surface: its projected location may be inside frame 1, but the pixel there records the foreground surface. An out-of-frame sample projects beyond the captured image. Neither case means that the original sample stopped moving.
The explorer knows these motions because its scene was authored. An estimator given only the rendered frames would have to infer them, and an occluded sample supplies no direct visible match. Keep a visibility or validity mask alongside flow when reconstructing images or comparing corresponding samples. Conversely, a newly revealed region in frame 1 may have no visible source in frame 0.
Flow direction and sampling direction are separate choices
Forward flow is indexed on frame 0. Backward flow is indexed on frame 1. For a mutually visible correspondence, their relationship is:
The reverse vector belongs at the corresponding location , not generally at the original array index . With the object moving three pixels right, forward flow at (4, 4) is (3, 0), while backward flow at (7, 4) is (−3, 0). Forward flow at (7, 4) can describe stationary background instead. Negating a forward-flow array at the same indices does not construct the backward field.
To reconstruct frame 0 on its own grid, visit each source position and gather a sample from frame 1:
This is a backward, or gather, resampling operation using a forward flow field. Each output position asks where to read its input. Fractional locations require interpolation; the explorer uses integer positions so no interpolation is needed. Occluded and out-of-frame samples are masked.
A forward splat instead visits frame-0 samples and deposits their values at target positions to construct frame 1. Multiple samples may land at one destination, while other destinations receive none. Resolving overlaps and holes is a separate problem from gathering one input value per output location. Always specify which frame is the output grid and which field supplies coordinates.
Brightness constancy provides evidence, not certainty
One useful assumption is that a visible pattern keeps its brightness as it moves:
The difference between the two sides is a photometric residual. In the authored explorer, corresponding visible samples have exactly equal assigned intensities and a zero residual. That equality was built into the scene. It does not demonstrate an estimator recovering motion.
Real images violate brightness constancy through changing illumination, shadows, reflections, exposure changes, and motion blur. Repeated texture creates a different difficulty: several incorrect matches can have similar brightness. A small residual therefore supports a proposed match under assumptions; it does not prove that the match follows the same surface. Occlusion requires excluding the comparison even if two unrelated surfaces happen to share an intensity.
Why an edge cannot determine both components
For a sufficiently small displacement, a first-order expansion of brightness constancy gives one linear constraint:
and are spatial brightness derivatives, like the signed gradients in edge detection. The equation has two unknown displacement components. One pixel does not supply enough independent information to determine both.
For an illustrative vertical edge, take , , and . The constraint becomes . Displacements (3, 0), (3, 2) and (3, −4) all produce zero linear residual. Motion across the edge is constrained; motion along it is invisible to this local measurement. This is the aperture problem. In a completely flat patch, neither component is constrained.
A neighborhood containing gradients in different directions can resolve the ambiguity if its pixels share a motion. Adding the independent constraint identifies (3, 2) in this exact example. Noise, near-parallel gradients, large displacement and motion boundaries make practical estimation harder. The linear equation is a local approximation, not an exact test for arbitrary large motion.
Different estimators add different information
The Lucas–Kanade method fits a shared displacement to a small neighborhood using image gradients. Corners with varied gradient directions provide better constraints than straight edges or flat patches. Image pyramids help bring larger motions into a range where local refinement is useful.
Horn and Schunck combine brightness evidence with a preference for spatially smooth flow. Neighboring estimates then constrain each other, although smoothness can be inappropriate across object boundaries. Both approaches supply information beyond a single pixel's brightness equation.
Sparse flow tracks selected locations, often well-conditioned feature points. Dense flow assigns a vector across the image grid, including regions with weak direct evidence. Density describes coverage, not reliability. OpenCV's tutorial demonstrates sparse pyramidal Lucas–Kanade and dense Farnebäck flow. RAFT is a learned dense estimator: it compares per-pixel features through all-pairs correlation volumes and recurrently refines a flow field. None of these estimators runs in the explorer.
From correspondence to video understanding
Flow can align neighboring frames for stabilization, propagate information through a video, or expose motion patterns. These uses depend on direction conventions, visibility and estimation quality. Propagating a label through an occlusion needs additional reasoning because a missing match cannot safely carry that label.
Video representations answer a broader question: what features describe the evolving scene? The V-JEPA paper explores prediction in a learned representation space. Such video features can encode temporal information without being a pixelwise flow field. Correspondence is a geometric description of where image evidence moves; a video embedding serves a different representation objective.
Related concepts
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
Learn how DETR assigns unordered object targets to prediction slots, why greedy choices can fail, and how matching differs from training loss and evaluation.
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
Convert between xyxy, xywh and center-size boxes, normalize coordinates correctly, and avoid mistakes when resizing, clipping or flipping detection labels.
Follow a 3D world point through camera extrinsics, depth division and pixel intrinsics, then see why one image point cannot determine its depth.
Follow a query from a spatial reference through learned offsets, bilinear feature sampling and weighted aggregation, with exact non-square coordinate arithmetic.
