LYING WITH PIXELS 2
Filed Under NEWS OF INTEREST, SECULAR
VANISHING ACT
The procedure for offing people from a live video stream goes something like this. The first step is to align successive video frames, using fast computational techniques that detect and reorient patterns of pixels in successive frames. The computer stitches together the sequence of aligned frames into a mosaic that is constantly rebuilt as the computer receives new frames. In the case of skater Katarina Witt, the mosaics store information about what is behind her as she moves. While this is going on, fast frame-to-frame comparisons based on a Sarnoff-grown computational technique called “pyramid processing” enable the system to isolate and track objects moving in the foreground.
The key to pyramid processing is the succession of lower resolution versions of each frame it produces. Rather than having to compare full sets of pixels to detect frame-to-frame differences, such as Witt moving a few inches on the ice, the computer can rely on the reduced pixel sets. That greatly simplifies calculations, such as ones that track patterns of pixels that travel together through sequences of frames.
To erase Witt, the system summons the piece of the mosaicked background and inserts those pixels wherever the foreground pixel patterns change. By detecting which sets of pixels move with respect to one another and which sets remain relatively fixed, pyramid processing computers can track moving objects against stationary backgrounds. Since these calculations are doable in less than the thirtieth of a second between frames, there’s time to replace moving foreground pixels with background pixels. Do that and you can make a skater—or anything else—vanish.
HOW TO ERASE A SKATER
1. Take the video. Camera at rink produces a sequence of video frames. A computer stitches the frames together in a mosaic showing the entire sequence and background.
2. Find the subject. Software isolates and tracks the skater in the foreground, keeping track of the background being occluded at every instant.
3. Poof! She’s gone. Pixels from the background mosaic are substituted for the parts of the image in which the skater appears.