Chapter One
The Basics of Matchmoving
Anytime a computer-generated (CG) element needs to be placed into a live-action sequence or vice versa, a matchmove is required. But what exactly is matchmoving?
Matchmoving is the process of matching CG elements into live-action footage. As a result, it's a crucial part of many visual effects shots. Despite its importance, it is completely invisible in the final shot-that is, if it's done right.
In this chapter, I explain the key steps involved in a matchmove and how matchmovers work with the rest of the visual effects team. I've also included a tutorial that is designed to help you get comfortable working with cameras and perspective.
A Typical Matchmove
In order to better understand what matchmovers do, let's consider a typical visual effects shot. The director has called for a CG creature to crash out of window of a building and run across the street and into an alley. Because the monster will need to interact with the window, the visual effects supervisor decides that in addition to the monster, the window-shattering effect should also be done in the computer.
On the day of shooting, the director makes artistic decisions as to how he wants to shoot the scene and eventually decides on a camera position that he likes. There is an opening on the building where the window should be, although the panes of glass are missing. The director and the cameraman practice the camera move a few times and watch the video playback to see how it looks. When they are filming, they move the camera as though it were following the monster crashing through the window and running across the street, even though the monster isn't there. Extras react to the imaginary beast, and props around the window are rigged with monofilament string (fishing line) to be pulled down on cue as though they were knocked over. Once the director is happy with the shot, the film is sent off to be digitized and then given to the visual effects artists to add the monster.
When the visual effects studio receives the digitized sequence (known as a plate), they decide that they will need an animator to animate the creature and a technical director (TD) to do the glass-shattering effect. And, of course, they'll need a matchmover to matchmove the plate.
The visual effects artists' goals are to make their 3D elements look as realistic as the scene that was filmed. The animator will need to make the creature move as though it were really crashing through a window, and the TD will need to make the window shatter like a real window. The matchmover needs to figure out where the camera was and how it was moving when the scene was filmed.
Matchmovers play an important role in this case, because in order for the creature and window to appear matched realistically with the scene, they need to make sure that the CG objects are "filmed" the same way with their CG camera as the real set was filmed with the real camera. Consider the window that needs to shatter-if the perspective of the window doesn't match the perspective in the plate, it will look out of place. Furthermore, if the real camera moves to the left and the CG window stays put, everyone will know it's a fake.
In our example effects shot, the visual effects supervisor measures key items on the set. For example, she measures the size of the opening of the window as well as its height off the ground. She measures the distance across the street and the size of the opening to the alleyway. She draws a rough picture of the set and makes notes about positions of certain props and lights that might be useful to know. She also measures how high the camera is off the ground, what lens is used, and how far it is from the window.
Typically the animator, TD, and matchmover all start at the same time. Since there are some measurements of the set, the animator knows how high the creature needs to jump to get through the window and how far to make it run across the street. The TD knows the size of the window that needs to shatter and how high it is off the ground. This is enough information to allow them to start setting up their scenes.
While they're doing that, the matchmover starts by first examining the footage to get an idea of how the camera was moving during the shot. He brings the footage into his matchmoving software and begins to track the 2D features in the scene as they move around the screen. 2D tracking usually involves identifying things in the scene that don't move (such as the corner of a building) and then letting the software follow that feature as the footage plays.
Once the matchmover has tracked a number of 2D tracks, the software analyzes these tracks and computes the position of the camera in relation to the items in the scene. At the end of this process, the matchmover exports a scene to his 3D-animation package that includes an animated camera and the 3D positions for all of the features he tracked.
Once the matchmover is happy with the camera he has generated, he goes about fitting that camera into a CG scene that is the same size as the one the animator and TD are using. When he's finished, he is able to look through his CG camera at the CG window and creature, and they appear in the right perspective and scale when compared to the original live-action footage.
Finally, he saves the scene with the matchmove camera in it. The animator and TD both use this camera to render their animations. If they've placed their window and creature in the right place in the environment, then they don't need to worry about whether the perspective matches or whether movement of the camera is the same. As long as they've rendered it using the matchmove camera, it will appear matched into the footage (Figure 1.1).
From 3D to 2D and Back Again
No discussion of matchmoving would be complete without discussing cameras. For matchmovers, it is important to understand how real-world cameras work and how they make the images we see on the screen. I'll discuss cameras more in-depth in Chapter 6, "Cameras," but for now there are a few basic concepts about real-world cameras that are important to know before getting started.
When a real camera films a scene, it is basically doing one thing: capturing the three-dimensional world as a two-dimensional image. That is, it gathers light from the 3D world around us and records it in a 2D form, such as a piece of film or a digital image. Let's consider for a moment exactly how this happens.
The light from the scene passes through the camera lens and is focused onto film that rests in the film gate on the back of the camera's inner chamber. The shutter closes, the film advances, the shutter opens again, and the process repeats. In the case of digital cameras, the film and film gate are replaced by a CCD (Charge-Coupled Device) that electronically captures the light information and records it to some sort of memory device.
The cameras in a 3D-animation program are based on real cameras, but they are represented in a slightly different manner. 3D-animation cameras represent a mathematically perfect model of the optics and construction of a real camera. Like real-world cameras, they have a focal length and a film back (the equivalent of a film gate). But rather than capturing light from the real world, they are simply capturing information of the synthetic, computer-generated environment in which they have been placed.
Whether you're dealing with exposed film or a 3D render, the resulting image is a projection. That is, the three-dimensional scene is flattened out into a two-dimensional representation of that scene. We have become so accustomed to these flattened images that we hardly notice them anymore, but every time we watch TV or a film, we are watching a flat recording of a three-dimensional scene (Figure 1.2).
The Matchmoving Process
So if a camera's purpose is to take the three-dimensional world and make a twodimensional image, a matchmover's job is the exact opposite. A matchmover must take a two-dimensional image and create a three-dimensional world. The portal between these two halves is the camera. If information about the camera can be reconstructed, it will go a long way toward figuring out how the 3D environment must have been at the time of filming. This information-the 3D environment and the camera-is what a matchmover will ultimately deliver to the animators to work with.
A matchmover's workflow (Figure 1.3) generally follows the same pattern for each shot, although there are a variety of ways to complete each task. The figure shows a typical matchmove pipeline. In the following sections, we'll examine each of these steps more closely.
Evaluate the Footage
This is perhaps the most important step in the matchmoving process, and unfortunately it is often overlooked. There are many ways to solve a matchmove, and careful scrutiny of the plate can help decide the appropriate tool to use, what pitfalls to watch out for, and how long it might take. This last item is particularly important, since clients and supervisors often put it at the top of their list of questions.
One reason the evaluation process is so often glossed over is that many of the things that determine the difficulty of matchmove require some experience to judge. Some typical questions asked during the evaluation of a shot could include:
What does the camera seem to be doing? Is it moving, and if so, how? Is it locked off or panning? How fast is it moving?
What is visible in the shot? Are there tracking markers? Is anything blocking the markers?
What format is the plate? Was it shot on film? DV? HD? Is there excessive compression, grain, or noise on the images?
What needs to be placed in the shot? How accurate does it have to be?
Who will be using the matchmove, and how will they be using it?
Of course, these only scratch the surface, but the more questions you ask, the more you will know what you need to do. The tutorials in this book are designed to help you learn what these key questions are and how to deal with their implications. A more thorough list of questions are included in Appendix A. The Evaluation Checklist there can be used as a guideline to help determine the difficulty of a matchmove.
Applying Information
As I've said, solving a matchmove can be like solving a puzzle (it's no coincidence that it's referred to as "solving" a matchmove). And as such, the more information there is, the easier it should be to achieve good results.
The amount of information given to matchmovers can run the gamut. There might only be an image sequence and nothing else, or perhaps someone was allowed on set to record all the camera information and take measurements. Usually it's somewhere in between. But the good news is that a surprisingly small amount of data can go a long way toward solving the matchmove.
The following are typical data a matchmover might include:
Camera information Such as focal length, aperture, and film type.
Set measurements Including camera height, focus distance, and measurements of various items in the shot.
Survey data This is very detailed measuring of the set, usually done by a professional surveyor.
Define the Camera
As stated before, the matchmover's job is to define all of the internal and external parameters of the camera, and there are many ways to do that. Knowing which method to use and under what circumstances comes with experience, but sometimes the only way to solve the matchmove is to experiment and see what works best. In broad terms, there are two major ways of solving for the camera: manual and automatic.
The manual methods harken back to the days before software existed to help matchmovers. This category would include perspective matching (matching the perspective of a single background image, rather than an image sequence, which is covered later in this chapter) and old-fashioned hand-tracking. This method of tracking a sequence involves making a speculation as to the camera's position and then refining it over many iterations until a match is achieved. Tracking a camera by hand is no small feat. It can often take weeks to truly figure out what is happening, because the process is nothing more than making educated guesses and tweaking them until they work.
In the past five or six years, software has emerged that allows a matchmover to track cameras somewhat automatically using a sophisticated technology known as photogrammetry. These software packages (which are covered in the next chapter) usually have a similar workflow to them. First, features in the image such as props in the scene or tape markers (commonly used on blue screen footage to mark points on the wall) are "tracked" as they move around the image in 2D. Then the software performs a calibration (or solve) for the camera by mathematically analyzing the movements of the 2D tracking markers. These packages usually generate an animated camera and 3D markers or nulls that represent the three-dimensional location of features that have been tracked in 2D. Matchmovers use this method most often since it is the easiest way to achieve a solution.
Some methods borrow from both manual and automatic techniques. Oftentimes, these are customized solutions that revolve around both types of workflows. For example, the 2D tracking information from matchmoving software could be used with a custom script that allows the matchmover to solve for pan shots.
Set Fitting
While cameras are the primary concern, they are only half of the process of matchmoving. Matchmovers must not only uncover all the facts about the camera, but they must also reconstruct the spatial layout of the environment on the live-action plate.
Figure 1.4 shows why this information is important. The first image shows an incorrect camera and building placement-what a mess. The second image has the correct camera, but the building is too close to the camera; therefore it doesn't match. But notice the third image. In this case, the building is in the correct position and distance from the camera, but since the camera isn't correct, the building still doesn't line up. The last image shows how it should be matched up with the correct camera and building placement. These images illustrate how matchmoving is not just solving cameras, and not just solving environments, but also figuring out the relationship between the two.
How much of the environment does the matchmover need to reproduce? That depends on what is being placed into the footage. If it is simply a character walking by, the animators and TDs might only need a simple ground plane. Other scenes might need rough geometry in order to cast 3D shadows. In some cases, such as digital set extensions, the matchmove might require an extremely accurate camera, detailed geometry, and spot-on positioning. Before beginning a matchmove, it is important to find out what type of 3D object is going into the scene and exactly where it will be placed.
3D environments might come from a variety of sources. Oftentimes, matchmovers create the rough geometry themselves or are provided a set to "fit" into the plate. And in some situations, the matchmover may provide a rough set from which a more detailed set is later constructed by a modeler. Many times, matchmovers use 3D markers they've calculated during the matchmove to provide information about the spatial relationships of the scene. But regardless of where the information comes from, it is often the matchmover's responsibility to establish the environment and set up the scene so that other artists further down the production pipeline don't need to worry about it.
Testing the Matchmove
Once the matchmove is solved, it needs to be tested for accuracy. A bad matchmove usually shows up as an obvious disconnect between the live-action plate and the CG elements. For example, the CG element seems to follow the motion and rotation of the live-action scene, but then suddenly the CG element pops to another location or gradually drifts away from the feature it's supposed to be resting on. Testing the matchmove consists of compositing the 3D objects over the image sequence and watching as the sequence moves to see if there are any unusual pops, drifts, or jitter.
(Continues...)
Excerpted from Matchmovingby Tim Dobbert Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
