The main purpose of Faceware Analyzer is to track facial expressions quickly and accurately so those expressions can be applied to facial animation using Faceware Retargeter. While, AutoTracking is very powerful, getting the most accurate and highest fidelity tracking generally requires a user's input. While AutoTracking uses a generic model of the face, Pro tracking lets the user create the shapes for a particular actor. This increases the accuracy of the track and does a better job at picking up the kinds of small movements that really help to sell a facial performance. This tutorial will cover the basics of Analyzer Pro tracking.
*Note - This tutorial begins from the point immediately after creating a new job. For details on job creation in Analyzer, see the following article: New Job Creation
Step 1: Import Tracking Models (optional)
Import Tracking Model takes the models and training frames that were exported and brings them into Analyzer to be used on a new job. Detailed information about the Import/Export Tracking Model function of Analyzer can be found here: Importing & Exporting Tracking Models
To import tracking data into a new shot:
- Begin a new job as normal
- Select 'Import Tracking Job' from the File menu
- Select the folder where the data exists
- If successful, a confirmation message appears detailing what was imported
Step 2: Markup
The way one gives the software the information about the actor’s performance is to “markup” frames that have extreme and distinct expressions, creating training frames. These frames are then processed by the software to allow it to hit all of the in-between expressions. The user input is moving points (called “landmarks” in Analyzer) around the face in a specific way which is demonstrated below.
*Note - Important Note - Do not track on the Face group. This group is simply present for reviewing a shot after tracking is complete.
- Select the “Eyes” group from the toolbar dropdown menu. This will display the Eyes face group and the associated landmarks. Note that all of the face groups contain the Nose landmarks.
- On the first frame, move the eye and nose landmarks into the appropriate places, similar to the picture. This is most easily accomplished using “Intelligent Drag.” Hold Shift while dragging landmarks and the other landmarks will move into the approximate areas they should be. Continue to move the landmarks around until satisfied with their placement. Intelligent drag will only affect those landmarks that have not been placed by hand. Landmarks that have been moved by hand will turn green while other will remain blue (colors can be changed in the options menu).
- Note that when you place a landmark a green marker appears on the timeline. This indicates that you have created a training frame which will be used by Analyzer in its calculations. A training frame will be created automatically once one landmark on a given frame has been placed. The software will, however, use all the landmarks on a training frame to derive data.
- Scrub through the timeline and place training frames on the most extreme eye poses, such as blinks, open wide, and the extremes of pupil movement. If the shot is particularly long, you can hold Ctrl and drag on the timeline to select smaller sections to work on. This is also useful if there are some segments of the shot that are valid and some that are not.
*Note - There must be at least three training frames to continue, but there is no maximum. A good rule of thumb is to start with a few and add more as necessary. The number needed will depend on the performance, shot length, video quality, and the user’s skill level and can vary greatly.
*Note - If using an Imported Tracking Model, only one training frame is necessary.
Step 3: Training a Model
Once you are satisfied with the training frames you have created, it is time to send this information to the software to analyze. “Training a model” is the name of this process and it consists of Analyzer looking at the training frames you have created and producing a statistical model of the performance. This model will then be used to track the remaining frames in the shot.
- From the “Track” menu, the user can choose to train the model using all of the training frames in the shot or just using the training frames in a selected area. The “Train” button on the toolbar defaults to using all of the training frames.
- Analyzer will then begin analyzing the training frames. Click “ok” when the process is completed.
*Note - If using an Imported Tracking Model, there is no need to train unless more training frames are added by the user. In this case, use 'Train Model - All' in the Track menu.
Step 4: Tracking
Though the entire process of using Analyzer can be referred to as “tracking,” within the process this refers using the model created by the software to create information for the in-between frames of the shot.
- From the “Track” menu, select whether you want to track the valid frames, all of the frames, the selected frames, or the area between the two training frames before and after the current time. The Analyzer will take a moment and write to those frames.
- Now it is time to review the data by scrubbing through the shot or using the playback controls. Ctrl + Space will start and stop playback. The tracking should follow the actor’s performance closely. Things to look out for include extreme popping or sliding of the landmarks and any shapes you may have missed. If there are areas that need to be fixed, create some more training frames where appropriate, then train & track the shot again until you are satisfied with the results and the tracking smoothly and accurately follows the actor.
- Select the Nose landmarks and press Ctrl + L to lock them. They will turn grey and cannot be moved until you select them and unlock them. The nose is in each of the face groups to act as an anchor to help track more accurately. Once it is done in one group, locking it makes it so that you do not have to worry about it again.
At this point, it is time to repeat the process for the Brows and Mouth groups
Once you’ve tracked the Eyes, Brows, and Mouth, select the Face group from the menu and review your tracking. The Face group is not used in tracking, only for review. Once you are satisfied, it is time to parameterize. You can find details about parameterization here: Parameterization
Step 5: Export Tracking Models (optional)
Export Tracking Model gathers both the statistical models that were used in the shot as well as the individual training frames and saves them for use in other shots. The depth of the model and the number of training frames can be added to with each subsequent shot if desired, making a more complete model. It is not a necessary step, but can help if there are a lot of shots with the same actor.
To export tracking data from a finished shot:
- All three face groups must have been trained
- Select 'Export Tracking Model' from the File menu
- Choose a folder to save the files to. It is recommended that a separate folder is created for each actor's data.
- You should receive a confirmation message saying confirming a successful export
*Note - If you are exporting from a shot that already used the Import Tracking Model function, then you will be asked if you want to overwrite the data that already exists. Selecting 'Yes' will add the new models and training frames to the folder, but you will still have all of the ones you previously imported. This is how the data pool grows and the model becomes more complete.