Facial capture - how to join the dots

Facial motion capture now lies within the reach of even small studios. From running a mocap shoot to cleaning data and setting up facial rigs, these expert tips will help you implement it in your own pipeline.

Once used solely to record full-body movements, motion capture is increasingly becoming a standard part of the facial animation pipeline, both for visual effects and games projects. In this article, six of the leading experts in the field provide their tips for getting the most of facial motion capture in your own projects.

Several different methods may be used to capture facial motion data. Pixel flow and structured light techniques give you an animated, textured mesh without the need for a complex rig, but the subject has very little freedom of movement, so body capture has to be done separately. Much of the data will also only be relevant if you are creating a digital version of the performer.

Therefore, we will focus mainly on marker-based and image recognition techniques. Both can be performed alongside body capture, but neither provides the dense 3D data of the previous techniques. As a result, the data recorded from the live actor must be applied to a facial rig: usually one that combines joints, blendshapes and muscle simulation. In turn, this deforms a facial mesh to generate a 3D animation of the real-world movements.

Marker-based facial motion capture, the technique used by our experts from Centroid 3d, uses widely spaced multiple cameras to track markers applied to the face. In contrast, Image Metrics’ markerless performance-capture technology uses an image-processing system to ‘recognise’ and track the features of the face from a single video camera.

In the tips that follow, we will be exploring the process of capturing and employing facial motion-capture data in a production pipeline, including running a successful shoot; how to generate clean facial topology and blendshapes; and how to create a rig that unites the two, including such key properties such as ‘soft eyes’ and ‘sticky lips’.

When producing animation, a distinction is often made between motion capture and hand-keyed animation. In production, this distinction is rarely cut and dried. There may not be anything technically wrong with the mocap data, but the performance it generates may not be quite right in the context of the scene. Therefore, the final section of the tips will explore the process of tweaking data by hand. We’ll also provide a brief overview of the Facial Action Coding System (FACS) used as the theoretical basis of most modern facial animation work.

RUNNING A SHOOT

01. USE ENOUGH MARKERS
With marker-based capture, the more markers you use, the more facial animation detail that can be captured. Use too many, and the complexity of the data can work against you. At Centroid, we use at least 17 markers; more for complex shoots.

02. CAPTURE A FACIAL ROM
Obtaining a facial ROM (Range Of Movement) prior to marker placement is key to understanding how your performer’s face will work on the shoot. This means filming the actor going through a range of exaggerated expressions (big grin, extreme sadness, and so on) to determine the maximum range of movement of his or her face; and ideally, recording test data based on those movements. Effectively, it’s a high-tech version of gurning…

03. MARKER PLACEMENT
As well as the main features, marker placement should follow the muscle groups of the face. There are 11 main muscle groups in the face (for a diagram, see here), all of which interact with overlying fatty tissues. For an example of typical marker placement, see the images on this page. Interesting character traits (fat folds, jowls and other wobbly bits) should also be markered. Faces with character add believability to the captured data.

04. GET THE DATA CLEANED
Raw capture data needs a little ‘data love’ before it can be used. This process is analogous to that for cleaning full-body data and involves quietening down any noisy channels and reconstructing any markers that have been temporarily occluded (hidden from the camera by the actor’s head movements) during the shoot. Get the studio to do this. Turnaround times will depend on the complexity of the data, but a reputable mocap studio should be able to deliver ultra-clean data in days, not weeks.

05. CAPTURE EXTRA DATA
If the animation data will be plugged into a CG version of the same actor, getting a 3D scan of the actor prior to the shoot ensures that the data marries up perfectly. If you are capturing animation for an historic personality, scanning a look-alike can go some way to achieving the best result possible, even if you only end up using key features as opposed to the whole. If you are capturing dialogue, remember to record the audio too. This sounds obvious, but it’s sometimes overlooked – and uttering the immortal words “I forgot to press Record” will do you no favours whatsoever. Mic the actor up properly before you start capturing data, use time code, and log your shots properly. This will enable you to marry up sound and animation perfectly, rather than fumbling around by guesswork.

QUICK TIPS

06. When casting, look for actors with strong facial features: they capture better.

07. Cast actors who have trained in mime. They act well with their faces alone.

08. Beards play havoc with a facial shoot. Get the actors to shave before you begin!

09. Similarly, cleanse first. Oily skin can prevent markers from staying in position.

10. If in doubt, ask the actor to overact: you can always scale the data down later.

PREPARING THE MODEL

11. MIMIC NATURAL TOPOLOGY
Construct your mesh to reflect what is under the surface of the face as much as what is visible. Lay out circular, linear and sheet muscles before thinking about what surface contours need to be added. Then resolve these lines to edge loops.

12. USE ‘RELAXED SCULPTING’
Pronounced facial features often get in the way when sculpting blendshapes, particularly in the mouth and nasal region. One solution is to create an intermediate shape so that you can work on a relaxed (featureless) version of the face that will then drive the final (featured) face. This enables you to smooth the deltas (vertex offsets) of the blendshape without smoothing details of the face.

13. GENERATE OBJS VIA THE RIG
Create a node and add channels to it that correspond to all of your facial movements (jawOpen, upperLipRaiser, pucker, and so on). Use this node as a common interface to drive your blendshapes, joints, clusters or even a muscle rig. If you require a full blendshape solution, write a script to go through each channel, set its value to 1 and export the result as an OBJ file. This means you can create facial movements using a combination of complex deformers, but always publish the results as blendshapes. Wrap-deforming your latest mesh on the rig before exporting will give you flexibility over the point order and resolution of your final mesh, without risking breaking the rig.

14. SET UP A ‘BAD MIX’ LIBRARY
When blendshapes combine, they often spike and overload certain areas of a model. When this occurs, you need to work out if it’s a problem with the animation or with the shapes themselves. To do this, don’t work problems individually. Instead, save the animation out as a pose: before long, you’ll have a library of bad mixes. Shapes that don’t work together will recur throughout this library, making them easier to spot.

15. USE CORRECTIVE SHAPES
Corrective blendshapes are usually associated with body rigs, but use them on joint-based facial rigs too. Sometimes joints alone are incapable of creating appealing shapes: for example, a ‘kiss’ mouth shape. A layer of corrective blendshapes – usually applied procedurally – fixes these ugly shapes. If your programmer says, “No blendshapes in the game engine,” try to strike a deal: your corrective blends will have very small delta (vertex offset) counts, and only these deltas will be stored per blend, not the entire head geometry.

QUICK TIPS

16. Need to determine the ‘flow’ of an actor’s face? Just sketch over a photograph.

17. To see where edge loops should go, look at the pattern of wrinkles on old people’s faces.

18. Create a ‘stripped’ version of your mesh without accessories such as ear rings, beards or eyelashes…

19. ...then Wrap-deform the final mesh to the stripped mesh: it makes work simpler

20. Don’t go mad with blendshapes: even a PS3 title like Heavenly Sword only uses 40

THE FACIAL RIG

21. SET UP AN OSIPA-STYLE UI
Animators hate keyframing joints or blendshapes directly: they need a user interface. Create an ‘Osipa-style’ UI (after LucasArts senior character TD Jason Osipa, author of Stop Staring: Facial Modeling and Animation Done Right!) and arrange the control boxes to correspond roughly to an actual face: eyebrow controls on top, eye controls below those, and so on.

22. MINIMISE CONTROL BOXES
Don’t put in too many control boxes in your UI. Around 15–20 is normal. In Maya, you can multi-select channels in the Channel Box, meaning you can use fewer control boxes in the scene for a tidier interface. The equivalents of the Channel Box in 3ds Max and XSI are less user-friendly, lacking multi-select.

23. USE FLESHY EYES
The fleshy eyes (or ‘soft eyes’) effect, in which the movement of the eyeball deforms the eyelid, is essential for realistic animation. There are many ways to set this up, but the simplest is often the best: paint some skin weights around the eyelids, so that when the eye joints rotate, they tug on the eyelids. The top lid should get more weight than the bottom lid. If you have blendshapes in your rig, make sure that the blendshape deformation happens before the joint deformation in the deformer stack.

24. MAKE LIPS STICKY
‘Sticky lips’ is the effect of the lips peeling apart as the mouth opens. To add it to your rig, simply create two or three sequential blendshapes that invert the vertices around the ‘sticky lips edge loop’, starting from the corners and moving towards the centre of the lips. You must design left and right-sided shapes for asymmetry.

25. USE ANIMATED NORMAL MAPS FOR FINE DETAILS
Using animated normal maps to generate the fine deformations of a character’s face gives game animation a new level of realism. For example, a simple eyebrow raise conveys a much stronger emotion when the forehead creases appear. In the high-throughput world of games, the facial rig must drive these maps procedurally. A base set of normal maps to design would be: brow raise, brow squeeze, eye squints, eyes shut tight, nose sneer, smile, chin raise and neck strain. For cartoon-style characters, you can even go crazy and add some veins popping out around the temple area.

QUICK TIPS

26. Need to see the Osipa UI in detail? Buy his book: it was reissued by Sybex in 2007 (ISBN: 978–0471789208).

27. Don’t be too purist. It’s often preferable to mix ‘in scene’ and Channel box controls.

28. Need the effect of skin sliding over bone? Use blendshapes; you can’t do it with joints.

29. Don’t make sticky lips a procedural effect: animators know best when to apply it.

FACIAL CODING AND ANIMATION

30. UNDERSTAND FACS
While a face can make more than 10,000 expressions, we can evaluate an expression in milliseconds. But what exactly are we reading? In the 1970s, two psychologists, Paul Ekman and Wallace Friesen, set about codifying this facial language. They calculated that there were many individual muscle-based facial actions, referred to as action units (AUs), that make up facial expressions: a categorisation known as the Facial Action Coding System (FACS). The manual that Ekman and Friesen devised to train others in reading AUs has proved very useful in interpreting facial motion capture. Facial rigs based on this system stand a better chance of recreating the full fidelity of this data. Revised in 2002 with the help of Joseph Hager, the FACS manual is now available on CD-ROM from www.face-and-emotion.com. If you can’t afford the $260 price tag, Paul Ekman’s book, Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life (2007, Holt Paperbacks, ISBN 978-0805083392), costs just $15.

31. USE FACS…
Base your naming on FACS to communicate the structure of your rig to people who haven’t previously used it, both inside and outside the studio. This applies particularly to your mocap provider. One way to kickstart this process is to sculpt complete expressions, then break them down into FACS units using masks. Most packages have a way of masking blendshapes.

32. ...IN MODERATION
However, don’t follow FACS religiously. Sometimes the animator needs ‘fake’ controls that generate non-physically realistic shapes – especially at the end of a project when the director wants something very specific and time is up… Remember that FACS is based on what a real human face can do, not what polygons can do. A facial rig is never final: an animator should be able to request a control like ‘Cheek Volume’ at any moment in the production.

33. USE SCALE AND OFFSET…
When editing mocap data, avoid reducing curve density, or you risk losing the subtleties that you’re paying good money for. Instead create layers to scale and offset the performance, and employ these where necessary. This non-destructive workflow will also allow you to update the mocap easily if the solve is improved in future. Create presets for typical scale and offsets: you’ll find that you use them again and again. Motion-capture data is often ‘off’, but providing it’s consistently off, presets are your best friend.

34. ...OR INTERPRET YOUR DATA
Motion capture gives you either the 3D coordinates of dozens of markers or a unique blendshape. Neither is easily editable. As we’ve just seen, one way around this is to use layers to scale and offset the raw data. This works well if the animator doesn’t expect to stray too far from the mocap reference. If the animator wants the option of interpreting the data more freely, it might be better to make it look as much as possible like keyframe animation the animator can modify. Facial rigs have controls that are designed for use by animators: use these to emulate sample frames of mocap data, then use these comparisons to generate a conversion script to fill in the in-betweens.

QUICK TIPS

35. Don’t rely on mocap completely: instead, think of it as traditional in-betweens…

36. ...so make sure someone ‘owns’ each pose, and ensures that goals are hit.

37. Watch out for the eyes when using mocap data: they almost always need tweaking.

38. When scaling data, use the formula: animation = (mocap x scale) + offset.