Using Masks

Les s on 7

Use the powerful vector-based masks and

their Edge Gradient feature to hide areas in a
clip. Then, make the scene spooky by
changing the surface properties and by
applying special effects.

7 Using Masks

Overview
Used as a stand-alone feature or as part of a Paint operator, masks are valuable for hiding parts of an
image and for rotoscoping purposes. The enhanced vector-based masks allow the creation of masks
with advanced feathering and animation control. Masks are compatible with other Discreet products
such as inferno, flame, and flint, enabling a parallel creative workflow.
You can use masks to remove the area outside the mask shape and keep the area inside the mask shape,
or you can use masks to remove the area inside the mask shape and keep the area outside the mask
shape. In this lesson, import two image sequences and use masks to reveal different areas in the clip.
Then, change the surface properties of a layer and add a glow effect to give the scene a spooky look.
Time to complete this lesson: 60 minutes.

Exercise: Hide Part of the Scene with Masks

The scene includes two image sequences: one image sequence shows a woods background and the
other image sequence shows a character representing a ghost. Use masks to have the character
appearing from behind the trees in the foregroundand to make the character look spookier.
In this exercise:

Create a copy of the background layer.

Mask the copied layer to have some trees show in the scenes foreground.
Animate the mask to match the camera motion in the clip.
Mask the ghost layer using animated Edge Splines to animate the opacity of the ghost.
Adjust the ghost layer to give it a surreal look.
Need Help?
If you need help completing this exercise, open the Lesson07.cws file as a reference.

Using Masks

Set Up the Workspace

First, preview the result of this exercise. Next, create a branch to begin building the composite.
1. Check the combustion preferences. For instructions, see the Welcome to combustion document.
2. Choose File | Open Workspace, or press CTRL+SHIFT+O (Windows) or COMMAND+SHIFT+O
(Macintosh), to open the Lesson07.cws file in the Lesson07 folder. For instructions, see Open the
Workspace in Lesson 1.
3. Preview the result of this exercise by playing the clip in the viewport. For instructions, see Play the
Composite in the Viewport in Lesson 1.
Note: If you do not have enough RAM to view the entire clip in real time, minimize combustion and
play Lesson07.avi using Media Player (Windows) or QuickTime (Macintosh). For instructions, see
Playing Movies in the Welcome to combustion document.

The clip features a ghost coming from behind the trees with a sabre in hand.

4. Choose File | Close Workspace (CTRL+W / COMMAND+W) to close the Lesson07.cws file.
5. Choose File | New (CTRL+N / COMMAND+N) to open the New dialog and create a branch with the
following properties:

Type: Composite
Name: Masking
Format: NTSC DV
Duration: 35 frames
Bit Depth: 8 Bit
Mode: 2D

7 Using Masks

6. Select the two-viewport layout.

7. Import two image sequences:
a) In the Workspace panel (F3), right-click / CTRL-click the Masking composite and choose Import

Footage (CTRL+I / COMMAND+I).

b) In the Import Footage dialog, locate and open the Lesson07 folder.
c) Click Thumbnails and enable Collapse.
d) Open the Woods folder and click the woods[####].jpg image sequence.
e) Double-click the Parent folder, then the Death folder, and click the death[####].png image

sequence.
f) Click OK (ENTER).

The viewports show a character in a cape composited over a background.

8. Set the viewport display options:

a) Choose Window | Fit in Window.

The entire frame is displayed in the left viewport.

b) Click the right viewport and zoom out until you see the borders of the frame.

The entire frame is displayed in both viewports.

Using Masks

9. Rename the layers:

a) In the Workspace panel (F3), right-click / CTRL-click the death0000 layer and choose Rename.
b) Type Death and press ENTER.
c) Right-click / CTRL-click the woods0000 layer and choose Rename.
d) Type Woods and press ENTER.

Mask Part of the Background

Make a copy of the woods then, mask part of the copy to hide an area of the footage so the trees to the
left appear in front of and behind the character.
1. In the Workspace panel (F3), right-click / CTRL-click the Woods layer and choose Duplicate.
Note: You can also select the Woods layer and press CTRL+C / COMMAND+C to copy, and press
CTRL+V / COMMAND+V to paste the copied layer in the Workspace panel.
The character is no longer visible in the viewport because the duplicated layer is above the Death
layer in the Workspace panel. The duplicate layerWoods (2) layeris at the same position as the
original layer, now named Woods (1).

2. In the Workspace panel, rename the Woods (2) layer Trees.

The numerical tag is removed from the original layer.
3. Apply a mask operator to the Trees layer to hide part of the background woods:
a) In the Operators panel (F5), select the Mask category to access the mask operators.
b) Click Draw Mask.

7 Using Masks

4. In the Workspace panel (F3), double-click the Draw Mask operator.

In the Workspace panel, the viewport icon next to the Draw Mask operator indicates the Draw Mask
operator is the current operator in the active viewportthe right viewport. The two-viewport
layout enables you to draw a mask in one viewport and see the result composite in the other
viewport.
5. In the right viewport, draw a mask starting from the upper left of the frame and extend the mask
along the right side of the tree line, making sure the mask extends beyond the top, left, and bottom
borders of the frame.

The left viewport shows the character appearing from behind the trees.

Using Masks

6. Choose Window | Outlines Only or, right-click / CTRL-click the right viewport and choose Outlines
Only.
The right viewport reveals the background and displays only the wireframe of the mask.

7. Set the mask options:

a) In the Mask Controls panel (F8), click Modes.

b) Set Feather to 2 to soften the edges of the mask.

Note: Invert is automatically disabled for the Global Apply Mode, so the mask is not inverted. When
Invert is disabled, the mask removes the area outside the mask shape and keeps the area inside the
mask shape. In this exercise, you want to remove the area inside the mask.
8. Click the left viewport.

7 Using Masks

9. Play the clip (SPACEBAR).

Notice that there is a slight shift between the mask and the footage due to camera motion. You need
to adjust the animation of the mask.

Animate the Mask

Animate the control points of the polygon mask so the mask follows the tree line throughout the clip.
1. Click the right viewport to make it active.
2. Go to the first frame (HOME).
3. Enable Animate (A) to enable automatic keyframing.

4. In the Toolbar (F2), click the Control Points option to access the control points of the Polygon Mask
object.
5. Zoom in (CTRL+= / COMMAND+=) on the control points along the tree line.
6. In the right viewport, drag the control points along the tree line to adjust the mask.
7. Go to the last frame (END) and adjust the mask along the tree line.
8. Examine the result:
a) Click the left viewport.
b) Play the clip (SPACEBAR).

The mask follows the tree line throughout the clip.

Note: If you are not satisfied with the result, go to the frames where the mask needs further
adjustments and adjust the mask.

Using Masks

Animate the Character

Animate the character to have it appear from behind the trees, then add a mask operator to fade the
edges of the character.
1. Click the right viewport.
2. In the Workspace panel (F3), select the Death layer.
3. Set the position of the Death layer throughout the animation:
a) In the Composite Controls panel (F8), click Transform to access the Transform controls.

b) Go to the first frame (HOME).

c) Set the X Position to -30.
d) Set the Y Position to 120.
e) Go to the last frame (END).
f) Set the X Position to 25.
g) Set the Y Position to 130.

4. Add a mask operator to the Death layer to fade the edges of the layer:
a) Go to the first frame (HOME).
b) In the Operators panel (F5), click Mask to access the mask operators.
c) Click Draw Mask.

7 Using Masks

5. View the output of the Draw Mask operator in the right viewport:
a) In the Workspace panel (F3), click the Draw Mask operator under the Death layer.
b) Click the Send Up button.

6. In the right viewport, draw a mask (of about six to eight control points) around the character.
Note: If necessary, click the Polygon Mask tool in the Toolbar (F2).

The black background in the Death image sequence doesnt show in the final composite because the
image sequence was saved with its alpha (opacity) channel (as a 32-bit .png file) to create the cutout
of the ghost.

10

Using Masks

7. Add edge gradients to define the outer softness of the Polygon Mask object:
a) In the Mask Controls panel (F8), select Splines from the Edge Gradient list.

The Splines option enables you to adjust the inner and outer mask offsets at each control point.
b) In the Toolbar (F2), click the Edges button.

By default, the spline of the mask is shown in white and the inner and outer edges of the mask are
shown in green.
8. Add and position three control points:
a) With the cursor (

), click to add three control points at bottom of the mask.

b) Position the control points in a zigzag pattern.

11

7 Using Masks

9. Choose Window | Outlines Only, or right-click / CTRL-click the right viewport and choose Outlines
Only to no longer show only the wireframe of the mask.

Fade the Characters Mask

In order for the character to look like a ghost, feather the edges of the characters mask in an irregular
pattern, change the surface property of the Death layer, and then add an operator to the layer to give
the character a scarier look.
1. Animate the fading effect of the mask:
a) Go to the first frame (HOME).
b) Drag the control points of the masks spline, and its inner and outer edges to get a fade-out effect.

12

Using Masks

Note: The spline is shown in white and the inner and outer edges are shown in green.

c) Drag the control points of the masks spline, and its inner and outer edges at every 10 or 20
frames.

2. Click the left viewport.

3. Play the clip (SPACEBAR).
As the character appears from behind the trees, the mask gives the character a ghostly look.
4. Go to the first frame (HOME).
5. Select the single-viewport layout.

13

7 Using Masks

6. Improve the ghostly appearance of the character:

a) Disable Animate (A).
b) In the Workspace panel (F3), select the Death layer.
c) In the Composite Controls panel (F8), click Layer.

d) From the Transfer Mode list, select Gray (HSV).

e) Set Opacity to 80%.

14

Using Masks

7. Add a glowing effect to the character:

a) In the Operators panel (F5), click the Stylize category to access the Stylize operators.

b) Click Glow.
c) In the Workspace panel (F3), select the Glow operator.
d) In the Glow Controls panel (F8), set Radius to 12.

e) Set Strength to 2.

15

7 Using Masks

8. Play the clip (SPACEBAR).

A ghostlike character appears to be coming from behind the trees with a sabre in hand.
9. Close the workspace:
a) (Optional) Render the clip as a QuickTime movie. For instructions, see Render the Result
Composite in Lesson 1.
b) Close the workspace (CTRL+W / COMMAND+W).

Things to Remember
When creating composites, selecting 2D Mode as opposed to 3D Mode speeds up rendering time.
When you add a mask operator to a composite from the Operators panel, double-click the mask
operator in the Workspace panel to make the mask the current operator in the viewport. When you
add a mask operator to a composite from the Workspace panel, the Timeline, or the top menu, the
mask operator becomes the current operator in the viewport.

Choose Window | Outlines Only, or right-click / CTRL-click the viewport and choose Outlines Only
to display the background footage and only the wireframe of the mask.

By default, Invert for the Global Apply Mode is disabled. When Invert is disabled, the mask removes
the area outside the mask shape and keeps the area inside the mask shape.

The Edge Gradient Splines option enables you to adjust the inner and outer mask offsets at each
control point.

16

Acknowledgments
Lesson development:

Pam Fernandez, Peter Greenways, Amer Yassine

Content:

Paul Bracegirdle, Pam Fernandez, Richard Gratton, Justin Soles,

Amer Yassine

Review and testing:

Jose Belhumeur, Anne Brierley, Peter Greenways, Miriam Sterle

Editing:

Mylne Pepin

Technical Support:

Anna Maria Romeo, Sandy Andrade, Maxine Gervais, Chris Mackenzie,

D.J. Rahming, Vickie-Lynn Roy

Layout design:

Brenda Barrie

Project management:

Miriam Sterle

Images:

Jose Belhumeur, Pam Fernandez, Stphane Tremblay, Amer Yassine

Additional images:

La Fabrique dImages, Academy of Art College - Elisa Stephens

 Copyright 2002 Autodesk, Inc. All Rights Reserved.

This publication, or parts thereof, may not be reproduced in any form, by any method, for any purpose.
AUTODESK, INC. MAKES NO WARRANTY, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT
LIMITED TO ANY IMPLIED WARRANTIES, OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE, REGARDING THESE MATERIALS AND MAKES SUCH MATERIALS AVAILABLE SOLELY ON AN
AS-IS BASIS.
IN NO EVENT SHALL AUTODESK, INC. BE LIABLE TO ANYONE FOR SPECIAL, COLLATERAL,
INCIDENTAL, OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH OR ARISING OUT OF
PURCHASE OR USE OF THESE MATERIALS. THE SOLE AND EXCLUSIVE LIABILITY TO AUTODESK, INC.,
REGARDLESS OF THE FORM OF ACTION, SHALL NOT EXCEED THE PURCHASE PRICE OF THE
MATERIALS DESCRIBED HEREIN.
Autodesk, Inc. reserves the right to revise and improve its products as it sees fit. This publication describes the state
of this product at the time of its publication, and may not reflect the product at all times in the future.
TRADEMARKS

3d studio max, 3D Studio VIZ, Autodesk, character studio, cleaner, cleaner central, combustion, Discreet, fire,
flame, flint, frost, inferno, mountstone, smoke, stone, wire, and 3ds max, backdraft, backburner, Colour Warper,
edit slipstream, jobnet, Multi-Master Editing, Render Queue, Real-Time Roto, and sparks are registered
trademarks or trademarks of Autodesk, Inc./Autodesk Canada Inc. in the USA and/or other countries.
THIRD-PARTY TRADEMARKS

All other brand names, product names, or trademarks belong to their respective holders.
GOVERNMENT USE

The software and documentation is provided with RESTRICTED RIGHTS. Use, duplication, or disclosure by the
United States Government or any agency, department or instrumentality thereof is subject to the restrictions set
forth in the Commercial Computer SoftwareRestricted Rights clause at FAR 52.227-19 or the Commercial
Computer SoftwareLicensing clause at NASA FAR Supplement 1852.227-86.
Images Courtesy of:
Title:
Part number:
Software version:
Print version:
Date:

Additional images courtesy of La Fabrique dImages and The Academy of Art College
- Elisa Stephens.
FRPEXVWLRQ  Demo Web Tutorials
62202-011300-5081A
2.1
NA
29 November 2002