Super Special Effects

For more than 10 years, superheroes have been a cinema staple. We’ve seen Iron Man soar through the air battling aliens, Thor summon lightning to strike down his foes, the Hulk turn green, and more. But real-life actors can’t fly, create lightning, or transform. It takes some high-tech creativity to make the leap from comic book page to movie screen.

That’s the job of visual effects designers like Marvel’s Dan DeLeeuw. He and his team use computer science to make the impossible possible. DeLeeuw is the visual effects supervisor for Avengers: Endgame, which hit theaters last month.

Visual effects are “anything you can’t actually make on set,” says DeLeeuw. The makeup and prostheses actors wear, the props they hold, and the realistic set pieces they act on are all practical effects. The actors can physically wear or interact with them. Everything else, like Iron Man’s high-tech flying suit or the towering body of alien villain Thanos, must be made digitally. And visual effects have come a long way since Marvel started making movies based on its classic comics in 2008. “We’re at a point where we can tell comic book stories the way Stan Lee and other comic book authors could envision them in their heads and in what they drew,” DeLeeuw says. 

For more than 10 years, superheroes have been a hit at the movies. People love watching Iron Man fly, Thor summon lightning, the Hulk turn green, and more. But real-life actors can’t do any of those things. It takes some high-tech creativity to bring comic book stories to life on a movie screen.

That’s the job of people like Dan DeLeeuw. He’s a visual effects designer at Marvel. He and his team use computer science to make the impossible possible. DeLeeuw was in charge of visual effects for the movie Avengers: Endgame, which hit theaters last month.

There are two kinds of effects in a movie. Anything the actors wear or physically interact with is called a practical effect. That includes makeup and prostheses, the props actors hold, and the realistic set pieces they act on. Visual effects, on the other hand, are “anything you can’t actually make on set,” says DeLeeuw. Things like Iron Man’s high-tech flying suit or the towering body of alien villain Thanos must be made digitally.

Marvel started making movies based on comic books in 2008. Visual effects have come a long way since then. “We’re at a point where we can tell comic book stories the way Stan Lee and other comic book authors could envision them in their heads and in what they drew,” says DeLeeuw. 

Before any filming begins, DeLeeuw and his team are already hard at work. Visual effects can take a long time to complete—between six weeks and six months. DeLeeuw will meet with the film’s director and go over the storyboards, or rough illustrations that represent each key movie scene. They’ll work together to decide what effects can be done practically and which ones will need to be made digitally. “If you stand on it or touch it, that’s something you should build practically,” DeLeeuw says, “and we’ll expand [the digital effects] out from there.”

Visual effects are drawn and coded using a computer, making them a type of digital animation called computer-generated imagery, or CGI. Everything from Guardians of the Galaxy’s out of this world backgrounds to Iron Man’s super suit were made with CGI.

DeLeeuw and his team start working before any filming begins. Visual effects can take anywhere from six weeks to six months to complete. DeLeeuw first meets with the film’s director and goes over the storyboards. These rough illustrations represent key scenes in the movie. DeLeeuw and the director work together to decide what effects can be practical and which ones will need to be digital. “If you stand on it or touch it, that’s something you should build practically,” DeLeeuw says. “And we’ll expand [the digital effects] out from there.”

Visual effects are created on a computer. Effects designers use a type of digital animation called computer-generated imagery, or CGI. The outer space backgrounds in Guardians of the Galaxy were made with CGI. So was Iron Man’s super suit.

CGI is also the key to transforming actors from regular people into incredible characters like the Hulk and Thanos. To overlay CGI on an actor’s face or body, DeLeeuw typically uses motion capture. His team covers actors like Robert Downey Jr. in about 150 reflective dots and reflective arm and leg bands. After a scene is filmed, the visual effects team places the character’s CGI appearance over the actor using these reflective dots and bands as reference points. 

DeLeeuw’s team also builds an animated version of each actor, based on hundreds of photos of the actor from every angle. This is called a character rig, and it can be manipulated like a digital puppet. Each rig has a skeleton, which is programmed to make sure the digital double moves realistically. Muscle texture and the actor’s appearance are layered over the skeleton. DeLeeuw’s team even needs to create a digital version of the actor’s outfit, complete with realistic fabric textures that ripple in the simulated wind.

CGI also helps transform actors into incredible characters like the Hulk and Thanos. DeLeeuw overlays CGI onto an actor’s real face or body for many scenes. To do this, DeLeeuw typically uses a technique called motion capture. His team covers an actor in about 150 reflective dots. The actor also wears reflective arm and leg bands. After a scene is filmed, the visual effects team uses CGI to alter the actor’s appearance. The reflective dots and bands act as reference points to make sure the CGI layer moves exactly how the actor did.

DeLeeuw’s team also builds an animated version of each actor. This digital double is called a character rig. It’s based on hundreds of photos of the actor from every angle. Once it’s completed, effects designers can move it like a digital puppet.

Each rig has a skeleton, which is programmed to make sure the character moves realistically. Effects designers add muscle texture and the real actor’s appearance over the skeleton. DeLeeuw’s team even creates a digital version of the actor’s outfit. They add details like realistic fabric that ripples in the wind. 

One of DeLeeuw’s biggest challenges was creating Thanos, the 8-foot-tall alien villain from Avengers: Infinity War and Avengers: Endgame. Making Thanos was so challenging that DeLeeuw’s team had to invent an entirely new way to develop visual effects! It was important that Thanos look alien but still be able to express a wide range of emotions. Using traditional motion capture, visual effects designers would have had to spend hours carefully animating each twitch of his mouth or making each pore on his face.

“What’s cool about Thanos is that we used machine learning software to recognize Josh Brolin’s performance.” says DeLeeuw. DeLeeuw’s team trained a computer program to recognize actor Brolin’s face as he made all sorts of expressions. The computer would then automatically map the model of Thanos’s face onto Brolin’s expressions, while keeping fine details like the pores in Brolin’s skin.

“Now we have this giant purple alien that shouldn’t look real, but because you’re adding in all these nuances, you create a believable character,” DeLeeuw says.

Thanos, the villain from Avengers: Infinity War and Avengers: Endgame, is an 8-foot-tall alien. He’s played by actor Josh Brolin. The effects for the character were one of DeLeeuw’s biggest challenges. Thanos needed to look alien but still be able to express a wide range of emotions. This would have taken an incredibly long time using traditional motion capture. Visual effects designers would have had to spend hours carefully animating each tiny detail on Thanos’s face.

Instead, DeLeeuw’s team invented an entirely new way to develop visual effects. “We used machine learning software to recognize Josh Brolin’s performance,” says DeLeeuw. That means his team trained a computer program to track changes in Brolin’s face. They recorded him as he made all sorts of expressions. The computer then automatically layered Thanos’s CGI face over Brolin’s. But it kept fine details from Brolin’s face, like the pores in his skin. That helped the actor’s performance shine through the character’s strange appearance.

“Now we have this giant purple alien that shouldn’t look real,” says DeLeeuw. “But because you’re adding in all these nuances, you create a believable character.” 

From the moment DeLeeuw saw some of the first movies that contained high-tech visual effects, like Star Wars and Indiana Jones and the Raiders of the Lost Ark, he fell in love with making movies. A huge part of that love came from those movies’ visual effects. “For me, it’s a kind of magic,” he says. “You’re making something that doesn’t exist and affecting people and their emotions.”

Now, DeLeeuw hopes his work with Marvel inspires the next generation of visual effects designers, who will have even more powerful technology to make even more mind-blowing effects. “You always hope you get a chance to work on a film that affects others the way you were affected as a kid,” DeLeeuw says. “That’s one of the best parts of working with Marvel.”

DeLeeuw grew up seeing movies like Star Wars and Indiana Jones and the Raiders of the Lost Ark. These films used some of the first high-tech visual effects. When DeLeeuw saw them, he fell in love with making movies. “For me, it’s a kind of magic,” he says. “You’re making something that doesn’t exist and affecting people and their emotions.”

DeLeeuw hopes his work with Marvel inspires the next generation of visual effects designers. In the future, they’ll have even more powerful technology to make even more mind-blowing effects. “You always hope you get a chance to work on a film that affects others the way you were affected as a kid,” says DeLeeuw. “That’s one of the best parts of working with Marvel.”