The smarts behind Smart Hulk

We’ve seen Hulk in a lot of Marvel films, but never like this. Avengers: Endgame features the fully computer-generated Bruce Banner (Mark Ruffalo) character in a form where Banner has merged his intelligence with the Hulk’s, well, Hulkish-ness. That required a whole new approach on the part of visual effects, which had to capture Ruffalo and then bring a performance to life that was somewhat closer to the human actor than they had done previously.

Tasked with the majority of ‘Smart Hulk shots’ was ILM, which also built the asset. Framestore also contributed Hulk shots to the film. Both studios worked under production visual effects supervisor Dan DeLeeuw. Here, along with DeLeeuw, ILM and Framestore talk to befores & afters about starting with a Smart Hulk test, how refined on-set capture methods and facial capture – including the new ‘Anyma’ system – were utilized this time around, how Ruffalo performed on set, and what else made the CG Hulk a fresh challenge.

The Smart Hulk test

To prove that this new-look Hulk could work in scenes in the film, ILM produced a test. It began with a new Hulk sculpt drawing off of Marvel artwork, and incorporating clear Mark Ruffalo features. Endgame producer Kevin Feige pulled some footage from an interview that he saw online of Ruffalo after the movie Spotlight, in which the actor appears, came out. “That’s a pretty heavy movie and it was sort of a matter of fact, it was Mark talking about that movie,” notes ILM visual effects supervisor Russell Earl.

“I think that seeing Mark just speaking in his normal very sincere, earnest tone was great,” adds ILM animation supervisor Kevin Martel. “We wanted to see what that looked like coming out of what had previously been a more ‘grunty’ creature that didn’t really speak.”

Framestore’s Smart Hulk. © Marvel 2019.

“So we did that test and then we sent it down to the client and they saw it and I think that really gave them the confidence to go forward,” says Earl. “They showed it to Mark and what it showed, I think, is that he didn’t have to try to accentuate or over-perform. He knew that we could get at the character without him having to act more ‘Hulk-ish’.”

Capturing Ruffalo

On location, Ruffalo generally wore a motion capture suit and a helmet-mounted camera system. The motion capture was handled by Profile Studios and the HMC was from Fox VFX Lab (formerly Technoprops). “The first priority was letting Mark be in a mocap outfit acting against the other actors,” outlines Dan DeLeeuw. “You get a much better performance because everybody’s together.”

Original plate featuring Mark Ruffalo in motion capture suit. © Marvel 2019.The final shot completed by Framestore. © Marvel 2019.

“When you have someone like Mark Ruffalo,” notes DeLeeuw, “they will bring something to the character that you won’t always find. And in the interactivity between other actors, you’re going to find something you wouldn’t know if you’d just used stand-ins.”

Things did move between recording full motion capture data and more ‘faux capture’, depending if the shoot was in a contained studio environment or outdoors. “When you’re outside, generally it becomes more of a faux-cap thing because there’s a couple instances where we took camera’s out, but in Atlanta, the weather changes on a dime,” relates DeLeeuw. “So then your towers are swinging in the wind, then your day is not that great.”

For Ruffalo’s facial performance, he would regularly wear dots on his face as part of each scene. The actor had gone through various scans, including a Disney Research Zurich ‘Medusa’ scan which allowed the studios to build a per-frame mesh to base their model and animation on. The idea was all about translate Ruffalo’s performance onto the CG Hulk as convincingly as possible.

Original plate for a scene in which Hulk and others venture to New Asgard to talk to Thor. © Marvel 2019.ILM’s CG Hulk. © Marvel 2019.