fbpx
Menu Close
1. December 2021. CREATIVE

MOCAP: PERFORMING ART FOR THE DIGITAL WORLD

You think you need the famous motion capture, but do not know where to start? Look no further and let us explain what MoCap is and how it works.

Motion capture (aka MoCap) is the process of recording motion (of people, animals, things) and converting it into a digital form. This kind of information is later used to bring digital character models to life in 2D or 3D computer animation.

Although Mocap is primarily used in the entertainment industry such as filmmaking and video game development, it can also be found in the military, sports, medicine, and increasingly in robotics and computing.

When we add face and finger MoCap, the process is called performance capture. An example of such a process is the filming of Josh Brolin as Thanos in The Avengers.

https://www.insider.com/avengers-infinity-war-thanos-without-special-effects-photos-2018-7

History

The first commercial use of motion recording, MoCap, occurred in the early 1940s when the first animated film, Superman, was made. The filmmakers wanted to give the animated characters real human movement. This is how the first form of rotoscoping came to life. It was a machine that projected a video onto a transparent drawing surface. The actors performed the movements and the animators used them for tracing.

https://www.rotoscopers.com/2015/08/05/forgotten-gems-of-animation-fleischer-studioss-superman/
https://www.skwigly.co.uk/the-rotoscope-of-max-fleischer/

The game Prince of Persia (1989) used the same rotoscope technique for its movement mechanics and was very successful. In this case, the creator filmed himself performing the movements he needed for the prince, and animated them within the game code. The prince ran, jumped, walked, and fought with real movements.

https://www.forbes.com/sites/sethporges/2017/12/19/how-the-original-prince-of-persia-changed-video-game-animation/?sh=426b1ac83f6d

But if we talk about modern motion capture technology, we can say that Jar-Jar Binks in 1999 was one of the first digital characters to use performance capture, as was Gollum from LOTR 2000. These films were the first to use technology based on optical cameras and markers for digital transformation. The actors wore suits that contained markers needed to track the characters.

https://beforesandafters.com/2019/05/24/jar-jar-binks-was-nearly-a-practical-suit-and-a-cg-head/

Technology

Rotoscoping is just one way of transferring “real” world motion into the digital. However, the process is time consuming and requires a lot of people doing in a short period of time. Also, this technique delivers a visual style that may not be everyone’s cup of tea.

Therefore, there is a modern motion tracing technology that can be divided into optical and deterministic MoCap systems.

Optical systems use a variety of cameras placed around the room where the actor’s performance is recorded, as well as markers placed on each part of the actor’s body. The cameras read the position of the markers and translate them into a wireframe within the program. Once it’s recorded, the data is transferred into 3D. But such a session is anything but simple and requires a team of people such as the director, actors, film technicians, animators, and producers, to manage the process from start to finish.

This technology provides the most realistic copy of human movement, but is often expensive and only used in large production companies for media content such as AAA games and Hollywood movies.

https://theconversation.com/why-motion-capture-performances-deserve-an-oscar-67138

For this reason, more affordable deterministic systems have been developed to further expand motion digitization technology and make it accessible to more users.

This technology uses motion sensitive inertial sensors that detect the movements and display the wireframe data in real time. These sensors are attached to each part of the actor’s body and fastened with belts or a spandex suit, so that they don’t move from their position once attached.

The lack of this technology is that it does not show the exact position of the 3D model because it assumes where the body is, which depends on the quality of the data from the sensor. But again, the quality depends on the price of the system – cheaper systems lead to more data cleaning and adjustment, while more expensive systems show the movement very accurately, almost like optical ones.

The MoCap Process for the Friendly Ghost of Biokovo museum multimedia project: working with an Actor

The “How to”

Although it sounds quite simple, the process of motion capture animation consists of several steps that require good planning and execution.

First, you need to script and storyboard the characters of your project so that you can direct them properly. It is possible to play any character, but the actor has to pay attention to the appearance of the character to deliver the physics of movement as credible as possible. For example, a giant will not move as fast as some other, smaller character – the actor will have to slow down his movements and ground his body and vice versa.

But what if the actors move well, but their voice does not fit the character? The advantages of MoCap recordings lie in the possibility of recording the movement and voice separately.
In that case, we can use three people for one character – one for the voice, one for the movement, and the last one for the looks on which the 3D character is based. And that’s exactly what the development team of Bioshock Infinite did for the character called Elizabeth.

https://www.polygon.com/2014/3/21/5530892/animating-bioshock-infinite-elizabeth-to-foster-emotional-connections

Actors are selected via a casting process that is coordinated by a casting agent and producer that select those who would portray each segment of our MoCap project. Once the actors are selected, extensive rehearsals with a director and stunt coordinator are required. Each stunt is well planned and it has to be as safe as possible for the actor, stuntman and the crew.

Mocap acting is a combination of theater and film acting. In theater acting, the movements are wide, large, sometimes excessive, but necessary to entertain the actor or spectator, and in film acting, the camera requires the focus and the magic of the solo acting performance.

The MoCap recording room should be equipped with appropriate technology, whether optical or deterministic, where technicians ensure that everything goes well during the recording and technical difficulties can be eliminated.

Props are an important part of the scene. A prop has to be made for every object that is touched or handled. In order to recreate the layout of the digital scene, it is necessary to make the skeletons of the scenography, so that the actors know what type of scene they are playing in. For example, if an actor has to walk through a door in the 3D scene itself, a movable wooden frame is built for the MoCap shoot to serve as a substitute for the digital door.

https://www.ilm.com/performance-capture/

Let us not forget the face MoCap.

Although it is filmed at the same time as the acting performance, using special cameras that are mounted on the actor’s head, the editing is done through a separate process. Since the face is complex, the challenge is to animate it in a 3D environment without making it look stiff and creepy. The actor has to wear a helmet with cameras recording his face and try to ignore them while acting, which is a big challenge. Also, dots are drawn on his face to detect facial movements, because every micro expression is essential for a high-quality 3D face.

https://develop3d.com/profiles/vicom-motion-capture-technology-body-moving/

After the actors have acted out the scenes and the MoCap data has been recorded, comes the most complex process of all, which is the data animation itself.

The first thing you need to do is bribe the animation team with a good expensive beer by lying to them that this is a simple project with very little movement. Once you have them on board, the animators take the MoCap data and “glue” it to the 3D models. They realize they have been tricked and that the amount of work is just as complex as making new Avengers, but they feel guilty about taking that expensive Guinness from you, and continue to work while crying.

The motion data is merged into a 3D model, “cleaned” up, and adjusted to avoid collisions due to the different body proportions of actors and 3D models, and it becomes movable. This process takes time, depending on the complexity and amount of data and movements.

https://www.awn.com/vfxworld/maya-entertainment-creation-suite-2011-review-killer-productivity

But let us pause for a moment.

You would say that by using more expensive technology, we have less “cleaning” in postproduction, right? Although logic would have it so, that’s not really true. Better technology provides better data, but there is not a perfect overlap with the 3D model. Although the processes that lead to this are still evolving, the animators still do most of the fitting work, “cleaning” up, and adjusting.

When the animation department is done, our 3D character should have a complete and flawless copy of the actor’s movements. We should also have a 3D environment, image composition, lighting, optimized sequence… but that is a complex story for another blog.

So, is MoCap fun from an actor’s point of view? Yes, but it’s also very complex to execute.

A responsible MoCap performer needs to understand not only the acting, but the movement and biomechanics of the creature he is bringing to life.

The MoCap Process for the film Faithless: working with an Actor

And from a programmer’s perspective? Well, it takes a lot of talented people to create the final MoCap product. Producers, directors, animators, 3D artists, and video artists are just a few of them, and the quality of a project depends on teamwork.

And in the end, the question remains, do you need MoCap?

If so, give us a call!

p.s. check out our services VFX – Visual Effects , Virtual and Augmented Reality, Game and Application Development, Video Production

What can we do for you?