9 September

Where are we with full-body effects in SocialAR?

SocialAR allows you to easily turn your face into an animal head or a demon mask. But when it comes to your body, things get complicated. Let’s see how close we are to realistic full-body avatars in AR.

 

Snapchat lenses and Instagram filters are easily the most popular application of AR in the world. The 600,000 lenses created using Snapchat’s Lens Studio have been used 15 billion times in just two years. Effects are becoming ever more complex and realistic – just look at these: 

And yet, there’s still one major limitation to SocialAR effects: almost all of them are applied to the face. Even those designed for your pet only work on their face:


Source: The Smart Local

 

You see almost no filters for the full body. Why is that?

 

The problem of body tracking

The idea of tracking a person’s whole body comes from VR, especially from gaming. When you are inside a virtual world, you want to see yourself interacting with it as your character. If you’re, say, an orc, the orc’s legs and arms should move as you move your own: 

Source: Yost Engineering Inc.

 

This is achieved by wearing a set of sensors, known as outside-in tracking. To play games, you need at least four of these, and each costs about $100

But let’s return to AR content for smartphones. In this case, the user doesn’t wear any trackers: you have to rely on the phone’s own hardware. This is known as inside-out tracking. The hardware in question consists of the phone’s camera, accelerometer, and gyroscope. 

Source: Coding Blocks

 

With just these sensors at your disposal, a body or a hand is much harder to track than a face. Just think of all the different possible positions and all the body parts moving independently from one another. Also, clothes or objects can obscure part of the body, complicating the task further.

Another problem is that a phone has a less powerful processor and much less memory than stationary VR setups. So, any body-tracking tech you might come up with has to be very light in order to run on the phone in real time.

 

Facebook pioneers digital avatars

When it comes to effects, Facebook usually lags behind Snapchat. But Facebook was actually the first to showcase full-body tracking technology back in 2018: 

Source: research.fb.com

 

In February 2020, Facebook finally previewed its hand- and body-tracking tools for Spark AR at its F8 developers conference: 

Source: Facebook

However, the new tools aren’t available to creators yet. Meanwhile, Snapchat got ahead of Facebook (again!) and released its own body effects.

 

Going skeletal

In August 2019, Snapchat introduced the Skeletal template. It tracks points on a user’s hands, elbows, shoulders, neck, and hands – essentially the whole upper body. Creators can choose any images they want to attach to the points. For example, you can become a cute robot: 

The template works best when the user stands away from the camera. With 8 points, it’s already something, but it’s still far from a full-body effect.

 

Full-body triggers with Snapchat: a step forward

Full-body tracking with Snapchat finally became possible in August 2020 with the introduction of two new templates: full-body triggers and full-body attachments. Both track up to 18 points, including 5 on the face. But the ways they’re used are a bit different:

 

– Triggers: a certain movement triggers an effect – an object appears, or there’s a burst of particles, or a short animation is played. Example: when the user raises their hands, shining balls materialize:

– Attachments: an object is attached to a tracking point (joint) and follows it. Note that the object even occludes the body – it seems natural, yet occlusion is actually hard to achieve in AR, so kudos to Snapchat for that.

For these templates to work, the user needs to stand quite far away from the camera. The most obvious application of these full-body templates is TikTok-style dance videos. But having little stars follow you around as you dance isn’t quite the same as a full digital avatar. So, where do we go from here?

The long road to realistic body AR

At the CVPR computer vision conference in June, Facebook showed a new algorithm that can recreate a human body in 3D using just one camera:

Source: facebook.com

 

This is the next step toward photorealistic avatars. But Facebook devs themselves admit that the technology is still years away from release. By that time, smartphones will surely have evolved: they’ll have more memory and more powerful processors. 

It will take improved hardware, better computer vision, and advanced motion-tracking to produce high-quality full-body effects, but just think of the possibilities! You could turn yourself into a Jedi, an alien, or a tiger – and then film a little dance on Snapchat. 

Sure, it’s a bit strange that the latest advancements in digital tech are instantly turned into entertainment for teens. But that’s how progress works today. And if anyone can deliver realistic and affordable augmented reality avatars, it will be either Facebook or Snapchat. ARzilla will follow their rivalry and report on the results!

Meanwhile, if you’re thinking about promoting your product with an AR filter, you’ve come to the right place. Drop us a message at elay@arzilla.io, and we’d be happy to share our ideas – and, of course, design a perfect filter for you.

Other articles

22 October Collaborative augmented reality: discover the hottest AR trend

From multiplayer AR games to business meetings with holographic participants, collaborative AR…

22 October Mobile AR revenue outlook: 5x growth by 2024

Mobile AR revenue outlook: 5x growth by 2024 A recent report claims…

2 September Computer vision & AR: teaching a machine to see

AR adds virtual objects to the real-world environment. But how does the…