Hand Gesture Arm Fatigue, Part I

A Non-Starter for Spatial Computing Apps?

TexasGreenTea

6 min readAug 23, 2021

Welcome to a three-part series on spatial computing ergonomics.

Part I: Is Arm Fatigue a Non-Starter for Spatial Computing? (you are here)

Part II: Why Drumming Tells Us Everything We Need to Know

Part III: Anatomy and Best Practices

An old wives’ tale has been shared hundreds of times throughout the spatial computing industry. It goes like this…

“On the set of Minority Report, Tom Cruise needed riggers to tie his wrists up to the scaffolding (marionette-style) because his arms got so tired from shooting the opening scene that he couldn’t lift them on his own.”

I… have no strings to hold me down, to make me fret, to make me frown!

I have no idea if that tale is true, but as we watch Tom Cruise’s arm motion in that film, no one is surprised. If you mimic his actions for ten minutes, you’ll certainly want to take a break, and he had to shoot those gestures all day long.

It begs the question:

Is hand-tracking ergonomically feasible? Like… at all?

Or are all hand tracking interfaces destined to fail just because we will not want to walk around with sore shoulders all day?

Tom Cruise has had plenty of time to ice his shoulders since then, but after almost twenty years since Minority Report hit theaters, the jury is still out on hand tracking ergonomics. The precogs didn’t predict this one very well.

There is some precedent with actual hardware in the spatial computing industry. We’ve had a slew of devices hit the shelves in the past decade that have (or had) full five-finger hand tracking.

We have thousands of people (albeit mostly developers) who have tested these devices...

And they often report things like…

I had to take a break after about twenty minutes. My shoulders were killing me.

These are not the results we were hoping for. Never happened.

So we’re not in great ergonomic shape yet. To be fair, things have improved. You don’t hear nearly as many complaints about Oculus Quest hand tracking, which has been used by many more hands than the others. We’ll see later why Quest is an ergonomic step up compared to its current competitors.

But as a spatial computing engineer who has implemented hand tracking apps for Leap Motion, Intel RealSense, Hololens, Magic Leap One, and Oculus Quest, I can tell you that most existing hardware is unfortunately quite unusable with regard to good hand ergonomics because it does cause your shoulders to fatigue in about ten minutes of use. Oculus Quest is better than previous generations, but it is not immune to the woes of shoulder fatigue.

The theme is clear. Most of these devices in their current form are ergonomic non-starters. Is this just an unsolvable problem? Is it proof positive that no one will ever use VR and AR headsets all day long at work?

You may have guessed I wouldn’t bother to be a spatial computing engineer if I believed that. I do have a good counter-argument, but it doesn’t come from spatial computing evidence, since our industry is brand-spanking new in a consumer sense.

What we need is correlation. The human body is the human body, no matter which field you study. Where shall we look?

Bruce Lee once said:

“A punch is just a punch.”

He was arguing against martial artists who claimed their style’s punching technique was superior. In his embryonic MMA philosophy, he subscribed to a sense of anatomical pragmatism. He argued that there is such a thing as good punching technique, but that the technique is obviously universal across all styles.

Anatomical first principles were Bruce Lee’s M.O. If a technique is feasible in any style or field, it can be done the same way in all styles and fields.

This means that if the nature of hand-tracked gestures in spatial computing is found to be nearly identical to that of a certain other field whose ergonomics are well-established, we can import the ergonomics from that field. We just need to identify the right corollary.

If I ask you to think of a field of study in which we actively use our hands rapidly for hours at a time, what’s the first thing that comes to your mind?

Did you picture Bruce Lee doing kung fu?

My fault. Sorry. I teed that up for you. Kung fu is fun to compare to spatial computing. Martial arts sport extensive hand activity, but most of it is high tension, high speed, and high impact. Most sports… similar story. These are not the best corollaries for hand tracking UI, since we need high speed sometimes, but we often need low tension and low/no impact.

Try brainstorming another example that has large amounts of hand activity.

…

Did you guess music? Me too!

I believe music is the best corollary for spatial computing ergonomics, but I’m biased. In a past life, I got a music degree at UCLA before ditching the low-paying music industry to become an XR coder. As a music composition student, I had to learn several instruments, including piano, guitar, clarinet, and my main instrument, drums and percussion. I also had to study to be a conductor.

With this background, I’ve been aware from the first day I started coding VR apps that musical instruments require actions from your hands and arms that are strikingly similar to hand tracking techniques in spatial computing UIs.

In Minority Report, the production designer intended for the choreography of the interface to resemble musicianship. That’s why classical music was playing during several scenes:

They wanted us to associate these UI gestures with the high level of skill required to perform music.

Interesting. A musician can play for hours at a time without significant fatigue, yet Tom Cruise allegedly could not. What are we missing?

To figure it out, we’ll deep-dive into the techniques of the musical instrument that is perhaps most similar to hand-tracking techniques:

In Part II of this series, we’ll discover why we’ve secretly known about hand gesture best practices for decades. To learn them, we’ll just need to take a quick drum lesson.

Follow me on Twitter for the link to Part II when it goes live tomorrow.