And here are some learnings from the project:

Haptic feedback

To provide real-time feedback for a motion, we need to communicate two things: a rotation error and a distance error.

Figuring out how to communicate both was the most difficult part of the design process, since we can usually only control the vibration’s strength and interval in most devices with haptic feedback (like the Quest 3 controllers I was using). That’s not a lot to work with.

If we make a rotation error cause stronger vibrations and a distance error cause weaker ones, users don’t really feel like the signals are different enough to be from different sources. It just feels a weaker versus stronger “general” error.

So the best solution ended up being to separate the two errors by having one cause constant vibration, while the other vibrated in pulses.

For whatever reason, pulses just feel tangibly different from constant signals (which might be why Morse code uses this separation too).

I also tried increasing the strength of vibrations to communicate the size of the error: a bigger rotation error would lead to a stronger constant pulse and so forth.

But this made it harder to quickly tell what was happening, since there were now 4 things to track, instead of 2 (stronger rotation error, weaker rotation error, stronger position error, weaker position error).

Small differences in vibration strength are also really hard to notice, so I would’ve had to use some sort of logarithmic scaling or else it’s very hard to know which way you should rotate when you’re very close to the final target.

Regardless, it turns out that keeping things discrete and simple was the best approach, especially when the feedback is immediate and so it can trigger the moment the user makes any error.

If it wasn’t immediate and instead delayed by, say, half a second, the user might’ve already accumulated a lot of error in both position and rotation. We’d need a lot of precision to communicate exactly how they need to get back.

But, since the feedback is instant, they’re stopped before they can accumulate much error at all. So they’ll intuitively know that it was exactly their last little motion that caused the error, which they can then easily roll back without the need for precise communication.

Lastly, I did also try variants where the haptic feedback was triggered while you met the conditions (as opposed to whenever you didn’t).

This worked alright, but it felt like it was easier to quickly react to a sudden vibration rather than the sudden loss of a vibration.

And constant vibration is also just distracting (since constantly saying “you’re doing things right” is a lot of unnecessary communication) and quickly drains battery life from haptic motors.

All in all, haptic feedback worked well, although it could be made even better if we had more knobs to tune, like the direction of a vibration pattern across a device.

If we had that, we could, say, trigger a vibration pattern to the right to precisely communicate that the user needs to rotate their hand to the right (although this might also just make it too hard to understand what means what).

Audio feedback

Audio feedback worked in much the same way that vibration did: it was easier to understand that the two signals (rotation error and distance error) were different when one was constant and another was pulsed. The separation wasn’t as clear when using pitch and volume (or a combination of both).

The big problem with audio feedback, though, is that you’re not directly getting feedback where you’re controlling the action (i.e., your hands).

So there’s extra processing time involved with mapping the sound you’re hearing to the error you’re making, which is a problem when we’re trying to communicate an error to you before you have time to make even more errors.

Then there’s an even bigger problem: the user’s “audio channel” is much more contested than their “haptic channel”.

In other words, while playing sports, you’re often talking to people and hearing all sorts of noises. If we give the user feedback with a beeping sound, it can disrupt conversations and get drowned out by music or wind or whatever else. A vibration isn’t really competing against much (at least until the moment of collision with a ball).

So, in a nutshell, haptic feedback is just way better for this problem and audio feedback is only in my video because it’s hard to communicate vibration otherwise.

Visual feedback

It turns out that it’s phenomenally useful to visually show the ideal path in advance, since the user will know exactly where to move, even if they’ve never seen the motion before.

With haptic feedback alone, completely inexperienced users would have to do a lot of slow swings to trace out the general path first.

But, until smart glasses come out, we can’t really rely on users having access to any form of augmented reality, so it’s not really an option.

For curiosity’s sake, I also tried visually showing rotation and distance offsets with lines around the user’s hand (see video below).

It didn’t really work well, mostly because it’s almost impossible to track changes in the lengths of lines on fast-moving objects and it takes a lot of extra time to process what some abstract lines are supposed to mean.

In sum, there’s no way you can react quick enough to correct swings in real-time with this type of visual feedback.

And, even if this weren’t a problem, the fact that you have to constantly look at your hands already makes this a non-starter for most applications.

In any sport, you’ll want to look at the ball or your target 90% of the time, which means you won’t be able to see any kind of visual feedback on your hands anyways.

Even if we moved the visual feedback somewhere where you can see it, this would just make the processing challenge even harder (because now the visual feedback isn’t even near your hands).

And it’ll also compete with other visual information that’s much more important, like the ball or the target that you’re trying to hit.