Work / Independent Projects / Gesture Orchestrator
Using a hand gesture to trigger live video effects
A camera system that recognizes a performer’s gesture and automatically plays video effects around them during a live broadcast.
Live effects depended on controls outside the performance
A producer, button, or fixed cue normally had to trigger the visual moment.
Gesture recognition orchestrates layered effects inside OBS
The performer is segmented between synchronized front and back animation layers.
The performer controls the production cue naturally
A familiar gesture triggers a reusable effect without another operator or interface.
Live effects normally need someone else to trigger them
Video effects are usually started by a producer, a control panel, or a fixed schedule. I wanted the person on camera to trigger the effect themselves with an ordinary gesture.
The effect also had to appear behind and in front of the person
Recognizing a thumbs-up was only the beginning. To look convincing, part of the animation needed to appear behind the performer and part in front, all without hiding the person or interrupting the live video.
I taught the browser to recognize the person and build the effect around them
I built a web application that can be added directly to OBS, a common live-broadcast program. It watches the webcam feed, identifies the performer’s body and hands, and recognizes a thumbs-up.
Each animation is split into a back layer and a front layer. The system separates the person from their background, places them between those layers, and plays both pieces together. This makes the animation appear to surround the performer instead of simply covering the screen.
The performer became their own control button
The thumbs-up is both part of the performance and the command that starts the next moment. No separate screen or off-camera operator is required.
The system resets itself and is ready to go again
When an animation ends, the system clears it and waits for the next gesture. New effects can be added as matching front and back videos without rebuilding the gesture-recognition system.
The same idea can support many kinds of live experience
The same approach could be used for livestreams, installations, games, branded events, or any experience where a person’s movement should control what happens on screen.
A thumbs-up became the production cue
The performer makes one familiar gesture, and the system recognizes it, starts the animation, keeps the person visible, and sends the finished scene to the broadcast.
Keep exploring
See what else I have built.
Browse more systems, tools, and consulting engagements across different industries and kinds of problems.