Notifications
Article
Gesture controlling an interactive projection mapping
Published 2 years ago
543
1
From static to motion to interactive graphics
What did and do I do for a living?
I have been a freelance interface and motion designer in the area of Köln / Colgne Germany for more than 10 years now.
In the first years I also did some HTML, CSS, Actionscript coding until I realised it is more effective to concentrate on one field and chose to go the visual designer path. Although my small scale developer background helped me in almost all my projects from Flash Actionscript, to After Effects Expressions and Cinema 4D Xpresso to learning C# for this prototype.
Throughout the last years I have been doing a lot of automotive GUI and TV showpackage / motion graphics design from concept to production ready assets and in my sparetime took a closer look at GPU rendering with Octane for Cinema 4D.
 
Why this prototype?
Besides all the none interactive 2D / 3D rendering I have done I always wanted to do something interactive. Partially because I grew up with games I always wanted to build one myself. But also because motion graphics design includes a lot of tedious, time consuming testing, compositing and rendering, which is drastically reduced with realtime 3D graphics. So why not take a closer look at a game engine?
After my non interactive projection mapping test in 2013 on vimeo or on youtube which is basically a rendering I did in Cinema 4D projected on to a car, I wanted to do something interactive. So because I already did some tests with Unity 3D years ago it was the most logic choice.
 
What was the goal?
My main goal was to learn how to rotate a physical object with hand gestures and project a fitting animation onto that moving object. Broken down into the main questions I had to answer:
  1. How do I get Unity to recognize gestures?
  2. Luckily Leap Motion provides assets and an API by which you can get access to the properties of abstract models of the recognized hands.
  3. How do I code C#?
  4. No shortcuts here just lots of tutorial and forum reading, still a lot to learn ...
  5. How do I connect a stepper Motor to Unity?
  6. I used an USB cable to connect to the arduino uno board to which I added an adafruit motorshield which is connected to the Nema 23 stepper Motor and an additional power source.
How does it work?
The Leap Motion works like a camera which sends pictures to your computer which can then be interpreted by its Leap Motion Unity API and converted into an abstract hand model. With some C# I drive the scene according to the hand model, send the visuals to a beamer and also data to an arduino board which then drives a stepper motor.
Here is a behind the scenes Video showing more of the hard and software as well as the running prototype, which you can also view, like and share on vimeo https://vimeo.com/173940321
What´s the gameloop?
Because this personal project was more about learning how to drive hardware with C# the gameloop is minimal: Find all the pets by 1st rotating the cube with your left hand 2nd grab a pet with the right hand and 3rd drop it onto it´s corresponding pillow. You have won the game when all pets are on their pillow, but can also reset it by holding both hands in close proximity for three seconds. 
 
Unity store gallore!
I think that Unity´s big asset store and huge community are its biggest selling points, especially for starters and for building prototypes like this one. And because my main goal was to learn how to control a stepper motor with gestures I used a lot of assets from the unity store instead of building them myself.
What´s next?
Although there is a lot of noise about VR none of it is on my todo list for 2016 which is also why I sold my Occulus Rift DK2 in February. At max I might take a look at AR (Augmented Reality).
As of now my next personal project is about some more Octane renders, but after that I would like to take a shot at some mobile / tablet / multitouch prototyping for which I already have some game mechanics in mind I would like to tinker with.
Got any questions or recommendations?
Do you need an Interface / motion designer?
Get in contact with me here:
http://www.pixelasm.com
http://vimeo.com/pixelasm
https://www.youtube.com/user/ralfb1981
https://www.facebook.com/pixelasm
https://twitter.com/RalfBreninek
 
Credits
Thanks Ilya Suzdalnitski for his portal shader.
 
Photos from the setup
[[{"fid":"78921","view_mode":"teaser","fields":{"format":"teaser","field_file_image_alt_text[und][0][value]":"Test setup showing leap motion, arduino board and stepper motor driving the hyper cube","field_file_image_title_text[und][0][value]":"Test setup showing leap motion, arduino board and stepper motor driving the hyper cube","field_file_image_caption_text[und][0][value]":""},"type":"media","attributes":{"alt":"Test setup showing leap motion, arduino board and stepper motor driving the hyper cube","title":"Test setup showing leap motion, arduino board and stepper motor driving the hyper cube","height":"124","width":"220","style":"width: 220px; height: 124px; float: left;","class":"media-element file-teaser"}}]] [[{"fid":"78924","view_mode":"teaser","fields":{"format":"teaser","field_file_image_alt_text[und][0][value]":"Closeups of arduino board, stepper motor and timing belt","field_file_image_title_text[und][0][value]":"Closeups of arduino board, stepper motor and timing belt","field_file_image_caption_text[und][0][value]":""},"type":"media","attributes":{"alt":"Closeups of arduino board, stepper motor and timing belt","title":"Closeups of arduino board, stepper motor and timing belt","height":"124","width":"220","style":"width: 220px; height: 124px; float: left;","class":"media-element file-teaser"}}]]
[[{"fid":"78924","view_mode":"teaser","fields":{"format":"teaser","field_file_image_alt_text[und][0][value]":"Closeups of arduino board, stepper motor and timing belt","field_file_image_title_text[und][0][value]":"Closeups of arduino board, stepper motor and timing belt","field_file_image_caption_text[und][0][value]":""},"type":"media","attributes":{"alt":"Closeups of arduino board, stepper motor and timing belt","title":"Closeups of arduino board, stepper motor and timing belt","height":"124","width":"220","style":"width: 220px; height: 124px; float: left;","class":"media-element file-teaser"}}]]
Ralf Breninek
Interface and motion graphics designer - Designer
2
Comments
TK
Tomas Killington
10 months ago
Excellent projection! By the way, maybe you also want to take a look at video mapping https://videomapping.store/
0