Worlds are Colliding!
Published 4 years ago
Merging real and virtual environments to make all the world a game.
Imagine you are at a nearby park, but instead of throwing a frisbee or tossing a ball around the field, you start an app on your phone that creates a virtual world within the space of that field, complete with virtual characters to meet and obstacles to avoid.  The screen of your device is a window into this world, and you hold it before you, turning to see the surrounding real and virtual landscape.  Take a few steps, and you move forward in both worlds, walking past virtual landmarks as though they were actually part of the real world but visible only through the window of your device.  This kind of experience is possible today with the apps we create at Virtex Apps.
To merge real and virtual worlds in this manner, two key puzzles need to be solved:
  1. The virtual world must be anchored within a real world space
  2. The user's movements must be tracked within that space
Existing augmented reality approaches to this problem include recognizing an image or landmark using the camera or identifying a specific location based on GPS.  However these options limit the range of motion and locations available to a user.  Our goal was to make this virtual experience available in any open space, anywhere in the world, without requiring hardware or headsets other than a standard phone or tablet, common devices already owned by millions.
After exploring a few options, we came up with practical solutions and put them to the test in our first app, Chicken Charge, a goofy game that is like being inside a life-size cartoon.  In the game you walk around to catch chickens and return them to their coop, but you need to watch out for rampaging buffalo, giant robots, and pianos that fall from the sky!  The game is purposely silly, but it also serves as a proof of concept, demonstrating the following ways to solve the key challenges of merging the real with the virtual.
The first hurdle was to somehow anchor the virtual world in a real space.  We wanted Chicken Charge to be playable in any open area of arbitrary size, but the app needs to know a few details about that area in order to render the virtual world.  Therefore, the game needs to get some information from the user before starting the game.  To be specific, it needs to know:
  1. The size of the real world open space in which to create the virtual world
  2. The starting location of the user within that open space
  3. The orientation of the virtual world within the real world
In Chicken Charge, this information is obtained by walking the user through a setup wizard in three simple steps.  In the first step, the user tells the app how much open space is available in the real world.  To keep this simple, the user selects from a variety of recommended spaces ranging from a small room to a basketball court.  Once the user chooses, the app knows how big the virtual world should be.
Next, the user is asked for their location within that space.  To again keep things simple, the user is given only four options: the four corners of the playing field.  After making a choice, the user is prompted to walk to that corner.
The last step is to determine the orientation of the virtual world.  To do this, the real-time image from the camera is shown onscreen and the boundary of the virtual world is overlaid on top.  The user is instructed to turn their body until the boundary of the virtual world matches the boundary of the real world, thereby aligning the orientation of the real and virtual environments.
After the user has completed this three-step wizard, the app knows the size of the space, the user's location within that space, and the orientation of the space relative to the real world...everything it needs to know to anchor the virtual world within the real world.  Now let the games begin!
Once the game starts, the user is surrounded by the virtual world.  Like many augmented reality apps, you can see the virtual environment around you, superimposed upon the image of the real world from the camera, but we wanted to take it one step further...literally.
Our goal was to enable the user to not only see and hear the virtual world, but to also move around within that world, as though the virtual world were truly there.  So our next challenge was to track the user's movements in the real world and translate them to movement in the virtual environment.  This creates the illusion that the virtual area is part of the real world, and the user moves synchronously through both.
GPS is commonly used to determine the position of a mobile device; however, a GPS reading on standard devices is not precise enough to recognize small changes at the scale of a single step.  Accelerometers can also recognize movement, but they are too noisy to accurately track velocity and positional changes over time.  So a new approach was needed.
We ultimately arrived at an ensemble solution that combines the data from multiple sensors to track changes in position.  It works a bit like an enhanced pedometer, applying the technique of dead reckoning to determine where a user has traveled.  With this approach, the app can detect relatively small changes in a user's position, and adjust the virtual world accordingly to create an immersive illusion.
Like the steps for setting up the virtual world, this approach for tracking movement is completely independent of the virtual world itself and its inhabitants.  No matter what appears in the virtual environment, the same methods for setting it up and for tracking movement can be used.  Therefore, we combined the world anchoring and user tracking capabilities into a patent-pending Unity-based framework which can be plugged into any Unity project.  We then used the framework ourselves to build the proof-of-concept game Chicken Charge to show what is possible.
With a few changes to incorporate the framework, any 3D environment created in Unity can become this kind of immersive mobile virtual long as it involves chickens (just kidding!).  We seek to explore novel ways of merging virtual worlds with the real world, through innovative games or other ways of engaging users, and we are looking for partners who are excited by the same opportunities.  If you have built your own 3D world in Unity and would like to walk into that world, please contact us to make these active virtual experiences a reality.
Jeff Green