So a surgeon and a game developer walk into a bar...
Osso VR is a low-cost, fully immersive surgical simulation platform with advanced hand tracking. We are compatible with the Oculus Rift with Touch controllers and the HTC Vive and are developing in Unity. While we are in the medical education space, our roots are grounded firmly in the world of gaming. I want to share with you how we started out on this journey and how, thanks to our team’s experience in gaming and psychology, we were able to overcome some of the most difficult VR interaction design challenges.
I’m a surgeon, but I’ve been a gamer as long as I can remember. One of my earliest addictions was a medically themed game called Life & Death. You played as a local surgeon and were tasked with diagnosing patients correctly using their histories, physical exams, and various labs and imaging studies. Following diagnosis, you had to decide upon the correct treatment, which sometimes involved performing a tremendously challenging procedure. For its time, the game offered amazing freedom to experiment, although you were reprimanded with a slap on the wrist after any malpractice-worthy actions. I played the game because it was so fun, but I found that I learned a lot in the process. In fact, Life & Death was part of my initial inspiration for ever even thinking of pursuing medicine as a career.
Fast forward a few years to my orthopaedic surgery training. During the hype and excitement of the recent VR renaissance I saw a video of Surgeon Simulator 2013 using the Rift DK1 and the Razer Hydra for hand tracking. Once I got over the hilarity of the game, a horrifying realization came over me. This game in which you are asked to perform a brain transplant in a moving ambulance with a hammer, saw and various other hardware store bargain implements was lightyears ahead of the training options currently available
While there is a thriving $1 billion market for surgical simulators, they are in general limited to specific types of surgery such as laparoscopy. These simulators are also expensive costing up to $250k. Currently options for practicing open orthopaedic surgery are typically limited to the rare and expensive cadaver labs. The result is that surgical trainees like me didn’t really have easy access to a way to repetitively rehearse procedures, and ultimately ended up practicing on people one way or another. Here we are looking at a technology that can actually simulate really any procedure for a fraction of the cost. I felt a calling to take my passion for video games and combine it with my medical knowledge to create a low-cost simulation platform, and thus Osso VR was born.
Initially, I started out on my own. I found a pair of Razer Hydra controllers off of Ebay, and received my Oculus DK1 after months of waiting. After looking at various engine options Unity was an obvious choice. It was compatible with both the DK1 and the Hydra and I needed to get something up and running quickly given my time limitations. I set out to create a proof of concept demo in which the user runs through a portion of a surgery to treat tibia fractures with a titanium nail. The experience was humbling to say the least.
A few months later I had the great fortune to meet veteran game developer Matthew Newport, who is now Lead Developer of Osso VR. Matt also has a degree in psychology, which I soon found out is tremendously useful for VR design. We started development on the product in earnest and once again hand-tracking was the primary challenge. Through trial and error and extensive user testing, our thinking on hand tracking has evolved significantly. We started our development using the Hydra controllers, but thanks to Oculus were able to obtain and develop on Touch Controllers.
Our very first task was to decide how to represent hands in VR. Two popular options are having human hand models and animations like in Oculus’s Toybox Demo, or models of the controller such as in Tilt Brush. We ultimately decided upon having human hands as this felt more appropriate for a medical simulator.
Our next challenge was to decide on how to deal with picking up objects. Once again we saw two options. In one, objects lerp to a predetermined position in the hand when “picked up.” While this looks more natural on VR video captures, we saw several drawbacks to this method. Due to the nature of this method a different grip needs to be animated for each object, which would impose significant development time and also limit the flexibility to add additional objects down the road. In addition, this method makes it difficult to “adjust your grip” on an object to grab it from different orientations. Finally it also makes it less natural to transfer an object from one hand to the other.
Another method we discussed was having the hands “disappear” when an object is picked up such as in Job Simulator. This provides several advantages. It saves significant development time, you can adjust your grip, and you can easily transfer objects from one hand to the other. It also makes it easier to examine objects up close since your hands won’t obscure your vision. One downside we have seen with this method is that while it feels natural in VR, it can look off-putting when showing a VR capture to an audience unfamiliar with this method.
At this point, we had just been accepted to the 2016 GPU Tech Conference Emerging Company Summit VR Showcase. We had about a week before the conference when we made a pretty significant discovery. When demoing Osso to the general population almost universally, and especially with first-time users, the first thing someone would attempt to do would be to try and throw an object, which was not a feature of our demo. Initially, implementing throwing would seem to be a waste in a medical simulator as typically it is frowned upon to throw objects in the operating room. However, thanks to Matt’s psychology background, we were able to recognize that this response appears to be a natural part of the brain adjusting to a new simulated universe. When users were not able to throw something, they reported decreased presence, disappointment and frustration. I thought back to the game Life & Death, where having freedom to experiment and play led to a valuable educational experience. We both agreed that we had to implement throwing, and fast.
Matt was able to implement the most impressive throwing solution I’ve seen to date, and it helped propel us to win an award at the GTC Emerging Company Summit and set us off on a path of continued growth and excitement. In fact, we were pitched to write this very article thanks to a tweet of Matt flipping a hammer.
The use-case of VR for surgical training is obvious, however its implementation is nontrivial. What makes Osso VR unique is our ability to execute a variety of pioneering hand-based interactions through careful design, polished implementation, and most importantly deep understanding of the brain’s expectations and reactions to VR. Through our experience in gaming, medicine, and simulation we are paving the way to provide high quality, low-cost surgical training for the world to improve healthcare for everyone.