Published a year ago
Challenges of photo-realism and photogrammetry
VRemedies is a Medical VR tool and content creation company. Our first product is a patient preparation VR application for kids. This is used to simulate non-invasive hospital procedures such as the MRI, CT, X Ray and Radio Therapy. Used in conjunction with Hospital processes, it can help hospital play therapists acurately assess reactions and desensitize the children to the procedures. 
The VRemedies tean were given special access to a children's hospital and were able to create photogrammetried environments of the medical imaging procedure rooms. All of the rooms have an identical look-alike in the real world. We created our own custom workflow to turn still images into low poly meshes that can be edited and manipulated in the Unity game engine.  
The story and gameplay was designed alongside hospital clinicians. Making sure every process was accurate to the real procedure. The characters are voiced by hospital play therapists and constant feedback was being provided by the hospital staff.  
In the experience the patient taken through a VR tutorial with one of our Robot characters. They are guided through the procedures, first being shown how they work – then secondly going through the procedures themselves. The children are rewarded with a balloon to pop at the end of every activity. 


We choose to develop with the Vive because of its ability to bring players into the space and physically move through the procedures. In the MRI and CT rooms, the players will lie down and move through the machines. We are also currently working on full arm tracking for any X Ray movements.  

Capturing the Rooms: 

Before we started capturing the hospital rooms we did a series of still photo tests in the office, working with processing in reality capture then converting the textures to low poly. We used these tests to create a checklist of techniques we can use to insure all the data is processed quickly and came up with the best results. Some of the biggest learnings we took away from these tests were, the distance between the object and the camera, the depth of field, lighting situation, background information, surface texture and colour. Any of these factors not captured correctly can make or break the process.  
We then had a window of about 5 hours per room to capture the hospital rooms. Keeping in mind which aspects of the room will be highlighted and interacted with in the game. There were some objects we kept out of the game, as they didn't serve a particular purpose and took time to develop. The entire process of creating the photo-realistic rooms was riddled with thoughts of 'is this important,' and relevant to the problem we set out to solve.   

Building the Rooms: 

When creating the low poly meshes, we also made sacrifices on which parts we would physically model and which parts can be represented in a texture map.  It all had to do with questions such as, 'will this be part of a key feature', 'how long will the player be in this space,' 'will this be noticed'.  See in the Image below, the red highlighted area is completely flat. The photogrammetry allowed us to create the illusion of these objects and save production time.  

Optimization in Unity: 

We used Unity's Area lights to really bring the scene to life. You can see here the difference between a scene created with the area lights in the same position as the lights in the captured room – as opposed to scene lighting with a skybox and point lights. It's important that the lights and shadows hit the objects in the same place it would in the captured space in real life. They should be complimenting the textures shadows instead of conflicting with them.  

Next steps: 

VRemedies is a VR application company that we will continue to develop VR tools for hospital use, we are constantly updating features and optimizing our products. Everything we learnt here we have applied to all of our other upcoming VR experiences, including our Lidar Scan photogrammetry work and other photo realistic projects. 
Krystal Thompson