Bringing a Van Gogh Painting to VR
Published 4 years ago
23.0 K
The process of building a painting in 3D
For The Night Cafe, I wanted the viewer to experience the feeling of stepping into this back alley cafe the way Van Gogh might have. I also wanted to express that feeling of wonder when you see his paintings, of being swept up by his exuberant colors where every brush stroke feels imbued with his lust for life as much as with his inner struggles.
I wanted to be fathful to his paintings but not completely beholden to them because I knew I would have to take some creative liberties in translating his colors and forms to a 3D canvas. As a result, I decided this would be a tribute experience rather than a direct adaptation.
Defining the layout
I knew I wanted a bigger space than what was shown in the Night Café painting so I designed a floor plan that added a separate set of rooms off to the back right of the main space. This separate room also gave me some freedom to add my own spin on the environment.
This was actually one of the biggest challenges because I had to match the overall tone and feeling of the Van Gogh painting along with finding colors and forms that compliment it — no small task considering Van Gogh is considered a master of color by many people. At the very least I would try my best to achieve some level of harmony in the colors. Even though the new space is completely invented, I drew on other Van Gogh paintings as well as other masters to inspire the design.
In addition to figuring out the overall floor plan, colors, and scope, there was also a lot of research to be done in making sure I was accurately representing the time period this was supposed to be set which was around the time the painting was made in 1888. I had to figure out things like: what would the outside of the window look like? What type of lamps were those hanging from the ceiling with the radiating paint strokes coming off of them? Some of these things seemed obvious at first but when it came to modeling I had to be sure I was finding the right reference images. In the case of this project, doing the research helped to give the environment more authenticity.
Building the environment
This was perhaps the most time consuming phase but one I had a lot of fun with. In order to achieve the painterly look, especially one as unique as Van Gogh’s, each asset would have to be carefully digitally hand crafted. Early on I had decided on a set of pillars regarding the visuals:
  1. Everything needs to look unique
  2. Objects, forms, and colors should blend into each other
  3. Objects should convey a mood
Modeling was generally a process of sketching the model on paper first. In order to understand what it is you are trying to make in 3 dimensions you need to imagine it from all sides. This is important because you can’t include a detail in the model if you hadn’t imagined it first and I find sketching on paper (or a Wacom) is the fastest way to figure out the details from all angles.
Once the sketch was done, I could then go into Maya and build the object using standard modeling techniques. I would often build a very straight and rigid version of the object first, almost like it would appear in reality (i.e. a chair with a straight back and 4 legs all the same shape and size). After I built the straight version, I could then take it into a sculpting program and warp the vertices around a bit, pushing and pulling different parts of the model to give it a more organic, painterly flow. Combining this with handpainted textures was the key to achieving the look I wanted.
Hand painted textures
For the most part I just painted in a very traditional way without using any fancy tricks to achieve the look. I used Mudbox because it allows you to paint directly to the bitmap rather than vertex painting. This meant I had a lot of surfaces to paint. The main difference between painting the objects and painting a canvas is the sheer amount of surface that needs to be covered. Instead of only painting the side that appears to the camera (or eye ball) you have to paint the back of it and the top of it, and every angle that could be seen by someone walking around it. Walls took the longest and were the most boring because I had to maintain that layered brush stroke look while just using the same 5 shades of a color to span a very large surface area.
Picking a shader
I had originally intended to give everything a Standard Shader within Unity that I would apply normal and specular maps to. I had already made several of the normal maps, mainly for the walls that I was able to test with. The specular highlights gave the look of the paint still being wet which was interesting but the overall look wasn’t as striking as I was hoping for. Normal maps up close don't tend to look great in VR. Parallax shaders produced very interesting results but caused all kind of issues at texture seams and ultimately would have been too heavy on the processor for the Gear VR.
On a whim I tried using the textures in an unlit shader and was surprised by the results. All the colors were vibrant the way I perceived them in the paintings and the shadow details looked more painterly as well because they were incorporated into the painted textures. It became apparent that this was the way to go. Not only did it look better but it also was the best for performance and I wouldn’t have to worry about eating up processing power with lighting.
That concludes this post about my attempts at achieving a painterly look in a VR experience. There are some other details I didn't mention here but are on my blog such as some early tests I did with particles instead of textures. Feel free to check that out at We are in the middle of development on our next game and will have blog posts going into that as well. 
carla bispo
2 years ago
Nice! Great Job! I loved so much!
Game Designer
What a genius idea! Great creativity guys. You have inspired me to try something similar. Hats off to you. Can we download this anywhere?
Akos S.
3 years ago
wow, great job, i'll try it for sure