How a Passion for Shaders Turned into a Music Video
Published 2 years ago
The Thin White Duke Travels Through Space Via An Infinite Wormhole
You are a benevolent multimedia creative health hazard producer. "Warning! This video contains flashing lights!" -- If this is your first foray into dangerous art I look forward to seeing your work evolve.
It started innocently. A midi controller hooked up to openFrameworks, feeding into an abstract shader - spawned by a couple of questions:
  • What happens when you try to manipulate a shader live, by hand?
  • Could it produce some interesting visuals?
As always, a steady stream of music was running in the background, and I found myself playing the bridge section of the same song over and over again, trying to match the visuals to the driving rhythm, intense guitar riff and alien synths. As I watched the recordings I'd taken of my spontaneous performance, it occurred to me that I'd basically made an impromptu music video. But it wasn't really anything. It could be better. I could see exactly what it wasn't, and exactly what it wanted to be.
And so it began.
Procedurally Animating Characters with a Vertex Shader
The abstract stuff was compelling, but it quickly became repetitive. It was clearly missing a narrative - something to make it more than just eye candy; a full story with unique characters and beautiful worlds. I turned to Unity, my other go to tool. Without any experience with 3D modeling, I decided instead to stick with what I know and am best at: shaders. The strange alien characters were born not in Blender and Maya, but in CG and GLSL, using only the simplest of models as a basis - the sphere primitive.
The video quickly came to incorporate more and more shaders: custom toon shaders for a unique style, a custom skybox that's done entirely with its own shader... even the well at the beginning of the video is done using the stencil buffer rather than an actual model of a well. One of the most important (and fun) shaders was the glitch effect on the main character.
(Fun fact: the timing of the effect is actually done in the shader itself, waiting a uniform amount of time between glitches. This is a testament to how insane the drummer for Wizard Farm is, that he was perfectly on beat at the same tempo every single chorus.)
Making the World a Prettier Place
The art style of the world was inspired very much by the cartoonish look of Firewatch. The textures are intricately detailed, but then brushed over by an ever so slightly transparent layer of a single color. When rendered by a toon shader, it gains a nice flat look, with just a little bit of nuance to it so that it doesn't just all blend together. Take that and drench it in some psychedelia (see James White, and Dali's The Persistence of Memory - both heavy inspirations for the color pallette) and you get the tripped out world of Thin White Duke.
can you post the project? i cant begin to figure out how this is done and i just started on shaders. this is the prettiest thing ive seen
Now we have a good looking world, a solid story, the crazy characters that the song dictates, and a whole lot of madness. I still loved the bridge from the original video... The shader on the reference image was just too perfect. It does a texture lookup based on some trig functions to make it look like it's wrapped around a cylinder that we're seeing the inside of - like a tunnel.
The way it flies past gave me a Star Wars Hyperspace vibe, and when it wraps the cylinder in so many iterations that it all gets totally aliased and just becomes a glitchy, static, alien, noise texture that overstimulates the senses and leaves you wondering where you went and how you got back to earth...
That's the essence of the whole song and of the whole video. I had to keep it, so I green screened the main character on top of it, getting fully distorted and mangled and glitched out by another vertex shader (side note - the reason I kept the take from OF is because when I tried to replicate it in Unity, it wouldn't give me the same crazy aliasing effect. It intentionally prevents that, thereby preventing the happy accident you see above). He, too, wonders where he went, and struggles with the lack of metaphysical laws in this place. The sensory blowout the viewer feels forces them to empathize with our cube friend. We go through an alien experience just as he does.
Letting the Video Dream
The whole thing was wrapped up by passing it through Google's Deep Dream (that's right, it's not just for producing eyeballs and slug-dogs). I used the inception_3b/5x5_reduce model to overlay each frame of the video with painterly strokes - placed beautifully by the AI's curve detection. This gave it more of the feeling of the paintings that had inspired it.
[[{"fid":"65240","view_mode":"default","fields":{"format":"default","field_file_image_alt_text[und][0][value]":"Process picture of the world after deep dream","field_file_image_title_text[und][0][value]":"with deep dream","field_file_tags[und]":"AI Deep Dream"},"type":"media","link_text":null,"attributes":{"alt":"Process picture of the world after deep dream","title":"with deep dream","class":"media-element file-default"}}]]
And with that, it was done. Out of an obsession with shaders and graphics programming - and an inability to make "real" art - was born a music video. One that's in love with games and paintings and film and music, and wants to bring the disparate worlds together for a common cause.
Watch the full video here:
And make sure to check out Wizard Farm.
You can also find me on twitter, reddit, and at my website.
Thanks for reading!
Chris Clapis
Graphics Engineer/Tech Artist