Puppet Capture & VR

A bit of story…

I grew up on Jim Henson’s Storyteller on Tv, The Labyrinth and The Dark Crystal, old black and white Czech and German stop motion films. I loved everything that had puppets in it.

I fell sideways into the film industry in London and was lucky enough to get a job as a Render Wrangler at Jim Henson’s Creature Shop in Camden, London just before it was closed down. My girlfriend has a Bachelor in puppeteering from UNATC (National University of Theatre and Cinematography) Bucharest. So I have always have been surrounded with anything puppets. I’m utterly fascinated by how a inanimate object can by manipulation of a puppeteer(s) come to life and tell complex stories both for children and adults alike.

I have always had an obsession with storytelling, especially through technology and how stories shape our lives from the brother Grimm, Hans Christian Andersen, Oscar Wilde but also how the old folk tales of my native Denmark and, the sagas of Scandinavia, Iceland and how the oral tradition of telling stories from the old days has evolved. I wanted to explore a way of emerging the “listener” deeper into the stories themselves and when Virtual reality presented itself I felt I had to seize the opportunity.

 

Virtual Puppet Capture and folktales.

The project is based on the idea of motion capturing real physical medium sized “bunraku” or rod puppet, via the Rokoko Smartsuit motion capture suit. This is done in order to retain the unique motion and style of a real puppet. The captured animation then drives a virtual puppet sculpted in Oculus Medium and rendered in real time inside of Unreal Engine 4.

 

 

  • Oculus Medium is used to sculpt the entire character, decimated on export to fit the 250K polygon limit of Maya LT.
    I have found that this limit is a perfect match for my simple characters in order to keep the character polygon size down and to support its visual style.

 

  • Rokoko Smartsuit motion capture suit / sensors are used to capture the physical puppets unique motion and behavior.

 

  • Using Maya LT as a intermediate to rig and skin, define material slots and to do a bit of motion capture cleanup. Mesh & rig and is then exported as FBX into Unreal Engine for scene creation. Animation is exported as a separated FBX

 

  • The character is imported in to UE4 and is laid out on the sequencer together with the imported animation. The Sequencer is a great tool for creating a fast linear workflow and excellent for quick prototyping.

 

Oculus Medium.

Oculus Medium has enabled me to try and pursue my own style through an easy to learn but hard to master tool that gives me the opportunity to unfold my creativity and help me create a fantastic and fast workflow. (Although I still have so much to learn!)

 

What’s the big idea?

I want to use all of this to create a episodical VR series for the Rift and the Oculus Quest

Where people can experience and interact with a series of Saga’s based on everything I grew up with. From the viking legend of the Valraven to the Norwegian personification of the plague – Pesta: the old woman, to the Bog Lady brewing the morning dew in her giant cauldron.

I want people to be able to step into virtual reality and straight in the the stories and through the characters, animation and music feel the story first hand, to learn from its message and to be transformed through emotional reflection.

The project is a one man passion project that I hope one day will take over my life and will give me a chance to fulfill my dream. To build a small studio of friends where we can explore and create more heartfelt stories for other people to experience.