Music Video for the Metaverse
At the end of 2021, we were approached by Soundr, a startup vying to be 'the world's first metaverse music lab'.
Watch a 30 second clip of Skullz singing 'Blackout', with footage wholly created in real time in FirstStage.
Until the singles are fully released we are limited to just this clip, but go to the end of the article to see the full video with no sound.
Who are Soundr and what did they need?
The founders of Soundr are music industry veterans: Pete Kirtley is an award winning writer and music entrepreneur behind seven number one singles worldwide amongst a long list of top 40 hits and over 250 million record sold; and John Black, is a talent manager of over 25 years who has developed dozens of global stars.
As Soundr, they embarked on a pioneering approach to music, embracing Web 3.0, NFTs, and virtual artists powered by AI, and had already styled a virtual band of avatars called Skullz and written a couple of catchy launch singles. They were keen to reveal it all to the world in one of the new and developing metaverse spaces and needed some pitchviz to excite investors to fund development and marketing, but of course they were pre-money, so they needed something low cost and delivered in two months as they were targeting a launch at Decentraland’s first virtual metaverse music festival in early December.
“The team at Moviestorm are so supportive, and always go the extra mile to help deliver your vision. Through their decades of experience they also provide insights to what else could be possible that you may never have thought of.”
Pete Kirtley, Soundr
Our plan and solution
Despite FirstStage still being in early Beta, we were highly confident that we could mocap live performances and deliver either a 3d experience to be appreciated in VR, or footage out of the tool to edit into compelling music videos.
With only a minimal budget, the first issue to resolve was where could we source some low cost sets that echoed the neon cyberpunk style of the band, and the Unity Asset Store yet again delivered with a number of very affordable sets and concert stage assets.
The characters were a bigger issue, as they had to look exactly right.
With FirstStage already supporting Reallusion’s Character Creator, the initial forays unearthed some costumes and hairstyles that were close.
Meanwhile Pete Kirtley had a play with the Ready Player Me character generator, as a solution that might be more compatible across the various implementations of the metaverse, and these produced some suitable avatars. Exported as .fbx they imported straight into FirstStage as rigged characters, so we soon had them standing on stage to compare.
The mocap test session findings
There were a number of aims for the test session: to let Pete become orientated with VR and understand what mocap will require of him; to see what we could get with the current build; and to identify the critical challenges to address for the mocap recording session.
We used a Vive headset, Valve controllers (as they are less bulky and did not need to be permanently gripped) and Vive trackers (waist and ankle), and we recorded lead, guitar and drum parts for one song, finding the priority issues to be:
- A lack of finger posing;
- A lack of facial animation;
- Sound to visual latency in VR;
- Skeleton distortion around the stomach.
Critical fixes
Poseable fingers were fairly straightforward, and enabled by simply including the finger bones in the skeleton and rigging up the inverse kinematics for the finger IK.
A facial animation module was engineered over a weekend, choosing Apple’s ARKit Face Actor component and using the free Unity Face Capture companion app to enable anyone with an iPhoneX onwards to be able to use it.
Improvements were also made to reduce latency to the minimum possible, so that Pete could see his drum sticks in VR hit the virtual drum on the beat, which massively improved the performance quality.
“Considering how complex and expensive getting hold of motion capture footage can be, it is amazing that with FirstStage we could create what we needed in just a few hours, in a small room with a basic VR setup and some trackers.”
Pete Kirtley, Soundr
Finally, the issue with skeletal distortion which was not such a simple fix. The problem being that the tracker is meant to track pelvis movement, but where and how the tracker sits in relation to the spine can vary depending upon your build. We tried siting the tracker on the back but that had other issues, and higher up (to avoid it rotating downwards if there was even a slight belly) which improved things, but we had to accept that such low cost mocap with so few tracking points was bound to have some hard to fully-prevent issues.
The mocap recording session
At the end of November we recorded the mocap session for all band members playing both songs, all of which were performed by Pete Kirtley in our small garden studio in an afternoon.
Pete performing the motion capture for Sneeka, Orkid and Hyde
“I cannot emphasise how effective it is to record motion capture in real time in virtual reality. You can see and interact with the rest of the band, the audience, and the stage furniture, making the performance truly grounded in the scene.”
Pete Kirtley, Soundr
In preparation for the scene, the band members were placed on their marks, and we added a number of audience members for reference.
As it was in VR, Pete could not only see things like his drum kit, but the whole scene including the other band members and audience around him, enabling him to mocap looking at characters or instruments, which added to the grounding of the characters within the scene.
We also rooted the performance by placing a physical box in the scene that Pete could place his foot on when performing, and then we simply placed a monitor speaker model in the scene and aligned it with the raised foot position.
Finally, the issue with skeletal distortion which was not such a simple fix. The problem being that the tracker is meant to track pelvis movement, but where and how the tracker sits in relation to the spine can vary depending upon your build. We tried siting the tracker on the back but that had other issues, and higher up (to avoid it rotating downwards if there was even a slight belly) which improved things, but we had to accept that such low cost mocap with so few tracking points was bound to have some hard to fully-prevent issues.
Post session
There were still a few jobs to be done. The audience was small and static, so we needed more animated people. Unfortunately, any dancing had to be performed by Moviestorm staff rather than professional dancers, but we did our best and sent Pete a huge selection of camera keyframes as reference for the camera direction.
“FirstStage delivers a truly flexible creative environment, where it is easy to go back and adjust the set, performance and cameras at any point.”
Pete Kirtley, Soundr
FirstStage includes a simple linear video editor but for the finished video, each camera sequence was exported so Pete could use his go-to, Adobe Premiere Pro, to edit the sequences into a finished video. So with those shots and moves set up, we proceeded to render out the full performance from every camera, and send them all to Pete, who produced the final edit. Enjoy …
Watch the full Skullz music video edited by Pete Kirtley using footage wholly created in real time in FirstStage.
Until the singles are fully released there is no sound to the video for IP protection, but trust us, they are great tunes!
Conclusions
There were a lot of challenges to delivering this production.
Overall, it was not really previs. Soundr wanted finished animation to create videos to promote the band with. Of course Soundr could have gone to any professional 3d animation house, but they had a pressing deadline and a very small budget, and they were also hoping to replay the footage of the band in 3d VR as ‘live’ performances in the metaverse, so the solution needed to technically provide for that real-time potential.
Quality takes time, and time equals money, and money was very tight. So our fast, flexible previs solution needed to deliver fast, flexible and polished output. And polished meant that details that are not too important for previs became very important, such as facial animation, poseable hands, and accurate mocap.
In the end, we were very proud of what we managed to develop and successfully deliver in such short order … but maybe not the audience Dad-dancing.
“If you’re looking for a fast and affordable solution to create 3d animated scenes, then look no further that this tool.
Working in VR makes FirstStage simple for non-techies to use, and can output anything from quick and dirty to release quality.
What you get really is up to you, which is fantastic when talking about creative vision!”
Pete Kirtley, Soundr
References and Copyright
Skullz artwork images copyright Soundr Limited
CyberPunk_City set, copyright Art Equilibrium
Stage Vol 1 assets, copyright 3DAssetLibrary
Night Club set, copyright Tirgames
Sci-Fi Females costumes, copyright Alley
Pigtails Hair, copyright fionaqunsh