Check Out a Full-Scale Motorcycle in AR With Theia Interactive
We met with Theia Interactive, an architecture services company for virtual tours. CTO Stephen Philips and 3D artist Cameron Schulz explain how they made an AR project in UE4, showcasing a Harley Davidson motorcycle with modifiable aspects.
Stephen Philips: I'm the CTO of the company, I manage our technical direction and oversee all the artists and developers for our various projects. Based on project needs, I'll look at new tools and updates of existing tools to make sure they suit the artists’ and developers’ needs.
Theia Interactive was founded in 2014, so we're a pretty young company. The team has grown to 18 members, with about eight 3D artists. We are working on a lot of projects; about 10 are either being developed or finalized for proposals, most of which are architectural projects.
We just wrapped up and are probably going to revisit the Harley Davidson AR experience, which was initially shown at Autodesk University 2017. We’ll be presenting it this year at GDC on the Unreal Engine booth, with Epic sponsoring the project.
Cameron Schulz: I’m a 3D artist at Theia Interactive. I mostly do texturing, lighting and run the AR and VR projects in Unreal Engine. Occasionally, I also do some coding.
I discovered Substance when I was looking for a way to texture my assets. At the time, I was using Photoshop and Mudbox, but it didn't work out too well. At California State University, Chico, they didn't teach Substance yet. They do now, but too bad I'm not a student anymore. My teacher and many students recommended Substance Painter, and when I tried it, it was far easier than anything else I had used.
The Harley Davidson experience is the project on which I used Substance the most. I used a lot of the materials that come with Substance Painter. Let's jump into the breakdown, so I can show you how we textured the Harley Davidson.
Theia Interactive was tasked with building an application that shows how Unreal Engine can be used to create a photoreal AR application using engineering-grade models. We received a highly detailed mesh of a Harley Davidson 1937 UL made by Amir Glinik. Using an iPad Pro, users would be able to walk around a life-sized motorcycle as well as restore it from “barn find” to “restored” and animate the bike into an exploded view to see the interior components of the engine. The mesh was 13 million triangles, composed initially of thousands of pieces, and we needed two complete sets of textures for it, so it was a challenge to organize it all.
Because the motorcycle was the only real asset in the iOS application, and we used the latest iPad Pro 10.5”, we had a significant budget for higher than normal polycount, high texture sizes, and a large number of maps. To easily cut up the model up to lay out UVs I was able to utilize Maya’s 3D cut and sew tool. After cleaning up and laying out UVs for textures and lightmaps, I was left with 31 material assignments and 109 meshes. Using the auto LOD feature in Unreal Engine, the motorcycle as a whole never showed more than 1 million triangles on screen, which allowed us to produce a stable 60fps AR experience.
Substance Painter recognizes material assignments rather than individual meshes, so to save on performance a duplicate of the bike was made and combined down into five meshes that we would bring into Substance Painter as a single FBX. To be able to quickly assign materials to different parts of the motorcycle, we also made an ID-mapped version of the bike. Using reference photos, we drafted a list of material types, and we assigned arbitrary RGB values to them. This would ultimately save time by not having to manually make masks for where each type of material would go.
Once in Substance Painter, we baked out all the maps needed: AO, Curvature, Position, World space, and ID. Because there were 31 material slots, and each required six maps baked out, most maps were baked out at 256-512 to save on initial file size, with select assets being baked out at 2048. This allowed for all the generators to have maps to work off but not increase the file size too much. In a first round of texturing, the model had 70+ materials assignments all with 2k textures causing file sizes to reach upwards of 8GB. Reducing the number of material assignments, as well as setting texture sets to 128 when not in effective use, decreased file sizes to 2GB, allowing me to work with a higher quality (128 spp).
Since the user would be able to switch between a dirty and restored version of the motorcycle, we needed to make two versions of all the textures. To avoid doubling the workload, we made the clean version first and then dirtied it up to make the decayed version of the bike.
Making the clean version was relatively straightforward. The first step was to paint any height information that was not represented in geo. This included text on the tires as well as ridges on the handlebars and tubes. We placed select materials like the leather on the seat and the paint with decals first.
Using materials that ship with Substance and a little tweaking, I was able to make a layer stack that had all the different materials needed for the rest of the bike.
By using ID masking and converting the layer stack to a smart material, I was able to duplicate the textures to all of the other texture sets.
Midway through the project, an update to Painter was released allowing me to instantiate layer stacks across selected texture sets. This feature enabled much faster revisions of textures across all sets. With duplication of smart materials, a change of value in one layer stack would require going into each texture set that also used that material in order to change its values. With instantiate, if you change the master materials values, it changes across all the texture sets, which helped when we had 30 objects using the same base material.
Once the clean version was locked down on materials, it was time to layer on the dirty version. Unfortunately, the instantiated layers needed to be converted back to a smart material duplication system. The way the decayed layer stack worked was through a pass-through system where levels were used to dull and slightly change the color of the base materials in the clean layer stack. On top of that, we layered rust, dirt, and grime using mask generators. At the time of doing the project, passthrough layers did not affect instantiated layers except on the master material. Having both the clean and rusted version in the same instantiated layer wasn't very feasible since, unlike the clean version, we needed to change the values on the masks that generated the decay on a per part basis. This was not too much of a setback since the clean version was mostly set in stone and did not need further revisions.
As with the clean materials, the masks and generators used to place rust were all amongst those that ship with Substance Painter. Every part of the mesh got a similar layer stack consisting of a levels adjustment to dull any metals and darken colors. On top of that, we added rust through a dripping rust generator overlayed with a custom mask editor that could be adjusted per mesh to be more or less rusted. The next layer was dirt and general wear (scratches, smudges, etc.), with a sharpen layer above that to make things crisper. The final layer is a darker grime layer; this depicts any oil or mud that might have collected over time.
Decay Breakdown: Levels Passthrough
At the bottom of the layer stack is a fill layer set to passthrough. Using passthrough on an empty fill layer will allow levels and filters to affect everything below it in the layer stack. Two levels were added on this, one targeting the roughness channel, the other targeting base color. The roughness was adjusted to have a larger range towards a value of 1, effectively dulling the texture. The levels used to target the base color were used to add more range to the dark values, causing the colors to darken.
Decay Breakdown: Rust
For the rust, we needed three things: a rust texture, a mask for where the rust would be applied, and a layer to add depth to the rust. We used the Substance material Rust Fine as the base rust texture. This was put inside a folder that was masked out with a generator MG_Dripping_Rust. To add depth to the rust, we added a levels channel inside the folder, with a height value set to -.015. It lowered the overall height range to less than that of the base metal, making it appear recessed.
Decay Breakdown: Dirt / Wear
On top of the rust is a layer of dust and general wear. The wear layer is a Substance material “Grunge_Scratches_Dirty.” The material was left at default values with a roughness levels adjustment and masked by an AO map. We added dust through use of the mask editor generator. This allowed for the dust to be masked in occluded areas using the AO maps and on the tops of surfaces using World Position Maps. To dial back the amount of dust and wear that was shown, we put these layers into a folder masked with MG_Dripping_Rust. This allowed us to add dust with the effect of it having settled on top of the mesh and streaked down over time.
Decay Breakdown: Sharpen
This consisted of another blank fill layer with a sharpen filter of .2 so all the rust and mask generators could get a sharper look.
Decay Breakdown: Grime
Initially, there was no “grime” layer. When looking at the model as a whole, everything had a very dull, dusty look. This was not visually appealing as there was a lack of contrast. While looking at reference photos most had a layer of oil or a dark black sheen.
The base fill is a black value with low roughness. We used MG_Ground_Dirt to mask the top of the mesh with a dirtied mask. Using a fill and Grunge_Dirt_Scratches, we were able to dial back the effect so as not to be too intense.
Export to Unreal
After we finalized texturing, all maps, AO, Curvature, etc. were rebaked at 2k to get cleaner masks for the generators. Using the Unreal Engine preset for packed maps, we exported everything out through a custom version of the packed maps preset and brought it into Unreal Engine.
The only map which we were not able to bake in Substance Painter was bent normals, so we did it in Substance Designer. We kept most Baker parameters at default, while increasing secondary rays and setting Anti Aliasing to 4x4.
The animated pieces had to be moveable objects in Unreal Engine, but we didn’t want fully dynamic lights in the scene. To keep these moving metal pieces looking good, we incorporated AO and bent normals into the material. While AO generally darkens areas, bent normals allows for more precise occlusion of specularity in tight corners and they recently became supported in UE4. As you can see, the mechanical pieces of the bike have highly realistic lighting, even though there is no baked lighting inside Unreal Engine.
Unreal Engine Material Setup
Needing to switch between two materials during runtime called for a particular setup in the material editor. Using material parameter collections, we could change every material at once by lerping between two different textures.
Textures, packed maps, and normals were all set up to lerp between two texture sets. The input for bent normals uses an input node separate from the base material attributes.