Friday, August 7, 2015

Problems with Matte Painting and Multi-Plane Effects in Maya

My current production (more on that later, I hope!) is making heavy use of painted backgrounds and flats to flesh out the world of the show. To get these to look correct through a Maya camera, one technique we've used is to import the Maya camera (and some NULLs) directly into After Effects, then attach the paintings as 3D layers in AE. But some paintings need to be rendered out of Maya because the painted geometries can be non-planar, they intersect each other, and so on.

We're using projection to get the paint onto our geometries. To do this, we set up a "paint projection camera" that views the entire scene, we render standin objects from that camera, and paint on top of that render at high resolution until it looks like we want it to look thru our "scene camera" (which of course might move through the scene). Once the painting looks good, we project the finished painting back onto the original standin geometries using a Maya projection shader through the paint projection camera.

It has taken me a long time to figure out the magical combination of Maya nodes to actually get the paint (and its opacity) to "stick" to the geometries. The default and obvious things to try simply didn't work, with the weirdest artifacts to boot. I tried googling things like "maya projected paint and alpha are different" and "maya matte painting problems" but got no results; hence those terms in this sentence and this very post!

Rather than talk through all the testing, I'll cut to the chase. Here's the shading network that works for us:

A functional shading network for projected paint (and alpha) in Maya

Walking through it backwards: We start with a Lambert shader. Diffuse = 1, with both Color and Transparency are coming from the painted texture. Note that to "light" the paintings we use an ambient light with Ambient Shade = 0 so that the rendered colors match exactly what the painter created.

Both the Color and Transparency come through their own projection nodes. We need two of these nodes because I could not get the transparency passthru of the projection node to actually pass through the transparency of the texture file no matter what I tried. But they both reference the same camera and are identical copies of each other in every way.

The file node references, in this case, a PNG image with alpha. I tried using the file node's Out Alpha parameter, but when I inverted that and passed it through a projection, it always ended up squeezed relative to the color channels (like a subtle non-proportional x scaling was applied). This made no sense to me and led to slight misalignments between the paint color and alpha that showed up as mysterious white and black streaks near the edges of the paint.

I truly believe there's a bug here in how Out Alpha is processed from the file node, because using Out Transparency from the same file node worked just fine. I also think the behavior of the Transparency pass thru on the projection node is either broken or quite counter-intuitive (aka: if you project both Color and Transparency through the node, they should both get transformed identically and remain accurately registered).

I hope this is helpful for any of you trying to get paint projected onto objects in Maya. If anyone has figured out other successful methods, please share them!