Cutout Scene with a Transparent Object... Asked with Paint!

Defenistrat

The TL;DR is that I think I am having the opposite problem that people have with transparent objects. In short, I want to do a two part render. I want to overlay the second render on the first, but there is an object in the first render that partially obscures the object in the second render. Hence, for the second render I would like to make the object in the first render transparent, but use it to obsure the geometry of the object in the second render, such that in the second render the space where the object in the first render overlaps the second object is transparent.

I'm not sure if that is making sense, so let me explain the problem more in depth:

So I have been working on just the right outline style for my toon renders. I am using Toony Cam Pro and Visual Style Shaders. Unfortunately, I am not happy with the outlines of Toony Cam Pro on complex hair meshes, with or without the texture maps, normals or depth based outlines. After trying lots of different things, the best I can do is turn off transmaps, and either settle for too many lines using normal mode, or really subtle this dark lines that kind of smudge the highlights of the hair all the hair look dark. However, I love the outlines I am able to get on every other object in the scene with ToonyCamPro.

Then I tried just using the outlines in the Visual Style Shaders on the hair. It looks great! I can tweek it to get subtle black lines, but not too many, and the hair is still vibrant with bright highlights. However, I can't get the dark outlines on normals of individual objects within the figure, For example, I can't get an outline between two different materials zones within the figure, such as where the shirt touches the skin.

So I thought about using Toony Cam but turning the outline rendering off for the hair, which, if you are familiar with ToonyCam you already know this won't work and results in something like figure B3 (Bad Render 3) in the image below.

So that's why I am thinking of creating a two part render. First render the scene and figure with all the outlines, and then render the hair to a transparent background and overlay it on the figure. The problem is, of course, that because the hair is obscured by the face, I can't just render the hair seperately, or the back of the hair will be over the face (B1). I can't do the reverse, or I would have B2.

Hence, in the image below, can I render O2 in such a way that it renders the parts of O2 as transparent where it is overlapped by O1?

If not, then maybe I am asking the wrong question. Any other work arounds for this?

(Yes... post work is technically a work around... my goal has been to avoid post work as much as necessary. Plus I'm really, really happy with the results of ToonyCam Pro except for what it's doing to hair.)

 

 

 

Richard Haseltine

One option might be to do a material ID render using IRay, giving the hair one colour and everything else another (this is accessed via the Advacned tab of Render Settings, in the Canvasses sub-tab). Then you could use that as a mask for the ToonyCam render of the hair.

Defenistrat

Thanks, Richard.

So I tried the MaterialID Canvas, there are two problems:

1. Material ID Canvas does not let me select a Node List, so it renders all the objects in the scene.
2. It includes the skull cap of the hair in the hair's material id, which obviosly partially covers the forhead.

However, I tinkered around with canvases a little more and seem to be able to make a mask using a Beauty Canvas. Here are the steps I'm taking, for posterity (whether that is me forgetting some day, or somebody trying to do the same thing):

1. Under Render Settings, select the NVIDIA Iray engine.
2. Select Advanced Tab
3. Select Canvases Tab
4. Follow the steps I am stealing from this post to avoid a few key strokes http://www.daz3d.com/forums/discussion/comment/877787/#Comment_877787 :

   1. Click the plus button, and added the Beauty canvas.

   2. At the bottom of the pane, click the + besides Nodes. Give your node a name -- the default NL1 is fine.

   3. Click the almost-impossible-to-see dot-dot-dot (...) button to the immediate right. A list of scene elements appears.

   4. Select the elements you want to render.

   5.  In the middle of the pane, click the Nodes pulldown, and choose NL1.

5. Switch back to the editor and make the following changes:

   1. Change the Render Mode to interactive (when set to photoreal it seemed to include part of the skull cap of the hair)

   2. Turn off Tone Mapping

   3. Be sure the canvas you just made is slected under Canvases.

6. Render

This creates a white siloette of the hair that is visible in the scene against a black background (perfect for a layer mask!). The only prblem I am having with it is that it takes forever long to render it! Right now I am staring at what looks like a good enough layer mask, but it took 3-4 minutes to create. Even now I have been rendering for about 7 minutes and it still says it's 0% complete. How can I speed this up?

Some additional follow up questions are:

1. As asked above, can I speed up this IRay render of my mask?
2. Can I render using a layer mask directly in D3D? Is there a way to render in 3Delight but say only render what is on the white part of this other image?
3. I am now rendering across 3 different "engines", IRay for my layer mask, Scripted 3Delight for my outlines generated by Toony Cam, and then vanilla 3Delight for for my hair render. Can this all be scripted in Daz3D? I'm hoping to create a pipeline for generating lots of images with as little post work as possible, and combining three different layers seems monotonous and could be automated if Daz supports it.

Thanks

 

Richard Haseltine

2. One thing I meant to suggest - change the diffuse and ambient colours of the hair to white, ambient strength 100% (this will kill all sahding), set everything else to black, black, 0% and also black out specularity, turn off all lights (just use the ehadlamp) - that should give you a mask image, if I am thinking clearly (never a safe assumption)

3. Yes, you can use scripting http://docs.daz3d.com/doku.php/public/software/dazstudio/4/change_log_4_8_0_59#public_beta_3_rc has the entry

- Added support for post-render script; Render Settings > Editor > General > Post-Processing > Post Process Script; executes post-save of a render

so you could render your images and use a script to composite them, or to launch (via DzProcess) an external tool to composite them

ben98120000

Perhaps you can use section node (from menu Create - New IRay Section Plane Node...) to cut off rendering of the portion of the hair that wouldnt be visible behind the head, like I did with the back of the hair in the picture..

Defenistrat

What I wound up doing for this was just to use the the method described above to render a layer mask using the IRay engine. To speed up the render, I turned off all shadows, light refracting... pretty much everything thing that would require extra processing. I cut the number of samples from 5000 to 2000. Then I set the max render time to 180 seconds. Obviously, what is really speeding this up is changing the max render time, but I am just trying to get the best shape in the least amount of time. When I am done, I use GIMP to combine my outline layer, and my non outline layer with the layer mask applied to it.

 

Thanks for the help and suggestions!