Denoiser sometimes not working?

ac_mr_blonde

Hello all,

I recently upgraded to version 4.12.1 and am now seeing a very strange issue where the denoiser sometimes simply does not run.  Same scene, same settings, one render will engage the denoiser, the next will not.

I typically render either 500 total iterations with the denoiser kicking in at 450, or 5000 iterations with the denoiser kicking in at 4000, and I have seen this issue in both, across multiple scenes.  Obviously, this is annoying after a ten minute render, but I'm ready to flip the desk when it happens after a 8 hour render.

When it happens, I close Daz all the way, make certain there are no Daz processes still running, open it up and try again.  Sometimes this works, sometimes it doesn't, and I am not certain of the why or wherefore of this pattern.

Has anyone else seen this?  Is it a known issue?  Any advice?

Richard Haseltine

Is it dropping to CPU? The denoiser is one of the few features that absolutely requires a GPU.

ac_mr_blonde

I didn't know that.  I don't think it is, but I will check.  Why would it drop to CPU on one render when it doesn't on another?

Richard Haseltine

If it's very close to the memory limit then almost any slight variation in what other applications are using or how much debris Iray itself has left may tip it over the edge - the fact that restarting usually fixes the issue is suggestive.

ac_mr_blonde

Ok, i took a look at memory usage and I don't think that's the issue.  Here's what I found:

• If I set the densoiser to start on iteration 1, it works fine, 100% of the time.
• If i set the denoiser start iteration to a later start iteration, it frequently won't kick on.  I can't figure out the repro for when it does/doesn't start.  Same scene, same settings.
• The GPU memory isn't maxed out in either case, but the task manager reports more memory being used when the start iteration is later than when it's iteration 1 (7.7 GB vs 7.1 GB).
• It's worth noting that this is a new behavior that started about the time I installed the latest Daz update through DIM.  No idea if that's connected, of course.

So I have a work-around, which is starting the denoiser on frame 1, but that slows down the render a lot.

Richard Haseltine

What is your GPU? 7.7GB would be close to, or over, the limit for an 8GB GPU.

ac_mr_blonde

It's an 8GB 1080ti.  But I'm not sure why that would make a difference between starting the denoiser on iteration 1 or iteration 450?

 

Richard Haseltine

I don't know why the early start would use less memory either, but I do think the slow downs may well be that it is borderline on memory and dropping to CPU - you can check the log file after rendering, it should give a summary of which device did how much of the work.

ac_mr_blonde

I have tracked down more information using the log, thanks for that suggestion!.

The culprit is whether or not I have used Iray as the render mode in the Daz studio viewport before rendering the scene.  When I select Iray as the view mode in the viewport, the log reports that the GPU runs out of memory and switches to CPU, and after that any scene render to a file runs out of memory as well.  If I render to a file without switching to IRay in the viewport, it works fine.

I looked at the two logs (one from rendering to a file, one from switching to iray mode in the viewport, relevant sections attached).  The numbers are almost the same, until it gets to the lines "allocated for frame buffer" and "allocated for work space", at which point the viewport render uses almost half a gig more memory.

So, two questions:

• Why on earth is the viewport render so memory hungry compared to the normal render?
• Why does it permanently trash my GPU for future renders? (the card shows almost a GB of memory still used after the viewport render is closed that never frees until I close Daz Studio, and, to be clear, that memory is not retained after a successful GPU render, I checked)

Richard Haseltine

Once the GPU has run out of memory it is necessary to restart DS, this seems to be an Iray limitation.

I don't know why the viwport preview uses more memory, you might want to compare the option in Draw Settings (with iray preview on) and Render Settings to see if there are any obvious differences.

Sevrin

Richard Haseltine said:

Once the GPU has run out of memory it is necessary to restart DS, this seems to be an Iray limitation.

I don't know why the viwport preview uses more memory, you might want to compare the option in Draw Settings (with iray preview on) and Render Settings to see if there are any obvious differences.

This leads to me wondering whether, since Iray is a plug-in, couldn't DS just reload Iray without the need to be itself restarted, something which causes us to have to reload our scene and assets?

Richard Haseltine

Sevrin said:

Richard Haseltine said:

Once the GPU has run out of memory it is necessary to restart DS, this seems to be an Iray limitation.

I don't know why the viwport preview uses more memory, you might want to compare the option in Draw Settings (with iray preview on) and Render Settings to see if there are any obvious differences.

This leads to me wondering whether, since Iray is a plug-in, couldn't DS just reload Iray without the need to be itself restarted, something which causes us to have to reload our scene and assets?

I don't think DS can reload plug-ins - that, presumably, is why it has to be closed when installing a new/updated plug-in.

mwasielewski1990

ac_mr_blonde said:

It's an 8GB 1080ti.  But I'm not sure why that would make a difference between starting the denoiser on iteration 1 or iteration 450?

 

1080ti has 11GB. Are you sure this is the model of your GPU?

Also, if you have a 8GB GPU, effectively, you have ~6.5 GB VRAM available. Some video memory is needed for displaying the operating system itself (+DAZ), plus IRAY needs some additional memory for it's internal purposes.

ac_mr_blonde

 

 

mwasielewski1990 said:

ac_mr_blonde said:

It's an 8GB 1080ti.  But I'm not sure why that would make a difference between starting the denoiser on iteration 1 or iteration 450?

 

1080ti has 11GB. Are you sure this is the model of your GPU?

Also, if you have a 8GB GPU, effectively, you have ~6.5 GB VRAM available. Some video memory is needed for displaying the operating system itself (+DAZ), plus IRAY needs some additional memory for it's internal purposes.

On this box it's a GTX 1080 with 8GB.  Yeah, I get that some memory is reserved outside of Daz, the issue is that there are differences within Daz itself depending on when you start the denoiser, which is compounded by the fact that IRay doesn't give back all the memory it grabs until you restart Daz.  The first seems really bizarre to me.  the log shows Daz reserving a really different amount of memory with the denoiser starting on iteration 1 than when it starts mid-render. 

I also wonder if the workspace amount listed in the log is something unhelpful like "everything left on the card", because the memory amount seems to go up even when the materials and geo go down..