GPU -> CPU -> CPU -> GPU -> CPU -> CPU ->... Strange render pattern!

dudeness

GPU -> CPU -> CPU -> GPU -> CPU -> CPU ->...

GPU -> CPU ->  GPU -> CPU -> GPU -> CPU ->...

Hi everyone, I am sure many of you have noticed or experienced the problem of GPU render falls back to CPU in the past. Same here! But the titled issue is rather new to me.. normally if GPU dies it doesn't revived..

I have 2 PCs one with 1080 and the other 2070, both experienced the same issue occassionally. Both got Daz 4.12 pro installed (with CPU fall back unticked) and newest GPU drivers.

What happen?

So I am creating an animation with 80 frames, (and with post denoiser on after iteration 30, mithcell 1.1) Before rendering the file series i rendered the first frame just to be sure the Vram is enough to render the pic. It worked well. I also got the GPU-Z (for those who doesn't know it's a software help to check various status of your GPU)  to monitor the usage of the GPU. It was 100% so I feel it's safe. Each frame is limited to 10 minutes (which normally will have the iteration reached 100+ so the post denoiser would surely have kicked in).

Then after one day I've noticed the titled CPU fall back pattern. (one time i experienced alternative pattern). So i think the CPU fall back is still an issue. And to summarize, i've seen 3 types of GPU failures as follow in the last 2 years since Iray kicks in:

GPU failure cases

Case 1) GPU failed from start --  too large scene with tonnes of geometry / CPU fall back from previous session, and without restarting the PC / in this case GPU-Z also showed NO GPU usage

Case 2) GPU was used for the first few frames during animation, but then fall back into CPU indefinitely -- which seems to be the most common case

Case 3) the funny pattern metioned in title -- can also be GPU -> CPU -> GPU -> CPU -> etc

Bonus Case) GPU fully loaded, BUT post denoiser not activated  -- now this is a strange one. Eventhough the GPU is shown as 100% usage (GPU-Z), the post denoiser was not activated and thus your image will look noisy. From some previous threads it was mentioned that actually post denoiser will only function with GPU render. So if CPU falls back then no post denoiser will take place. But this might not be the only cause.

I've been using Daz for 5 years now and very happy to see how it developed. Especially it's a free software. If this long lasting issue can be fixed as well then it'll really be perfect..

I have contacted the tech support already. Have you guys experienced this too? or have some thoughts of the cause? (for the pattern case i suspect it has something to do with each render not freeing up the vram.. or something similar..)

p.s. attached with a screencap showing the pattern.. it seems that when denoiser got activated then the file size would also be smaller...! interesting...

Richard Haseltine

Exactly which version of DS is this - Help>About Daz Studio? The last couple of versions have had a chnage to the way resources are managed that is designed to address this issue.

dudeness

Thanks Richard for the responding.

The version I'm using is v4.12.2.6 Pro Edition (64-bit) Public Build.

After some further digging into the issue i believe the GPU->CPU->GPU swap falling thing was most likely due to the denoiser. The end of this post shows the extract from the log file when the fall back happened.

An user from other forums suggested the follow which i believed is what i am experiencing: (started with GPU, denoiser supposed to kick in, Vram overloaded then the rest of that frame will be CPU rendered. And the next frame will start with GPU again etcetc)

The denoiser requires a lot more VRAM. If the software detects that you do not have enough VRAM to render a scene with the denoiser enabled, it will be disabled automatically. A 'soft-fallback' to 'normal' Iray so to speak.

It seems even unticking the CPU fall back does not prevent this from happening.

Thanks again for looking into it.

 

WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(302): Iray ERROR - module:category(POST:RENDER):   1.0   POST   rend error: A CUDA error occurred: out of memory

WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(302): Iray WARNING - module:category(POST:RENDER):   1.0   POST   rend warn : Failed to allocate device memory, falling back to CPU

WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(302): Iray WARNING - module:category(POST:RENDER):   1.0   POST   rend warn : Image denoising is not available on the CPU and will be disabled.

WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(302): Iray WARNING - module:category(POST:RENDER):   1.0   POST   rend warn : Postprocessing is running on the CPU, but Denoiser is requested. Denoising is not available on CPU.

kenshaw011267

The denoiser does use some VRAM when it starts. So if you're right on the verge of running out of VRAM just from the scene it can cause the render to drop to CPU.