Iray drops renders to CPU despite having RTX 3090 (24GB !!!)
So, as in topic title suggests... I recently upgraded to RTX 3090, updated DAZ to 4.14 with a fresh install of nvidia studio driver 460.89. Yesterday I noticed, sometimes upon changing the preview window from texture shaded to IRAY, nothing came up. I checked the VRAM usage, and it was sitting around 5GB (one g8 actor with some enviroment). Later that night, I was rendering around 20 scenes through the night with a custom batch rendering plugin, which worked great on DAZ 4.11 and 4.12. 10 out of 20 scenes were rendered, and for the rest IRAY spitted out pitch black images with error codes in the log file:
2020-12-24 10:17:36.090 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [WARNING] - POST:RENDER :: 1.0 POST rend warn : denoiser is not available on CPU and will be disabled.
2020-12-24 10:17:36.090 Iray [INFO] - POST:RENDER :: 1.0 POST rend info : Postprocessing denoiser is enabled
2020-12-24 10:17:36.091 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating geometry.
2020-12-24 10:17:36.091 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating motion transforms.
2020-12-24 10:17:36.091 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Importing scene graph.
2020-12-24 10:17:36.091 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Importing geometry for motion time 0
2020-12-24 10:17:36.292 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Geometry import (1 triangle object with 671k triangles, 0 fiber objects with 0 fibers (0 segments), 1 triangle instance yielding 671k triangles, 0 fiber instances yielding 0 segments) took 0.200s
2020-12-24 10:17:36.294 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating materials.
2020-12-24 10:17:36.310 Iray [INFO] - MATCNV:RENDER :: 1.0 MATCNV rend info : found 259 textures, 0 lambdas (0 unique)
2020-12-24 10:17:36.315 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Emitter geometry import (7 light sources with 52700 triangles, 1 instance) took 0.001s
2020-12-24 10:17:36.315 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating environment.
2020-12-24 10:17:36.406 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating backplate.
2020-12-24 10:17:36.409 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating lens.
2020-12-24 10:17:36.409 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating lights.
2020-12-24 10:17:36.409 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating object flags.
2020-12-24 10:17:36.409 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating caustic portals.
2020-12-24 10:17:36.409 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Updating decals.
2020-12-24 10:17:36.438 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Using iray core convergence estimate.
2020-12-24 10:17:36.441 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Allocating 1-layer frame buffer
2020-12-24 10:17:36.473 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Using batch scheduling, caustic sampler disabled
2020-12-24 10:17:36.473 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Initializing local rendering.
2020-12-24 10:17:36.481 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: No device specified or usable
2020-12-24 10:17:36.482 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering with 0 device(s):
2020-12-24 10:17:36.482 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering...
2020-12-24 10:17:36.482 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: No worker to render with: aborting render
2020-12-24 10:17:36.482 Iray Render error: Internal rendering error.
The thing is, I checked the failed scenes one by one, trying to render them individually, and the vram usage topped at 11GB in one scene. Anyways, the problem still occurs randomly even with scenes with very low VRAM usage. Why is IRAY dropping to CPU? Driver issue? DAZ issue? Iray plugin issue?
Help please :)
Comments
Ok, a little update... some new errors came up!
2020-12-24 12:47:38.963 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.10 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while launching CUDA renderer in <internal>:947)
2020-12-24 12:47:38.964 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.10 IRAY rend error: CUDA device 0 (GeForce RTX 3090): Failed to launch renderer
2020-12-24 12:47:38.964 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): Device failed while rendering
2020-12-24 12:47:38.964 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [WARNING] - IRAY:RENDER :: 1.11 IRAY rend warn : All available GPUs failed.
2020-12-24 12:47:38.964 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: Fallback to CPU not allowed.
2020-12-24 12:47:38.964 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while initializing memory buffer)
2020-12-24 12:47:38.964 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: All workers failed: aborting render
2020-12-24 12:47:38.965 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.965 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.965 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.965 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.965 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.965 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.965 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.966 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.966 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.966 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.966 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(353): Iray [ERROR] - IRAY:RENDER :: 1.11 IRAY rend error: CUDA device 0 (GeForce RTX 3090): an illegal memory access was encountered (while de-allocating memory)
2020-12-24 12:47:38.966 Iray Render error: Internal rendering error.
This happened when I rendered one of the scenes manually, saved and closed the final picture window, opened another scene and hit render. What the hell is going on?
EDIT: I think I found the source of the problem. Iray Section Planes. When I delete them, the scene renders normally. Enyone know why?
Yes, Section Planes can do that - a work-around (it may soemtimes have undesirable cosnequences so isn't a fix) is to enable the capping of the mesh in the Editor tab of Render Settings.
Thank you Richard for answering, but can you elaborate on the "mesh capping" option - where is it located?
thanks
EDIT - nevermind, found the option - I will test it and post results :)
Also, about the bug itself - is it going to be fixed anytime soon?
It's been logged with nVidia, I can't recall if it's fixed fixed or if the betas just have the end caps enabled by default.
I got a similar problem and updated windows and Nvidia drivers, now it works. I did not re-add the section planes but it didn't work without the section planes or mesh cap on anyways before the update.
One thing I did notice before the windows update is Client server Runtime process using my GPU at 10%. This could have been the reason for the crash with conflicting programs using the GPU at the same time. Now that process is less than 0.4%.
Can I ask what exact driver version you have, and what exact windows build/current update installed?
I just recently picked up an RTX 2070 super and just tried a render today with selection planes (scenes that would render with my half-sized GTX 970) and it dropped to CPU using the latest nVidia drivers (460.89) and I just checked for new updates (there are none)