Loading...
> >

Iray suddenly using FAR more memory

hesperasmithhesperasmith Posts: 52
in Technical Help (nuts n bolts)

When trying to run an Iray render, DAZ Studio has suddenly started using 3-4x as much system memory as it was previously for the same scenes, to the point where some even very simple scenes won't render at all. This has just started happening in the last few days.

Any ideas on what could be causing this and/or how to fix it?

«1

Comments

  • Richard HaseltineRichard Haseltine Posts: 100,747

    Is the Render SubD level still the same on any SubD items (Parameters>Mesh Resolution)? If it was somehow changed in its default value that would affect everything not using a custom value.

  • PadonePadone Posts: 3,688

    interesting .. does this mean that somewhere there's a global subd value that can affect all the scene items at ones ? If so where do we set the default value for subd ? and how do we set an item to use the default value or a custom value ?

  • LinwellyLinwelly Posts: 5,944
    hesperasmith said:

    When trying to run an Iray render, DAZ Studio has suddenly started using 3-4x as much system memory as it was previously for the same scenes, to the point where some even very simple scenes won't render at all. This has just started happening in the last few days.

    Any ideas on what could be causing this and/or how to fix it?

    Do you have older renders still open? That will use a lot of your gc capacity even if there seems to nothing happening.

  • Richard HaseltineRichard Haseltine Posts: 100,747

    No, but the setting on the asset file will, in general, be used if there isn't an override - I'm not wholly sure that applies to the SubD levels but it's the first thing that came to mind that might affe t multiple scenes that havcen't been changed directly..

  • hesperasmithhesperasmith Posts: 52

    I've restarted Daz, restarted my computer and haven't changed any settings. The only things I altered were a piece of clothing in one scene, but this affects all scenes, and updated the drivers for my ATI card which runs my monitors, the rendering is done on a separate NVidia card.

  • PadonePadone Posts: 3,688

    ok thanks, I was asking because a global value would be useful for quickly setting a whole scene.

  • Richard HaseltineRichard Haseltine Posts: 100,747

    What is the Render Sub Division level for any items that have the property - if it's affecting multiple scenes I would guess you need look only at the common elements, probably the human figures.

  • PadonePadone Posts: 3,688

    From what I can understand, as a first step you could try to render a new scene where there's only standard content that you are sure to be not modified. Just to be absolutely sure, you could reinstall some items in a new content folder and temporarily redirect DS to the new content folder. This way you can test if the issue is related to some "broken" content or not.

    If the issue is not related to content then I guess it is time to check the drivers. Of course I assume that you already checked the DS log to see if there's some warning and to check that the OpenGL and CUDA drivers are reported fine. Since this is the first thing to do.

  • hesperasmithhesperasmith Posts: 52

    It seems from the log that my CUDA card is failing with an out of memory error for some reason and Daz Studio is trying to revert to a CPU only render, but I don't know why that error is being thrown.

    I checked the SubD level for a scene with a single figure and it was at 1, render subD 2. I tried turning it down to 1 and 1, but to no effect.

    I ran a render with nothing but core assets. An unmodified Genesis 3 female figure, Toulouse hair and Darkstorm outfit. This managed to start rendering, but when I checked the log file it was still failing on my card and rendering on CPU only. The strange thing is that it was saying out of memory at approximately 500MB, but the card has 4GB of memory. The card is definitely still working as I have it running one monitor just to be sure and it's doing that fine. 

  • PadonePadone Posts: 3,688

    If you can post the relevant part of the log it would be useful. Also, did you change the final resolution of your renderings ? Since this would affect the memory required by iray. See my post "iray failed to allocate frame buffer".

  • hesperasmithhesperasmith Posts: 52

    From the log file;

    2017-03-27 02:16:00.995 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Initializing local rendering.
    2017-03-27 02:16:01.872 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering with 1 device(s):
    2017-03-27 02:16:01.872 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info :     CUDA device 0 (GeForce GTX 660 Ti)
    2017-03-27 02:16:01.872 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering...
    2017-03-27 02:16:01.965 Iray VERBOSE - module:category(IRAY:RENDER):   1.4   IRAY   rend progr: CUDA device 0 (GeForce GTX 660 Ti): Processing scene...
    2017-03-27 02:16:02.523 Iray VERBOSE - module:category(IRAY:RENDER):   1.2   IRAY   rend stat : Geometry memory consumption: 547.685 MiB (device 0), 0 B (host)
    2017-03-27 02:16:02.646 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.2   IRAY   rend error: OptiX Prime error (Device rtpModelUpdate TL): Unknown error (Function "_rtpModelUpdate" caught exception: Encountered a CUDA error: cudaMalloc(&ptr, size) returned (2): out of memory, [15466617])
    2017-03-27 02:16:02.650 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.4   IRAY   rend error: CUDA device 0 (GeForce GTX 660 Ti): Scene setup failed
    2017-03-27 02:16:02.660 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.4   IRAY   rend error: CUDA device 0 (GeForce GTX 660 Ti): Device failed while rendering
    2017-03-27 02:16:02.712 WARNING: dzneuraymgr.cpp(307): Iray WARNING - module:category(IRAY:RENDER):   1.4   IRAY   rend warn : All available GPUs failed.
    2017-03-27 02:16:02.712 Iray INFO - module:category(IRAY:RENDER):   1.4   IRAY   rend info : Falling back to CPU rendering.
    2017-03-27 02:16:02.713 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.4   IRAY   rend error: All workers failed: aborting render
    2017-03-27 02:16:02.714 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : CPU: using 4 cores for rendering
    2017-03-27 02:16:02.720 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering with 1 device(s):
    2017-03-27 02:16:02.720 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info :     CPU
    2017-03-27 02:16:02.720 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering...
    2017-03-27 02:16:02.721 Iray VERBOSE - module:category(IRAY:RENDER):   1.4   IRAY   rend progr: CPU: Processing scene...
    2017-03-27 02:17:09.657 WARNING: cloud\dzcloudtasknotifier.cpp(178): recv failed errno=10054

  • Richard HaseltineRichard Haseltine Posts: 100,747

    That doesn't immediately look like a memory issue - have you tried updating your nVidia driver?

  • PadonePadone Posts: 3,688

    Also it seems there's a Optix error. If updating the drivers doesn't work you could try disabling the Optix Prime acceleration in the advanced cuda properties tab. With my specific card Optix fails often and I found it is better to keep it disabled. Hope this helps.

  • hesperasmithhesperasmith Posts: 52

    I tried both with and without Optix enabled to no effect, driver update also no change.

    "cudaMalloc(&ptr, size) returned (2): out of memory, [15466617]" - This is a memory allocation error, the card is saying it ran out of memory for some reason.

    I'll try manually removing and reinstalling new drivers later and post again if it works.

  • PadonePadone Posts: 3,688

    personally I'm out of options sorry
    refreshing the content folder, reinstalling the drivers and disabling optix was my last guess

    it could also be useful to compare with another app
    if cuda works fine with max or blender then it's ds having issues
    otherwise it's drivers

  • hesperasmithhesperasmith Posts: 52

    No effect from reinstalling or rolling back drivers. Same for uninstalling and reinstalling studio itself. The only thing left I can think of is rolling back recent Windows updates.

  • ToborTobor Posts: 2,300

    Something is throwing an exception that results in this line:

    Encountered a CUDA error: cudaMalloc(&ptr, size)

    and the OptiX Prime error in the rtpModelUpdate module has cropped up here before, including with cards with much more VRAM. You might do a deep search (using Google or Bing) into the forum to see if anyone has resolved this.

    You certainly don't need Optix acceleration, so it's safe to just switch it off. You might post the relevant portion of the log with Optix off so it doesn't confuse matters. Also make sure that ALL of your render options have been reset to factory defaults. There's a Defaults button right under the Render button in the render panel. Hit that, and then check to make sure everything looks normal.

     

     

  • hesperasmithhesperasmith Posts: 52
    edited April 2017

    Resetting to defaults has allowed me to do some spot rendering, but Iray is still using MUCH more memory than before and this is what seems to be causing the problem. The following log is for a scene that I had previously rendered several times with no problems whatsoever;

    2017-04-06 16:54:13.895 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating geometry.
    2017-04-06 16:54:13.973 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Using built-in ray tracing (1.0.0).
    2017-04-06 16:54:13.973 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Importing geometry.
    2017-04-06 16:54:24.285 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Geometry import (1 object with 20230k triangles, 1 instance yielding 20230k triangles) took 10.312059
    2017-04-06 16:54:24.285 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating materials.
    2017-04-06 16:54:24.982 Iray INFO - module:category(MATCNV:RENDER):   1.0   MATCNV rend info : found 215 textures, 0 lambdas (0 unique)
    2017-04-06 16:54:29.597 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Emitter geometry import (13 light sources with 38199k triangles, 1 instance) took 4.60s
    2017-04-06 16:54:29.597 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating environment.
    2017-04-06 16:54:29.618 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating lens.
    2017-04-06 16:54:29.626 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating lights.
    2017-04-06 16:54:30.568 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating object flags.
    2017-04-06 16:54:30.568 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating caustic portals.
    2017-04-06 16:54:30.568 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Updating decals.
    2017-04-06 16:54:30.571 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Allocating 1 layer frame buffer
    2017-04-06 16:54:30.579 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Using batch scheduling, architectural sampler disabled, caustic sampler disabled
    2017-04-06 16:54:30.579 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Initializing local rendering.
    2017-04-06 16:54:30.579 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Available GPU memory has increased since out-of-memory failure. Re-enable CUDA device 0.
    2017-04-06 16:54:30.592 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering with 1 device(s):
    2017-04-06 16:54:30.599 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info :     CUDA device 0 (GeForce GTX 660 Ti)
    2017-04-06 16:54:30.599 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering...
    2017-04-06 16:54:30.600 Iray VERBOSE - module:category(IRAY:RENDER):   1.3   IRAY   rend progr: CUDA device 0 (GeForce GTX 660 Ti): Processing scene...
    2017-04-06 16:55:05.211 Iray VERBOSE - module:category(IRAY:RENDER):   1.5   IRAY   rend stat : Geometry memory consumption: 544.021 MiB (device 0), 0 B (host)
    2017-04-06 16:55:27.848 WARNING: cloud\dzcloudtasknotifier.cpp(178): peer performed orderly shutdown errno=0
    2017-04-06 16:55:48.683 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.5   IRAY   rend error: Unable to allocate 36000000 bytes from 22783590 bytes of available device memory
    2017-04-06 16:55:48.683 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.3   IRAY   rend error: CUDA device 0 (GeForce GTX 660 Ti): Scene setup failed
    2017-04-06 16:55:48.683 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.3   IRAY   rend error: CUDA device 0 (GeForce GTX 660 Ti): Device failed while rendering
    2017-04-06 16:55:48.690 WARNING: dzneuraymgr.cpp(307): Iray WARNING - module:category(IRAY:RENDER):   1.3   IRAY   rend warn : All available GPUs failed.
    2017-04-06 16:55:48.690 Iray INFO - module:category(IRAY:RENDER):   1.3   IRAY   rend info : Falling back to CPU rendering.
    2017-04-06 16:55:48.690 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER):   1.3   IRAY   rend error: All workers failed: aborting render
    2017-04-06 16:55:48.692 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : CPU: using 4 cores for rendering
    2017-04-06 16:55:48.695 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering with 1 device(s):
    2017-04-06 16:55:48.695 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info :     CPU
    2017-04-06 16:55:48.695 Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering...
    2017-04-06 16:55:48.696 Iray INFO - module:category(IRAY:RENDER):   1.5   IRAY   rend info : Initializing light hierarchy.
    2017-04-06 16:55:48.696 Iray VERBOSE - module:category(IRAY:RENDER):   1.3   IRAY   rend progr: CPU: Processing scene...

    Whilst trying to do this my system memory usage leaps up to around 10GB before becoming unresponsive as the CPU is used to attempt a render, previously only around 2GB was used to render this scene, so this is a lot more than can be accounted for by not using GPU memory. I have changed NOTHING about the scene since it rendered fine, just loaded it up and tried to render with default settings.
     

    I have submitted a support ticket to Daz in the hope that maybe they have a clue.

    Post edited by hesperasmith on
  • SixDsSixDs Posts: 2,384

    "and updated the drivers for my ATI card"

    I would be inclined to suspect something with this, since it is a change that you made prior to experiencing the problem. Running two different drivers simultaneously, while possible according to some users, is generally not considered a good idea as conflicts are bound to occur. If this is the problem, it will not necessarily show up in the DAZ log file. I would check in Windows Control Panel as a first step to ensure that no driver conflicts are identified there. Have you tried rolling back the Ati/AMD driver to what it was before to see if the problem is corrected?

  • hesperasmithhesperasmith Posts: 52
    edited April 2017

    There are no driver conflicts showing, but I'll try that when I get a chance thanks. It's absolutely "possible" according to this user though because I'd been doing it for months with no problems. It shouldn't cause problems since the CUDA code executes through it's own specialised driver which isn't even capable of touching the other card (from my own CUDA coding experience).

    Post edited by hesperasmith on
  • PadonePadone Posts: 3,688

    That's a really hard issue here .. if I was you I'd try starting as little as possible with something that works fine. Reset the log (just delete it). Place a plane and a shere in an empty scene and render. Then look at the log if everything is fine. Reset the log again and see how far you can go before getting errors. At this point I'm afraid it could also be a hardware issue. You can run some card benchmark and see how it scores, for example the Heaven Benchmark, just to see if the drivers work fine.

    Sorry I can't help better .. as I said really hard issue

  • hesperasmithhesperasmith Posts: 52

    I tried rolling back my ATI driver to no effect, I'll try benchmarking the card later but it really seems more like a software problem, as it's requesting about 4x as much memory as it was before and that seems to be what's causing the failure.

  • PadonePadone Posts: 3,688

    you can also download gpu-z to see exactly how much memory is allocated on the card and when ..

  • ToborTobor Posts: 2,300

    Have you mentioned what version of D|S you are using? Do you happen to have more than one version installed, for example 4.8 and a 4.9 beta? If so, and you are experiencing this in one version, have you tried the other? (Obviously, restarting your PC in between.)

  • hesperasmithhesperasmith Posts: 52

    I've only been using 4.9 this whole time, I had 4.6 installed a long time ago but uninstalled it before installing 4.9.

  • ToborTobor Posts: 2,300

    I was hoping you had a previous version installed to check against, but then, you would have tried it by now.

    All I can think of for advancing the troubleshooting is to install a program like GPU-Z. Restart your PC and immediately run the program, keeping it open. Hit the Sensors tab and keep a watch on it as you load D|S, then your scene, then render. This will help you track the actual memory the card is using, whether there are any performance caps being applied, how the memory might be filling up during tge scene building phase, and so on. 

     

  • hesperasmithhesperasmith Posts: 52
    edited April 2017

    I tried to run a spot render with GPU-Z open, I left it for quarter of an hour but it never got so much as 1 iteration done, here are the screencaps.

    As you can see Daz Studio is using a ridiculous amount of RAM, as well as all available CPU time. It used to only use around 2GB of RAM and 50% CPU when rendering the same scene before. This is despite it still using the GPU memory, committed as per the Iray readout in Studio. 

    GPU-ZIray1.png
    385 x 479 - 13K
    TaskManagerIray1.png
    379 x 35 - 2K
    DSRenderprogressIray1.png
    353 x 349 - 12K
    Post edited by hesperasmith on
  • PadonePadone Posts: 3,688
    edited April 2017

    If you have a 3GB card I feel that scene will barely fit. From what I can see it has 2GB textures and 500M geometry so it leaves just 500M for the iray workspace. If the final resolution is not extreme it should be enough, but anyway it's an "edge" case for your card. You could try a smaller scene to see if it works. As I already said I would suggest you to start from small things.

    Anyway as always you should check the log for errors to see why iray reverted to CPU. You didn't post the log. Also, at this stage it is essential that you understand if your card works fine. Did you run the benchmark ?

     

    p.s. On a second check I see that gpu-z reports 2GB vram for your card. While you told your card is 4GB (I knew the 660 ti for 3GB). If that's the case then it's very likely that you have a hardware issue.

    Post edited by Padone on
  • hesperasmithhesperasmith Posts: 52
    edited April 2017

    I posted logs earlier that show the errors coming up.

    The issue is that the scene should not BE that large, it's a single clothed character and some props and I was doing a low res spot render. I have rendered another scene with everything in this one AND another clothed character in much higher resolution on this card multiple times without issues, the problem is WHY Daz Studio is suddenly requesting so much memory now, and not just for this scene.

    Unigine Heaven ran on Ultra settings at around 25-30fps with no issue. Not amazing but fine.

    I apologise for the memory mistake, as I said I have 2 cards and must have been looking at the wrong one.

    Post edited by hesperasmith on
  • PadonePadone Posts: 3,688
    edited April 2017

    Well then. So your card is a GTX 660 TI with 2GB vram. And you are sure that the same scene did require far less memory in previous renders. Then the first thing that comes up my mind is that you could have played around with the texture compression thresholds in the iray panel. Check that they are at default levels, 512 for mid compression and 1024 for high compression. If in some way you disabled compression then the textures will take much more memory on the card. And that would explain everything.

    Also a simple test that we can do is to compare an asset that we both own. For example the G3F. Load the G3F in an empty scene, leave everything at default, lauch a iray render and see in the log how much memory it requires. Disable optix just to avoid it crashing. You should have exacly the values below. Hope this helps.

    Iray VERBOSE - Geometry memory consumption: 13.7058 MiB
    Iray VERBOSE - Texture memory consumption: 354.251 MiB
    Iray VERBOSE - Materials memory consumption: 70.2031 KiB

     

    p.s. so 4GB is the ATI card running your monitors I guess ..

    Post edited by Padone on
Sign In or Register to comment.