Speed up preparing render stage

2»

Comments

  • lilweeplilweep Posts: 2,405
    edited May 2020

    I'm still of the opinion that this was all about being out of VRAM. I just tried pulling half my RAM, down to 16, and started a bunch of renders, all too big for my 2070 but that do fit on my 1080ti, and not a single one took an unusual amount of time to start. So there is definitely some other issue. Also cutting his start time in half drops it to a half hour not to anything acceptable.

    as this thread has progressed, i have changed my approach throughout so am not necessarily comparing apples/apples.  E.g., at some point in thread i adopted strategy to incrementally add stuff to scene so as not to go over 11GB VRAM.  Originally i was just rendering a scene that may well have been over 11GB VRAM.  Maybe thats what caused original 1 hour prep time - i cant really be bothered to reproduce the conditions of that to double check.  So i cannot accurately diagnose problem.  Maybe i will try to render that same scene with current setup and see if there's a difference.

    Since i have been incrementally adding stuff to scene, and maintaing a vram budget, the render prep stage has never gone over 1 hour.  It was much lower- but still very slow -  maybe like 15 min and it would still cause occasional freezes during the render preparation stage at certain points.  After increasing system RAM, the time to render is much faster and does not cause freezing.

     

    Post edited by lilweep on
  • Saxa -- SDSaxa -- SD Posts: 872
    lilweep said:

    Going to 32GB has made a drastic difference in experience for me.  For the particular big scene i was having problems with, the render preparation stage is noticeable faster. 

    yes

    If you do dF hair, or really big scenes, you may really want to consider at some point 64GB.  Drop to CPU (or maybe delays  now with the new DS that am not using yet - still on 4.12.0.86) is much rarer for me.  Partly it's knowing the limits.  But even though didn't want to spend the money, the upgrade to 64GB was totally worth it, cos I spend alot of time with DS/Iray.

    Kenshaw, think you would need Lilweeps scenes setup/specs, and compare your systems to make sure that everything is apple to apple.  Cool that you tested like that.  All i can tell you is my system Ram upgrade was worth every penny having come from 16GB, then 32GB, and now 64GB.  The system/CPU bottlenecks apparently were real.  Saw it on my monitors.  And more mem made it better.  I didn't do it as scientifcally comparing numbers before and after.  Results that made me smile with less delays on this and that spoke for themselves.

    Mainly got the extra RAM for just overall system performance.  Especially the dreaded drop to CPU.  Which that helped with ALOT.  Perhaps the speed up wasn't crazy, but it helped enough.  Overall, if you can afford it, would strongly recommend it.  Have had times where my Ram usage almost hit 50GB during a DS session.  No drop to CPU either with that.  If had more time to be scientifc would test that more.  But only so much time.

     

  • kenshaw011267kenshaw011267 Posts: 3,805
    lilweep said:

    I'm still of the opinion that this was all about being out of VRAM. I just tried pulling half my RAM, down to 16, and started a bunch of renders, all too big for my 2070 but that do fit on my 1080ti, and not a single one took an unusual amount of time to start. So there is definitely some other issue. Also cutting his start time in half drops it to a half hour not to anything acceptable.

    as this thread has progressed, i have changed my approach throughout so am not necessarily comparing apples/apples.  E.g., at some point in thread i adopted strategy to incrementally add stuff to scene so as not to go over 11GB VRAM.  Originally i was just rendering a scene that may well have been over 11GB VRAM.  Maybe thats what caused original 1 hour prep time - i cant really be bothered to reproduce the conditions of that to double check.  So i cannot accurately diagnose problem.  Maybe i will try to render that same scene with current setup and see if there's a difference.

    Since i have been incrementally adding stuff to scene, and maintaing a vram budget, the render prep stage has never gone over 1 hour.  It was much lower- but still very slow -  maybe like 15 min and it would still cause occasional freezes during the render preparation stage at certain points.  After increasing system RAM, the time to render is much faster and does not cause freezing.

     

    That's still too long. I don't think I've ever had one go over 2 minutes.

    I think your problem is in your drive or some sort of software conflict. When was the last time you ran a defrag on your storage drive? Have you checked the drive health recently, the S.M.A.R.T. status? 

  • i53570ki53570k Posts: 212

    It has been established in the forum that the GTX line needs a lot more memory overhead than RTX using RTX ready drivers.  So the 11 GB on 1080Ti and 2080Ti are not apple to apple.  Your 1080Ti has 1-2 GB fewer VRAM than OP's 2080Ti as far as Daz is concerned.  Also the decompression ratio ranges from 1.5-3x so most scenes won't go over DRAM budget in your 2070-1080Ti-16 GB DRAM tests but some extreme usage cases could.  Finally, you have no idea if OP needs multitasking in the same machine, just web browsing alone could take another 2-4GB away and not to mention memory hogs like Photoshop (hence many in the forum insist that 2080 Ti should step up to 64 GB though 32 GB could suffice in a dedicated render machne).

    Just because you could skirt within the limit without hitting a system criplling bottleneck in certain usage cases does not mean that is true in all usage cases. Daz memory math and OP's own empirical test all indicate the issue in question was mainly system RAM shortage and I don't know why you still insist in something else.  If there are other issues as you claimed OP would need to show more symptoms other than what has been addressed.

  • kenshaw011267kenshaw011267 Posts: 3,805
    i53570k said:

    It has been established in the forum that the GTX line needs a lot more memory overhead than RTX using RTX ready drivers.  So the 11 GB on 1080Ti and 2080Ti are not apple to apple.  Your 1080Ti has 1-2 GB fewer VRAM than OP's 2080Ti as far as Daz is concerned.  Also the decompression ratio ranges from 1.5-3x so most scenes won't go over DRAM budget in your 2070-1080Ti-16 GB DRAM tests but some extreme usage cases could.  Finally, you have no idea if OP needs multitasking in the same machine, just web browsing alone could take another 2-4GB away and not to mention memory hogs like Photoshop (hence many in the forum insist that 2080 Ti should step up to 64 GB though 32 GB could suffice in a dedicated render machne).

    Just because you could skirt within the limit without hitting a system criplling bottleneck in certain usage cases does not mean that is true in all usage cases. Daz memory math and OP's own empirical test all indicate the issue in question was mainly system RAM shortage and I don't know why you still insist in something else.  If there are other issues as you claimed OP would need to show more symptoms other than what has been addressed.

    He doubled the RAM and his scenes still take FIFTEEN minutes to start. That is definitive proof that this was not RAM.

  • i53570ki53570k Posts: 212
    edited May 2020
    i53570k said:

    It has been established in the forum that the GTX line needs a lot more memory overhead than RTX using RTX ready drivers.  So the 11 GB on 1080Ti and 2080Ti are not apple to apple.  Your 1080Ti has 1-2 GB fewer VRAM than OP's 2080Ti as far as Daz is concerned.  Also the decompression ratio ranges from 1.5-3x so most scenes won't go over DRAM budget in your 2070-1080Ti-16 GB DRAM tests but some extreme usage cases could.  Finally, you have no idea if OP needs multitasking in the same machine, just web browsing alone could take another 2-4GB away and not to mention memory hogs like Photoshop (hence many in the forum insist that 2080 Ti should step up to 64 GB though 32 GB could suffice in a dedicated render machne).

    Just because you could skirt within the limit without hitting a system criplling bottleneck in certain usage cases does not mean that is true in all usage cases. Daz memory math and OP's own empirical test all indicate the issue in question was mainly system RAM shortage and I don't know why you still insist in something else.  If there are other issues as you claimed OP would need to show more symptoms other than what has been addressed.

    He doubled the RAM and his scenes still take FIFTEEN minutes to start. That is definitive proof that this was not RAM.

    First, that is not what lilweep said.  Lileweep said one hour was with scenes that won't even fit into the 11 GB VRAM budget and more like 15 minutes for scenes within VRAM budget.  Hence the actual symptom WAS 15+ minutes of preparation time AND unable to alt-tab to other tasks.  AFTER the memory upgrade, the preparation time IS now much shorter AND no longer freezes using alt-tab.

     

    Post edited by i53570k on
  • kenshaw011267kenshaw011267 Posts: 3,805

    Much shorter is not less than a minute which is what it should be unless he is still not providing useful info. 

    What he reported was that the amount of time needed was lower once he wasn't rendering on CPU. I've just now run a CPU render using the biggest assets I have, a couple of big mansions each of which takes up something like 8 Gb by itself plus an 8k HDRI and 2 dozen G8's, with just 8 Gb of RAM in the system. Render started in 2 minutes 34 seconds. This is not and never was a RAM issue.

    Of course he couldn't alt-tab when he was rendering on CPU. Even on my 8 core that is not reliable. On a 4 core? 

  • i53570ki53570k Posts: 212

    Much shorter is not less than a minute which is what it should be unless he is still not providing useful info. 

    What he reported was that the amount of time needed was lower once he wasn't rendering on CPU. I've just now run a CPU render using the biggest assets I have, a couple of big mansions each of which takes up something like 8 Gb by itself plus an 8k HDRI and 2 dozen G8's, with just 8 Gb of RAM in the system. Render started in 2 minutes 34 seconds. This is not and never was a RAM issue.

    Of course he couldn't alt-tab when he was rendering on CPU. Even on my 8 core that is not reliable. On a 4 core? 

    I have a very old 4 core i5 and I alt-tab without problem even while rendering as long as the rendering process does not drop to CPU.  But that is not the point.  Again, rendering "prepration" is CPU only and does not involve GPU and most likely a single threaded process, so alt-tab out of Daz3D should not be a CPU issue for anything equal or better than two cores while in preparation.  In Daz3d, CPU rending and GPU-Iray uses memory very diffrently.  GPU-Iray in Daz3D relies on Nvidia driver and AFAIK the way Nvidia designed it as of now for GTX and RTX lines of GPU needs everything loaded in memory, DRAM and VRAM, in one shot to engage.  CPU-only rendering likely does not have such handicap as it is beyond Nvidia control.  Nvidia liikely did it to protect the margins on professional cards.  I am done with this conversation. It's not that hard to intentionally trigger virtual memory disk swap if you know what you are doing yet you just refuse to recognize the elephant in the room.

    @lilweep, I am glad you have seen vast improvements.  A faster , i.e. higher clock CPU (number of cores not as important) and faster SSD in NVMe could also improve prepartion time, but it will likely be seconds rather than minutes of improvement comparing to not having disk swap.  

     

  • kenshaw011267kenshaw011267 Posts: 3,805

    You are flat refusing to understand the point I'm making.

    The problem he was having was happening after the render dropped to CPU and started loading all the cores. Otherwise you are positing multiple failures. The OS would have to be failing, badly, to not be doing disk swapping and time slicing right but only in DS. The CPU or motherboard or GPU hardware would also be failing for this to continue for so long.

    Occam's razor says that is much less likely than the simple fact that the CPU was overloaded because it was trying to render the scene and hadn't yet updated the DS dialog.

  • vonkomicavonkomica Posts: 20

    Much shorter is not less than a minute which is what it should be unless he is still not providing useful info. 

    What he reported was that the amount of time needed was lower once he wasn't rendering on CPU. I've just now run a CPU render using the biggest assets I have, a couple of big mansions each of which takes up something like 8 Gb by itself plus an 8k HDRI and 2 dozen G8's, with just 8 Gb of RAM in the system. Render started in 2 minutes 34 seconds. This is not and never was a RAM issue.

    Of course he couldn't alt-tab when he was rendering on CPU. Even on my 8 core that is not reliable. On a 4 core? 

    I use an A45050 (4x1.0g slow slow slow) with 4g ram and NO gpu to speak of, a 20g SSD Drive on USB setup as dedicated swap. On linux. Everyday I crank out a new Iray render. At this moment I am rendering a 1600x4096 image consisting of the Opulent Spa, G8F, and a ton of custom volumetric steam and water effects consisting of 4096x4096 textures (it took 12min to process the scene and is cooking along at aprox 2min per Iray itteration), surfing this forum, watching my system moniter (showing that I am currently useing 17.4g ram at the moment) and downloading from sharecg. Even with all that going on I get no hiccups. Alt-Tab lag... HA! I have 4 complete desktops I am switching between with NO lag, CPUs parked permently in to 98-100% useage zone. 

    This is a dog of a computer and Daz runs like gangbusters. 

    If you want to speed up your Render Preprocess and just plain scene loading, here are some old tricks.

    Merge all of your scenes layered image assets, That G8F makeup you got on is probably three or four layers of 4096 images. You need exactly one. Boom! Unless you are doing an exterme closeup, that face will be small potatoes in your pic. Resize it accordingly. Measure it by pixil if you don't believe me, all that other data gets wasted.

    If you are not useing dforce (me, hi!) You can try this neat trick. Preprocessing the sceen for render Daz will Increase Geometry resolution bases on your figure's slider settings, Things like hair will gum up the works. I export things like this as objs, and replace the figures in the scene with the imported version. The result is a streight up loadtime and preprocess boost. And it saves on the ram as well, insted of the figuer+morphs+geometry-resoulution-bump at render time, you get a no bull load up of just the geometry you want. I have recently picked up a free script from yashiro_amamiya that will do all this work for you with one click. 

    That sudden computer hang the original poster mentioned, is literally daz trying to rebuild projection maps for hair and clothes and build layered images, this will stop all that rot. 

    That whatever K HDRI... fuggedaboutit. Open it in gimp reduce it to something like say 256x512, turn OFF show dome and render your scene to png format. Use your photo software to add the background AFTER you are done with the render. The smaller hdri will work fine for reflections (the exception would be a mirror surface outdoors, which would probably double you cycle time just because it's a bad idea).

    Consider this: a scene of a character walking on the HDRI beach. Are you rendering the beach, or are you rendering the character? The HDRI is allready as good as it is going to get. Render the character, photoshop the beach back in after. Save yourself a bucket of time.

    And, not for nothing, Mint and Wine are good for Daz. Keeps it happy and limber... that IS importent.

  • BobvanBobvan Posts: 2,652
    edited May 2020

    I agree making fake DOF's can work well as well. I render the BG make the adjustments in ps then import it through the enviro plane and create more effects with blur tool.

    377.png
    1658 x 950 - 2M
    Post edited by Bobvan on
  • TheKDTheKD Posts: 2,677

    It seems weird, even when I was doing my renders on a crap prebuilt dell duo-core with 16GB RAM, the preperation stage never took anywhere near 15 minutes before iterations would start.

  • BobvanBobvan Posts: 2,652
    edited May 2020

    I forgot about exporting and re importing characters as OBJ could work well when one wants many in one scene..

    Post edited by Bobvan on
  • jestmartjestmart Posts: 4,449

    At the rendering stage a mesh is a mesh so importing as an object well have little to no impact.

  • BobvanBobvan Posts: 2,652
    edited May 2020
    jestmart said:

    At the rendering stage a mesh is a mesh so importing as an object well have little to no impact.

    Edit I dont think the point was made for less rendering time, more to make a scene less VRAM intensive..

    Post edited by Bobvan on
  • jestmartjestmart Posts: 4,449

    An object will still have all the surface data so won't free up video ram.

Sign In or Register to comment.