benchmark load of Genesis 8 Female; asset load times worsen with larger content sets

CO3DR

I have not been able to work in DS for a while because of other obligations. The few times that I have been able to start it up, I had problems loading assets and figured I'd look into it later.

First, I have a huge library of purchased content from both DAZ3D and Renderosity installed into multiple content directories. I have also purchased lots of add-in's.

So, today, I decided to look into the problems. I created three Content Sets: Full, Normal and Minimal. Full has everything, about 20,000 products, if you count them the way Install Manager does. Normal is only ONE of the content directories, just under 9,000 products. Minimal has ONLY the DUF's that I have authored for a particular project: three character presets based on Edgar.

If I use the "Full" Content set, loading Genesis 8 Female takes 270 seconds

If I use the "Normal" Content set, loading Genesis 8 Female takes 40 seconds.

If I use the "Minimal" Content set, loading Edgar takes about 10 seconds.

I've watched the memory usage of DS in Task Manager under these scenarios. Minimal goes by too quick. Normal climbs to about 4GB, and then backs off to a little over 2G once the load is complete. Full climbs to about 9GB, and then reduces, I think to about 6GB (not sure of Full).

I know other discussion threads have covered the single-threaded loading problem, so I thought slow loads were just due to that. But, now I've found this relationship to assets available; maybe I should have bought less.

My system has 64GB of 2400Mhz memory. Disks might be SSD's or spinners, depending on the content directory; Edgar is certainly on an SSD. Normal is a single spinner directory, and Full is some spinner and some SSD, but Genesis 8 Female is on the spinner.i7-6700K CPU @ 4Ghz.

Does anyone know where the elbow is in the curve of when the number of products accessible in your Content Set decimate your asset load times?

PerttiA

"Does anyone know where the elbow is in the curve of when the number of products accessible in your Content Set decimate your asset load times?"

If we are talking about products that are not Genesis 1-8 "characters" and/or morphs, the bigger number of installed products in content library only takes up more space and makes it more difficult to find stuff if not organized properly, but does not effect loading times.

For Genesis 1-8 "characters" and morphs, it's difficult to find which has the biggest effect, the total number of files, number of folders and/or the combined size of the character/morph files, but the more morph files and JCM:s a "character" installs, the bigger effect it will have on the loading times of even the basic version of that base figure. The more morph files you have for a figure (G8F), the more conflicts and errors you will also have, and sometimes it can take up to minutes for DS to get over even one error.

CO3DR

PerttiA said:

For Genesis 1-8 "characters" and morphs, it's difficult to find which has the biggest effect, the total number of files, number of folders and/or the combined size of the character/morph files, but the more morph files and JCM:s a "character" installs, the bigger effect it will have on the loading times of even the basic version of that base figure. The more morph files you have for a figure (G8F), the more conflicts and errors you will also have, and sometimes it can take up to minutes for DS to get over even one error.

Thanks for the comment. I want to make sure I understand this.

Are these morph files that come with the product I am trying to load, or do I pay a penalty for loading every Genesis 1-8 character for each "shape" control I have installed? (These are the products I bought that just alter the DS interface and claim to have no "user facing" files). Like, after loading a G8F, my Shaping tab shows maybe dozens of sliders from "01 Smile" to "Chris_body" with such things as "Grandmotherr HD" and "SSBody01", etc in the list, not including the tabs under "Actor" like "Thorax->RealWorld->BMK->ThoraxShape1BMK"

That is, if we just compare "Normal" and "Full", I am loading the same character, and it should only ever touch the same files. But the load time is dramatically different.

Short of creating specialized Content Sets for different scenarios (like, don't access sci-fi stuff and Old West stuff at the same time), do you have any ideas on how to reduce character load times when you are trying to use a large Content Set?

PerttiA

Almost everything you can buy for G8F is just a morph or set of morphs, they may be called "characters", but in reality they are just a set of morphs, where DS is instructed how much and in which direction the G8F vertices should be moved and some JCM's to control possible issues with posing the morphed figure.

If you have setup your system so that when loading the "Minimal" content set, DS can't see the other content sets, it reads only the morphs it can find for G8F in that set and creates morph dials for them in parameters tab. With the other content sets it can see more morphs, therefore it reads more files and creates more morph dials for them and this takes more time than with the "minimal" set.

Apart from what you have done with separate content sets, there is not much that can be done - More files to read and dials to create = Takes more time.

The morph files for G8F can be found in "...\Data\DAZ 3D\Genesis 8\Female\Morphs\...", whatever DS finds there, will be read when loading a G8F character or a scene that has G8F character in it - If you check the "Properties" of that folder, you will also see the total number of files and folders and how much space they take combined.

CO3DR

PerttiA said:

 

The morph files for G8F can be found in "...\Data\DAZ 3D\Genesis 8\Female\Morphs\...", whatever DS finds there, will be read when loading a G8F character or a scene that has G8F character in it - If you check the "Properties" of that folder, you will also see the total number of files and folders and how much space they take combined.

Thanks. Full has about 25,000 morph files, Normal has about 7,000 files, and Minimal has 10 files.

It sounds like I need a way to organize/trim Content Sets. And, I guess it's a lesson about owning too many morphs - because I know I'm not blending morphs with anywhere near all the flexibility available to me. (Of course, it would be great if DS could defer loading the morph/morphs until I touch the slider; that is, just build the list for the Parameters tab but not hit the disk and consume the memory to load all of them).

(In Full, loading a second G8F character into the same scene, takes virtually the same amount of time; a little less)

Thank you again for the clear explanation.

CO3DR

Just one closing correction/thought.

So, I thought that maybe buying some SSD's would fix this, and I noticed I made an error above when I said "Normal is a single spinner directory"

It turns out "Normal" is a single SSD directory. And, the number of products and morphs within Full that are on a spinner is a very small percentage. Most of my products, except for tutorial videos, are stored on SSD's.

So, I can't fix this just by buying more SSD's. My times would be even worse if the bulk of my content was on spinners.

 

Bobvan

CO3DR said:

I have not been able to work in DS for a while because of other obligations. The few times that I have been able to start it up, I had problems loading assets and figured I'd look into it later.

First, I have a huge library of purchased content from both DAZ3D and Renderosity installed into multiple content directories. I have also purchased lots of add-in's.

So, today, I decided to look into the problems. I created three Content Sets: Full, Normal and Minimal. Full has everything, about 20,000 products, if you count them the way Install Manager does. Normal is only ONE of the content directories, just under 9,000 products. Minimal has ONLY the DUF's that I have authored for a particular project: three character presets based on Edgar.

If I use the "Full" Content set, loading Genesis 8 Female takes 270 seconds

If I use the "Normal" Content set, loading Genesis 8 Female takes 40 seconds.

If I use the "Minimal" Content set, loading Edgar takes about 10 seconds.

I've watched the memory usage of DS in Task Manager under these scenarios. Minimal goes by too quick. Normal climbs to about 4GB, and then backs off to a little over 2G once the load is complete. Full climbs to about 9GB, and then reduces, I think to about 6GB (not sure of Full).

I know other discussion threads have covered the single-threaded loading problem, so I thought slow loads were just due to that. But, now I've found this relationship to assets available; maybe I should have bought less.

My system has 64GB of 2400Mhz memory. Disks might be SSD's or spinners, depending on the content directory; Edgar is certainly on an SSD. Normal is a single spinner directory, and Full is some spinner and some SSD, but Genesis 8 Female is on the spinner.i7-6700K CPU @ 4Ghz.

Does anyone know where the elbow is in the curve of when the number of products accessible in your Content Set decimate your asset load times?

How do you pick normal or minimal amount to load when loading g8f?

PerttiA

CO3DR said:

Just one closing correction/thought.

So, I thought that maybe buying some SSD's would fix this, and I noticed I made an error above when I said "Normal is a single spinner directory"

It turns out "Normal" is a single SSD directory. And, the number of products and morphs within Full that are on a spinner is a very small percentage. Most of my products, except for tutorial videos, are stored on SSD's.

So, I can't fix this just by buying more SSD's. My times would be even worse if the bulk of my content was on spinners.

I wouldn't be that concerned about SSD vs "spinners", it doesn't have that big an effect.

I have used Sysinternals Junction (https://docs.microsoft.com/en-us/sysinternals/downloads/junction) to move older stuff and textures to external USB "spinners" - Daz Studio still thinks they are in their original location - This way I don't have to upgrade the internal SSD's that often and still the stuff that "needs" a faster drive, can have it.

CO3DR

Bobvan said:

CO3DR said:

How do you pick normal or minimal amount to load when loading g8f?

I want to make sure I understand your question. You are either asking (1) What are Content Sets and how do I use them, or (2) Given that I can improve [character] asset load times using Content Sets, how on earth do I use them because I want to browse ALL of my characters before picking the one character I'd like to load in 10 seconds?

I only used Content Sets to test the effect of having a lot of content available when loading a character. I don't know how to practically use Content Sets to actually work around the problem.

CO3DR

PerttiA said:

CO3DR said:

 

I wouldn't be that concerned about SSD vs "spinners", it doesn't have that big an effect.

I have used Sysinternals Junction (https://docs.microsoft.com/en-us/sysinternals/downloads/junction) to move older stuff and textures to external USB "spinners" - Daz Studio still thinks they are in their original location - This way I don't have to upgrade the internal SSD's that often and still the stuff that "needs" a faster drive, can have it.

Thanks! I do know about junctions and use them to solve other problems.