Memory usage questions

droidy001

Hoping someone can answer is simple terms to feed my curiosity. I had 16gb of ram and 8gb of vram. When rendering with gpu ony, Daz would use around 12-13gb with a scene using around 6.5gb of vram. When I started to do other thing with my pc while the render was running I would hit around 15gb of total ram usage. I decided this was the point to add more. I've upgraded to 32gb. Now with the same scene Daz is using 18gb of ram. Where did that extra 5gb of data come from, and what was it doing before? My assumption is that it was swapping a lot of data with my ssd, is this correct? If so that's a lot of extra writes on an ssd. Thanks.

PerttiA

That was the reason for me to upgrade from 32GB to 64GB, if it can't fit everything in RAM, it starts swapping to the HD/SSD.

Responsiveness of the system was improved dramatically also with the upgrade.

TheKD

The more RAM your system has, the more RAM programs will use. It might have been doing a lot of swapping, or not.

TheMysteryIsThePoint

Like @TheKD said, it's your OS using all of the available RAM to optimize the user's experience, most likely I/O caching. For example, your system is probably guessing at what parts of files you're going to request next, and reading them into memory ahead of time. If the OS actually needs that memory for something more important, it simply gives it away. From the OS's standpoint, unused memory is memory that could be contributing to a better running system, but isn't. What you are observing is a good thing.

PerttiA

TheMysteryIsThePoint said:

Like @TheKD said, it's your OS using all of the available RAM to optimize the user's experience, most likely I/O caching. For example, your system is probably guessing at what parts of files you're going to request next, and reading them into memory ahead of time. If the OS actually needs that memory for something more important, it simply gives it away. From the OS's standpoint, unused memory is memory that could be contributing to a better running system, but isn't. What you are observing is a good thing.

Not true. It may seem that way if you don't have enough - Then the OS starts swapping to HD/SSD, which you can notice for example by opening a program that was minimised while doing rendering in DS. It takes a while for the OS to organise the available memory and bring the minimized program in front, even then it may take awhile before it shows the content correctly.

Previously I had 32GB and a 4GB GTX960, GPU-Z showed both RAM and VRAM close to max when rendering a scene that took 10.5 hour to finish.

Now when I have 64GB and a 8GB RTX 2070 super, GPU-Z shows around 42GB RAM used and around 5.8GB VRAM used, the same scene renders in 25 minutes and I can do other things while waiting, without the system being sluggish.

droidy001

The extra 5gb is being used by Daz. I'm sure the OS will also be using more. I haven't had chance to look in detail. It just sparked my curiosity as to where this data had come from and how the system was handling it previously. It is also now leaving me with enough spare that I'll try opening another instance of daz and see if I have enough headroom to work on my next scene, while the render is in processing.

TheMysteryIsThePoint

PerttiA said:

TheMysteryIsThePoint said:

Like @TheKD said, it's your OS using all of the available RAM to optimize the user's experience, most likely I/O caching. For example, your system is probably guessing at what parts of files you're going to request next, and reading them into memory ahead of time. If the OS actually needs that memory for something more important, it simply gives it away. From the OS's standpoint, unused memory is memory that could be contributing to a better running system, but isn't. What you are observing is a good thing.

Not true. It may seem that way if you don't have enough - Then the OS starts swapping to HD/SSD, which you can notice for example by opening a program that was minimised while doing rendering in DS. It takes a while for the OS to organise the available memory and bring the minimized program in front, even then it may take awhile before it shows the content correctly.

Previously I had 32GB and a 4GB GTX960, GPU-Z showed both RAM and VRAM close to max when rendering a scene that took 10.5 hour to finish.

Now when I have 64GB and a 8GB RTX 2070 super, GPU-Z shows around 42GB RAM used and around 5.8GB VRAM used, the same scene renders in 25 minutes and I can do other things while waiting, without the system being sluggish.

It is absolutely true. That is just how OSes work.

The OP's increased usage is the OS using all the available RAM for elective things that can cheaply given up, i.e. there is no need to swap anything because the data was not precious in the first place, like file read-ahead. Swapping is what happens when there isn't even enough RAM for mandatory things, like page faults that indicate a program tried to access data in its virtual address space that was not resident and needs to be swapped back in. What you are saying is true, but that is a separate phenomenon from what OP was observing.

PerttiA

TheMysteryIsThePoint said:

PerttiA said:

TheMysteryIsThePoint said:

Like @TheKD said, it's your OS using all of the available RAM to optimize the user's experience, most likely I/O caching. For example, your system is probably guessing at what parts of files you're going to request next, and reading them into memory ahead of time. If the OS actually needs that memory for something more important, it simply gives it away. From the OS's standpoint, unused memory is memory that could be contributing to a better running system, but isn't. What you are observing is a good thing.

Not true. It may seem that way if you don't have enough - Then the OS starts swapping to HD/SSD, which you can notice for example by opening a program that was minimised while doing rendering in DS. It takes a while for the OS to organise the available memory and bring the minimized program in front, even then it may take awhile before it shows the content correctly.

Previously I had 32GB and a 4GB GTX960, GPU-Z showed both RAM and VRAM close to max when rendering a scene that took 10.5 hour to finish.

Now when I have 64GB and a 8GB RTX 2070 super, GPU-Z shows around 42GB RAM used and around 5.8GB VRAM used, the same scene renders in 25 minutes and I can do other things while waiting, without the system being sluggish.

It is absolutely true. That is just how OSes work.

The OP's increased usage is the OS using all the available RAM for elective things that can cheaply given up, i.e. there is no need to swap anything because the data was not precious in the first place, like file read-ahead. Swapping is what happens when there isn't even enough RAM for mandatory things, like page faults that indicate a program tried to access data in its virtual address space that was not resident and needs to be swapped back in. What you are saying is true, but that is a separate phenomenon from what OP was observing.

How come my system is currently using only 200MB of RAM (out of 64GB installed), if the OS supposedly uses all the memory it can find?

TheMysteryIsThePoint

PerttiA said:

TheMysteryIsThePoint said:

PerttiA said:

TheMysteryIsThePoint said:

Like @TheKD said, it's your OS using all of the available RAM to optimize the user's experience, most likely I/O caching. For example, your system is probably guessing at what parts of files you're going to request next, and reading them into memory ahead of time. If the OS actually needs that memory for something more important, it simply gives it away. From the OS's standpoint, unused memory is memory that could be contributing to a better running system, but isn't. What you are observing is a good thing.

Not true. It may seem that way if you don't have enough - Then the OS starts swapping to HD/SSD, which you can notice for example by opening a program that was minimised while doing rendering in DS. It takes a while for the OS to organise the available memory and bring the minimized program in front, even then it may take awhile before it shows the content correctly.

Previously I had 32GB and a 4GB GTX960, GPU-Z showed both RAM and VRAM close to max when rendering a scene that took 10.5 hour to finish.

Now when I have 64GB and a 8GB RTX 2070 super, GPU-Z shows around 42GB RAM used and around 5.8GB VRAM used, the same scene renders in 25 minutes and I can do other things while waiting, without the system being sluggish.

It is absolutely true. That is just how OSes work.

The OP's increased usage is the OS using all the available RAM for elective things that can cheaply given up, i.e. there is no need to swap anything because the data was not precious in the first place, like file read-ahead. Swapping is what happens when there isn't even enough RAM for mandatory things, like page faults that indicate a program tried to access data in its virtual address space that was not resident and needs to be swapped back in. What you are saying is true, but that is a separate phenomenon from what OP was observing.

How come my system is currently using only 200MB of RAM (out of 64GB installed), if the OS supposedly uses all the memory it can find?

Like the sceduler that chooses the next process to run, the OS uses a heuristic to decide what it should do. There are instances when that heuristic can decide that there is nothing useful if can do. If your system is only using 200 megs, you haven't given it much to make an educated guess on how to help you.

I'm not at all sure why this is so hard for you to believe? But I think at this point I should ask if you have any experience in OS implementation, written a device driver, even debated with colleagues trying to prove that their favorite kernel is cooler than your favorite kernel, or are you sitting in front of a Windows computer, looking at the task manager while it tries to explain to you something very complicated with a single number?

 

PerttiA

TheMysteryIsThePoint said:

PerttiA said:

TheMysteryIsThePoint said:

PerttiA said:

TheMysteryIsThePoint said:

Like @TheKD said, it's your OS using all of the available RAM to optimize the user's experience, most likely I/O caching. For example, your system is probably guessing at what parts of files you're going to request next, and reading them into memory ahead of time. If the OS actually needs that memory for something more important, it simply gives it away. From the OS's standpoint, unused memory is memory that could be contributing to a better running system, but isn't. What you are observing is a good thing.

Not true. It may seem that way if you don't have enough - Then the OS starts swapping to HD/SSD, which you can notice for example by opening a program that was minimised while doing rendering in DS. It takes a while for the OS to organise the available memory and bring the minimized program in front, even then it may take awhile before it shows the content correctly.

Previously I had 32GB and a 4GB GTX960, GPU-Z showed both RAM and VRAM close to max when rendering a scene that took 10.5 hour to finish.

Now when I have 64GB and a 8GB RTX 2070 super, GPU-Z shows around 42GB RAM used and around 5.8GB VRAM used, the same scene renders in 25 minutes and I can do other things while waiting, without the system being sluggish.

It is absolutely true. That is just how OSes work.

The OP's increased usage is the OS using all the available RAM for elective things that can cheaply given up, i.e. there is no need to swap anything because the data was not precious in the first place, like file read-ahead. Swapping is what happens when there isn't even enough RAM for mandatory things, like page faults that indicate a program tried to access data in its virtual address space that was not resident and needs to be swapped back in. What you are saying is true, but that is a separate phenomenon from what OP was observing.

How come my system is currently using only 200MB of RAM (out of 64GB installed), if the OS supposedly uses all the memory it can find?

Like the sceduler that chooses the next process to run, the OS uses a heuristic to decide what it should do. There are instances when that heuristic can decide that there is nothing useful if can do. If your system is only using 200 megs, you haven't given it much to make an educated guess on how to help you.

I'm not at all sure why this is so hard for you to believe? But I think at this point I should ask if you have any experience in OS implementation, written a device driver, even debated with colleagues trying to prove that their favorite kernel is cooler than your favorite kernel, or are you sitting in front of a Windows computer, looking at the task manager while it tries to explain to you something very complicated with a single number?

 

I'm just a newbee with these computer things, since I started only 33 years ago.

kenshaw011267

Doesn't really matter when you started if you don't understand that this is how these things work.

An application starts up and consumes an amount of RAM based on how much the program it self uses, some programs dynamically load libraries and the OS tries its best to track that and allow for it but that isn't very easy so there has to be some slop added in, plus a big gob of RAM for the programs stack(s) and heaps. The OS knows to err up is better so it tracks over time both how much each program actually uses and hom many programs you use at the same time and tries to strike the best balance.

Why is it then using so little of yoru RAM? Becuase your system is idling and the programs that are running don't use any RAM to speak of.

PerttiA

kenshaw011267 said:

Doesn't really matter when you started if you don't understand that this is how these things work.

An application starts up and consumes an amount of RAM based on how much the program it self uses, some programs dynamically load libraries and the OS tries its best to track that and allow for it but that isn't very easy so there has to be some slop added in, plus a big gob of RAM for the programs stack(s) and heaps. The OS knows to err up is better so it tracks over time both how much each program actually uses and hom many programs you use at the same time and tries to strike the best balance.

Why is it then using so little of yoru RAM? Becuase your system is idling and the programs that are running don't use any RAM to speak of.

The claim was "it's your OS using all of the available RAM to optimize the user's experience", now you are talking about applications.

TheMysteryIsThePoint

PerttiA said:

TheMysteryIsThePoint said:

PerttiA said:

TheMysteryIsThePoint said:

PerttiA said:

TheMysteryIsThePoint said:

Like @TheKD said, it's your OS using all of the available RAM to optimize the user's experience, most likely I/O caching. For example, your system is probably guessing at what parts of files you're going to request next, and reading them into memory ahead of time. If the OS actually needs that memory for something more important, it simply gives it away. From the OS's standpoint, unused memory is memory that could be contributing to a better running system, but isn't. What you are observing is a good thing.

Not true. It may seem that way if you don't have enough - Then the OS starts swapping to HD/SSD, which you can notice for example by opening a program that was minimised while doing rendering in DS. It takes a while for the OS to organise the available memory and bring the minimized program in front, even then it may take awhile before it shows the content correctly.

Previously I had 32GB and a 4GB GTX960, GPU-Z showed both RAM and VRAM close to max when rendering a scene that took 10.5 hour to finish.

Now when I have 64GB and a 8GB RTX 2070 super, GPU-Z shows around 42GB RAM used and around 5.8GB VRAM used, the same scene renders in 25 minutes and I can do other things while waiting, without the system being sluggish.

It is absolutely true. That is just how OSes work.

The OP's increased usage is the OS using all the available RAM for elective things that can cheaply given up, i.e. there is no need to swap anything because the data was not precious in the first place, like file read-ahead. Swapping is what happens when there isn't even enough RAM for mandatory things, like page faults that indicate a program tried to access data in its virtual address space that was not resident and needs to be swapped back in. What you are saying is true, but that is a separate phenomenon from what OP was observing.

How come my system is currently using only 200MB of RAM (out of 64GB installed), if the OS supposedly uses all the memory it can find?

Like the sceduler that chooses the next process to run, the OS uses a heuristic to decide what it should do. There are instances when that heuristic can decide that there is nothing useful if can do. If your system is only using 200 megs, you haven't given it much to make an educated guess on how to help you.

I'm not at all sure why this is so hard for you to believe? But I think at this point I should ask if you have any experience in OS implementation, written a device driver, even debated with colleagues trying to prove that their favorite kernel is cooler than your favorite kernel, or are you sitting in front of a Windows computer, looking at the task manager while it tries to explain to you something very complicated with a single number?

 

I'm just a newbee with these computer things, since I started only 33 years ago.

That was an unsuccessful appeal to athority. You are still factually incorrect. The concept of a cache being prefilled so that data that is expensive to get is already present when it is needed exists from the very lowest levels where a CPU will speculatively execute instructions just to toss the results if it mispredicts a branch, to read operations that read and retain the data all the way up to the block size because the app is so likely to request the subsequent data as well, to web browsers that prefetch the pages that the page you're looking at links to, because you are likely to click on one of those links, even though you might not. I'm not at all sure why you refuse to accept that this is just how computers work.

kenshaw011267

PerttiA said:

kenshaw011267 said:

Doesn't really matter when you started if you don't understand that this is how these things work.

An application starts up and consumes an amount of RAM based on how much the program it self uses, some programs dynamically load libraries and the OS tries its best to track that and allow for it but that isn't very easy so there has to be some slop added in, plus a big gob of RAM for the programs stack(s) and heaps. The OS knows to err up is better so it tracks over time both how much each program actually uses and hom many programs you use at the same time and tries to strike the best balance.

Why is it then using so little of yoru RAM? Becuase your system is idling and the programs that are running don't use any RAM to speak of.

The claim was "it's your OS using all of the available RAM to optimize the user's experience", now you are talking about applications.

!

The OS controls how much RAM each application is allowed to access, in most circumstances. The application asks and the application responds.

When an application starts, or spins off a thread, the app asks for a stack of a certain size and the application either gets it or not, it almost always does but it may take a bit of disk shuffling and virtual RAM stuff. Then when the application asks for a new pointer, or reference or whatever some other system calls that stuff, the OS checks if there is currently room in the application's currently assigned heap. If so it returns a pointer. If not the OS tries to grab a new block of unassigned RAM. If so new RA M and you're golden. If not the OS sees if there is room in the page file or room for the page file to grow shuffles some stuff to disk and again you get some more RAM and again you're golden but that call took, relatively, forever.

So the OS is doing all of the assigning of RAM for applications to use. The applications ask for RAM from the OS but it is pretty pointless to track what is going on in blocks allocated to an application, the OS won't let any other application use them. That is what one cause of access violations are.