Help Understanding Some Errors and Warnings from the Log File

Hi. I've been seeing really slow render times on even small 800x1200 renders. I'm not sure what's changed. I have updated to the latest beta version of Daz Studio that DIM offers, but this has been happening for a while and hasn't changed over the course of the last... five or six updates at least. I used to be able to render simple pose promos in a matter 5-10 minutes. Now they're taking an hour each and it's slowing my workflow to a painful crawl. I need to track down what's going on or what's changed. Or what I'm suddenly doing wrong. I looked at my log file (which I can't understand most of the time) and I notice that there's a warning and an error that come up frequently. I don't know if it's something that's slowing my renders down, but could someone tell me what it means? Here they are:
"2018-01-14 18:01:55.934 WARNING: /src/pluginsource/DzIrayRender/dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.0 IRAY rend error: CUDA device 0: cannot get memory statistics (unknown error)
2018-01-14 18:01:55.934 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : iray photoreal CPU fallback is enabled."
"2018-01-14 18:03:43.090 WARNING: /src/pluginsource/DzIrayRender/dzneuraymgr.cpp(307): Iray WARNING - module:category(IRAY:RENDER): 1.0 IRAY rend warn : The 'progressive_rendering_filtering' scene option is no longer supported."
UPDATE: The last render I tried to run also had this:
"2018-01-14 20:45:25.140 WARNING: /src/pluginsource/DzIrayRender/dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 980 Ti): device cannot be used as it does not support mapped memory (canMapHostMemory)"
Comments
That looks as though it may be a device driver issue, Mary. Can you tell us what driver version you are using for your 980 Ti? If you are using Windows go to Control Panel > System > Device Manager > Display Adapters, expand that and select your 980 Ti and right click. Choose Properties and look under the Driver tab.
Alternatively, look under All Programs > Accessories > System Tools > System Information. Choose Components > Display.
You may also be able to get the info from the Nvidia control panel if you have an icon for it in your taskbar.
Hi SixDs. Thanks for stopping by. :)
Okay, the NVIDIA Driver Manager says I'm using NVIDIA Web Driver: 378.05.25f01 and in parentheses next to that it claims it is up to date. I had it check for updates and it found none. What I've got probably isn't the most up-to-date Windows driver because I'm using a Mac and the Mac drivers always seem to be updated less frequently.
I decided to try rendering a thumbnail image (251x251) with just the default render settings and no extra lights or anything and just a G3 figure with hair on her head. It's still grainy and undercooked after 15 minutes. Once again, I'm seeing things in the log file I've never seen before but I don't know what to do to fix them. Does anything in in this mess mean anything to you (or anyone else who cares to look)? I just grabbed the section that showed the errors rather than copy everything related to the one render. I don't know if that's helpful or not.
This is also coming up fairly frequently:
2018-01-15 01:29:20.744 WARNING: /src/pluginsource/DzIrayRender/dzneuraymgr.cpp(307): Iray WARNING - module:category(IRAY:RENDER): 1.0 IRAY rend warn : Expecting restart after abort. Frequent abort and resume in batch mode can severely reduce performance.
What are the temperatures like on your 980 Ti during rendering? The logs seem to indicate a problem with the device that could be hardware-related.
BTW, I assume that you have also installed the CUDA driver for your MAC OS?
I have an app called Temperature Gauge, which is about the only way I seem able to find out anything about temperatures and this is what it shows after 10 minutes of rendering a scene. Er... I feel like a dunce, but the 980 Ti isn't on the list by name and I'm not even sure it's there at all, unless maybe it's the Northbridge core.
After the render finished, this was the end of the log file:
2018-01-15 03:03:56.624 Total Rendering Time: 26 minutes 16.4 seconds
2018-01-15 03:03:56.718 Loaded image r.png
2018-01-15 03:03:58.722 Loaded image Environment 1.hdr
2018-01-15 03:03:59.461 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Device statistics:
2018-01-15 03:03:59.461 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : CUDA device 0 (GeForce GTX 980 Ti): 0 iterations, 185.299s init, 0.010s render
2018-01-15 03:03:59.461 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : CPU: 186 iterations, 186.747s init, 1337.637s render
2018-01-15 03:03:59.462 WARNING: /src/pluginsource/DzIrayRender/dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 980 Ti): an illegal memory access was encountered (while de-allocating memory)
2018-01-15 03:03:59.462 WARNING: /src/pluginsource/DzIrayRender/dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 980 Ti): an illegal memory access was encountered (while de-allocating memory)
2018-01-15 03:03:59.468 WARNING: /src/pluginsource/DzIrayRender/dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 980 Ti): an illegal memory access was encountered (while de-allocating memory)
Hi, Mary. Unfortunately Temperature Gauge will only display onboard (built-in) motherboard/CPU graphics sensors. It will not monitor addin PCIE graphics cards, which is why the 980 Ti is not displayed. I am not a Mac user, but from what I understand, there is simply nothing available to do this on a Mac. Bummer.
I had asked about a Cuda driver in my previous post. On a Mac you must install two drivers for the 980 Ti to work properly. One driver is the normal video card driver which you have indicated that you have installed. But, in order to use Iray, you must also install a second CUDA-specific driver. Again, have you installed this second driver as well?
Hi again. Ah, that explains it. I did a lot of hunting today and actually did finally find an app I could download and use - but you're right that there's nothing built-in to a Mac OS that will do that. Anyway, if any other Mac folks out there are following along with this, there's an app called iStat Menus that will show you what's going on with non-built-in components like the 980 Ti.
So what iStat shows me is that my CPU is being run in place of the 980 Ti. The Nvidia card is being shown as 'inactive' and I'm still trying to figure out what exactly that means and how to get the card 'active' again.
Oh, that's true. I had forgotten. Yes, I just checked and I do have both drivers installed. The Web Driver and the CUDA driver (version 387.99).
The more I look at this, the more I find, of course. I'm still curious about the errors in the DS log, but I expect sorting out whatever's going on with the graphics card will also sort a lot of those out. My latest discovery is that System Information lists the graphics card twice under PCI Slot 1; once as the Display Controller and once as NVDA,Parent. Does anyone know if it should be listed twice or not? The former says it has drivers installed and the latter says it does not. I expect that is an indication of the problem, but haven't figured it out yet.
Thanks for sticking with me this far, SixDs, I appreciate it. We seem to be straying away from DS errors and into graphics card troubleshooting, which isn't what I expected. My card is only ten months old and hasn't given any trouble until now.
Hi,
a couple of questions first:
- Which version of Cuda is installed?
- Which DAZ Studio version are you running? (There are different Iray builds in different versions)
- What OS X version?
I had this happening (but worse) with a 11 months old GTX-780 6GB, suddenly the machine could not boot with the card inside though, it was broken.
But usually it's the software combination that is the problem.
/ Totte
Hi Totte,
Here you go:
- Cuda version 387.99
- Daz Studio 4.10 (the beta)
- OS 10.12.6
Okay, so today I did a lot of troubleshooting and have managed to prove that it is neither my system or the GPU that's at fault. They are both now working just as they should. So the problem must be in DS, and I'm right back where I started.
Can anyone help me sort out these errors? Or would it better for me to submit a ticket to Daz and see if maybe it's a bug? Here's the log of a quick test render I ran tonight: https://www.dropbox.com/s/rjs6i8pgbdq8fgr/DSLog_01-17-18.zip?dl=0
I see the same errors as I've listed above (toward the middle and bottom of the log), and a couple others I either missed or haven't seen before. There are also a lot at the beginning, but I don't know that those have anything to do with Iray not using or not being able to use the GPU for rendering. Any ideas?
You could use Systempal on your Mac to get a low down of tempreture readings etc... I am sure it's not the answer you're looking for right now but give it a try.
Plarff, thank you for the suggestion. I’ve attached screenshots that show what both Systempal and iStat have to say about temperatures while a render is running.
I’d really love to know what these warnings and errors mean if anyone can explain. I don’t know what they’re telling me so I have no idea how to fix them, or if they can be fixed. Don't misunderstand, I very much appreciate all the suggestions and thoughts so far. I'm only as frantic about this as I am because I'm a PA and I can't finish and submit any of my new products until DS can render the artwork properly again and at a reasonable speed. Maybe I've been spoiled (my old iMac was pretty slow sometimes and leaving that behind was lovely), but 10-20 minutes per thumbnail and hours for every full image is too slow for me to get anything done on time.
This one shows up every four or five lines during a render:
As does this group (What in the world is hpclscruffy.daz3d.com and why is it involved in my render?):
This is what it says at the beginning of a render (I’m particularly curious about the parts I’ve put in bold):
Hey, Mary. I hope that you're not feeling abandoned. It is not unusual for the DAZ Studio log file to contain warnings, and often they can be ignored unless something is not working properly, as is the case with your installation. The important warnings are those related to the video card. For Iray to function correctly using the GPU and CUDA, three things are necessary: a properly functioning and compatible graphics card, properly functioning drivers to allow communication between the hardware and software, and a properly functioning installation of DAZ Studio. All three of these things need to be working in concert. Since the card is relatively new and seems to otherwise to be functioning normally, and you seem to have both of the most up-to-date versions of the necessary drivers installed, that obviously leaves us with DAZ Studio as you have suggested. Why there is a failure to correctly identify the 980 Ti and utilize it, I have no idea. It should be working. The warnings found in the log file normally indicate that either drivers or DAZ Studio needs to be updated, but in your case you seem to have done that. I believe trying to contact DAZ support about the issue is really your only option. I hope that they can sort it out for you. Sorry that I couldn't have been more help. I'm certain that Totte must have been left scratching his head as well.
Mary, I work at a college and we have a number of those "trashcan" Mac Pros that are the same or similar to what I think you have and I asked about your problems. According to our tech people, we've had about a 50% failure rate on those units due to issues with the video cards, related to some combination of hardware problems with the card, the PCIe bridge, or heat issues under load. The machines were all returned to Apple for servicing. It seems clear there's a hardware issue with your Mac as well, unfortunately.
Hi again SixDs,
Oh, no, I'm not feeling abandoned at all. I really do appreciate the help and suggestions everyone has offered so far. Thank you all very much. You're right about the three things having to work together. According to Daz, there's still a certain amount of incompatibility between the version of Iray I've been using in the beta version of Studio and the current Mac CUDA driver. They suggested I go through last year's drivers and roll back until I find one that fixes the problem. I will be trying that later this week when I have a bit of time to test things. However, I also tried going back to Studio 4.9 (which, luckily, I had installed on another computer and forgotten about). 4.9 works pretty well with the most current CUDA driver, though using 4.9 will lose one the ability to use dForce. Updating to 4.10 (not the beta version) works almost as well. It's good enough to allow me to catch up on some projects and has dForce capability. Still not as good as it was before, but it's an improvement. Rendering with GPU works now. The Daz folks said that there will probably be improvements with stability when Iray gets another update. So I will hang back in 4.10 for a while and test out the beta later with older CUDA drivers and wait for the next update.
Hi TomDowd (I lke your avatar, by the way),
Actually, my Mac Pro is pre-trashcan, (I've got the good ol' fashioned silver box style) but yes, that's very true. I read some reports about the performance of the trashcan Macs when they were new and have steered clear of them because they were pretty limited as far as upgradability and customization goes. I didn't realize they were quite that prone to failure, though. Luckily what looked like hardware trouble here turned out not to be. Whew! I'm very happy about that. I'd like to avoid any big replacement or repair costs on my computer for a while, so fingers crossed!
Great news!
If you are using MSi afterburner, or other clocking software it`ll mess up with the driver and DAZ studio will freak out and send everything back to the CPU - no overclocking while rendering and it should be fine! This is at least my experience with this disgusting issue.