How to fix a damaged .DUF file?

Imago

After a bad HD adventure, some of my old scenes got damaged and won't load properly anymore, giving me an error.

I know that the DUF files are saved using a GZ compression, so I guess a GZ repair tool could be needed... But never found one anywhere, free or commercial!

Someone knows a solution for this problem?

Mattymanx

You can use Daz Studio's Batch Process Tab to compress or decompress DUF files

On the off chance that wont work, just use 7zip

 

Imago

I know how to decompress them.

The problem lies when the file is damaged, I often decompressed some of those files (simply renamed them as *.GZ, extracted the files and then renamed those extracted files as *.DUF) buf even like that, the files still gives me errors on loading.

Basically I need to know how to fix those compressed GZ files.

Richard Haseltine

If the compression is broken - and I ssupect if you check the uncompressed version it will turn to gibberish part way through - then you would approach it the same way you would any broken archive file. There are tools which claim to be able to fix issues, but I have no idea how well any of them works.

Mattymanx

As for fixing it, it depends on whats broken.

What is the error you get.  It could be an issue where older content which is .daz based like optitex or poser content was auto converted to duf by DS on loading (like it always does) and the old converted files aren't in your data folder anymore.

Once its decompressed, try opening it up in Notepad++ to see if you can read stuff there.

 

 

Imago

Mattymanx said:

Once its decompressed, try opening it up in Notepad++ to see if you can read stuff there.

I'll try that, thanks!

Imago

Nothing to do, I can't extract nothing from those damaged files.

The size sems to be right, I can even see the file inside it but I can't extrract.

I really need a GZ files fixer. Anybody knows one? I already tried with some of them but no one handles GZ files.

pctech4ny

I suspect you'll have to venture into using Linux tools, as GZ is primarily used for Linux. Not sure if this would be of any help to point you in right direction:

https://askubuntu.com/questions/826576/how-do-i-recover-corrupted-tar-gz-archives-in-ubuntu

 

Imago

pctech4ny said:

I suspect you'll have to venture into using Linux tools, as GZ is primarily used for Linux. Not sure if this would be of any help to point you in right direction:

https://askubuntu.com/questions/826576/how-do-i-recover-corrupted-tar-gz-archives-in-ubuntu

 

No problems finding a nice distro and using a VM with it. Thanks, I'll try it!

JD_Mortal

If the files were from a "corrupted sector", there will be little that you can do. Same if you "undeleted" them, using a recovery program. Something has damaged the compressed data within them, and it no longer exists. The data will be data from another unrelated file, which will not "decompress", as it has no coralation the the compression library "dictionary". If it does, the output will not be the same. Like Richard says, it will all be jibberish.

The "fix" tools will only try to extract as much data as it can read. Resulting in you seeing that jibberish. Unless it is only a damaged "dictionary", which is contained in the header, and can sort-of be rebuilt, based off the file contents, as long as they are not password locked or otherwise "obscured".

Dare I ask what this "issue" was, which you say has led up to the collection of damaged files?

Seeing the "File list" is not an indication of file integrity. Just like windows, that is just a list of contents, of files within. A mini text database, so you know what files may need to be pulled from other sources, when the archives data has been damaged. If you see those, then you know that the file header or footer is still readable, as text. (I don't recall if the list is saved before or after the data, in GZip.)

Richard Haseltine

JD_Mortal said:

If the files were from a "corrupted sector", there will be little that you can do. Same if you "undeleted" them, using a recovery program. Something has damaged the compressed data within them, and it no longer exists. The data will be data from another unrelated file, which will not "decompress", as it has no coralation the the compression library "dictionary". If it does, the output will not be the same. Like Richard says, it will all be jibberish.

The "fix" tools will only try to extract as much data as it can read. Resulting in you seeing that jibberish. Unless it is only a damaged "dictionary", which is contained in the header, and can sort-of be rebuilt, based off the file contents, as long as they are not password locked or otherwise "obscured".

Dare I ask what this "issue" was, which you say has led up to the collection of damaged files?

Seeing the "File list" is not an indication of file integrity. Just like windows, that is just a list of contents, of files within. A mini text database, so you know what files may need to be pulled from other sources, when the archives data has been damaged. If you see those, then you know that the file header or footer is still readable, as text. (I don't recall if the list is saved before or after the data, in GZip.)

This happens from time-to-time, it seems to be a single byte error (write error or disc corruption I have no idea) - the file usually uncompresses fien until part way through, then breaks from that point on. I sispect that ranbomly adding a byte after the last good one, from 0-255, or chnaging the existing first byte of gibberish, from 0-255, might eventually unlock the rest of the file.

Imago

JD_Mortal said:

If the files were from a "corrupted sector", there will be little that you can do. Same if you "undeleted" them, using a recovery program. Something has damaged the compressed data within them, and it no longer exists. The data will be data from another unrelated file, which will not "decompress", as it has no coralation the the compression library "dictionary". If it does, the output will not be the same. Like Richard says, it will all be jibberish.

The "fix" tools will only try to extract as much data as it can read. Resulting in you seeing that jibberish. Unless it is only a damaged "dictionary", which is contained in the header, and can sort-of be rebuilt, based off the file contents, as long as they are not password locked or otherwise "obscured".

Dare I ask what this "issue" was, which you say has led up to the collection of damaged files?

Seeing the "File list" is not an indication of file integrity. Just like windows, that is just a list of contents, of files within. A mini text database, so you know what files may need to be pulled from other sources, when the archives data has been damaged. If you see those, then you know that the file header or footer is still readable, as text. (I don't recall if the list is saved before or after the data, in GZip.)

The files got damaged after my external backup disk fell while doing some data transfer, the surface got no problems but some of the files got damaged by the sudden power down, I guess the file index got a bad hit.

If the recovery tool allows me to extract as much as it can, it would be a step forward, at the moment I can't extract nothing. Once uncompressed and extractef I can try to fix the wrong parts in the file, since it is basically a text that can be opened and modified with a notepad.

I'm still trying to figure out how to get that Linux stuff, it's harder than I thought. 

stitlown

Mattymanx said:

You can use Daz Studio's Batch Process Tab to compress or decompress DUF files

Thanks for this tip Mattymanx - it was the only way out of tens of approaches, including commercial compressed files recovery utilities, that I tried that at least delivered some recovery of a corrupted file. I managed to get back almost all of the 6 "..._library" sections of the .duf and most of the "scene" section of the .duf, but lost everything after that. I'm still doing the revcovery, but I'm hopeful that a whole lot of puppeteering hs been recovered.

I attempted Richard H's suggestion to modify the bad byte in the corrupted file (if I understood his suggestion) and while that appeared to allow D|S to read further into the corrupted file, when I decompressed it, there was just miles of blanks after the corrupted byte. Admittedly, I only tried one of the 256 characters I could replace the corrupted byte with, but it's a slow cycle to see what has emerged so I've gone back to manual recovery for the rest.

FWIW, 7-zip and every other standard compression tool I tried refused to do anything with the corrupted file. Only a specialist recovery utility appeared to be able to read the file, but the outline it reported looked like rubbish (and I did not pay for the "save" feature to see if it was, indeed, rubbish).

Thanks, again. Batch uncompress wasn't a feature I knew of.

UPDATE - after a tonne of mucking around and making baby steps, a line here, another line there, I went back to gunzip, mastered it and have come to the conclusion that the issue in my file is that the data is simply not there. The file is expected to be "x" long and is in fact somewhat shorter, so the corrution may not be a bad byte but just no data to chew on. Ah well - back to the "recreate / rework" strategy.