Metadata problems - (Reason found)
mpadubrin
Posts: 12
Hello,
I am quite new to DAZ and had a previously working installation until I noticed I got dubious messages about duplicate keys when updating metadata and some of the products I bought not showing in the install pane in DAZ studio. Since I have used DAZ Studio, DAZ Installation manager and DAZ Central at the same time, I decided to come up clean and uninstalled everything I could find associated with DAZ Studio.
Then I tried a "clean" install using DAZ Central only. The application hung up several times, UI freezing while some background processes were running. It took me the whole day yesterday to get to 58% completion when installing with DAZ Central. This morning I wiped that piece of software from my disk. Instead I downloaded the DAZ studio windows installer from my product library and within the hour DAZ Studio was up and running. Then I tried to update my metadata. Surprise ... again I got primary key violations only this time with a different product.
2022-08-14 10:16:34.249 [WARNING] :: ..\..\..\..\..\src\sdksource\cloud\dzcloudtasknotifier.cpp(178): Query (UPDATE "product" SET name = 'dForce Night Dress Outfit for Genesis 8.1 Female', default_name = 'dForce Night Dress Outfit for Genesis 8.1 Female', store_id = 1, token = '84805', artists = 'Arki,Shox-Design,Anna Benjamin,Daz Original', description = '', guid = '6825d37c-4871-43b8-93a1-b8ad966979f8', meta_hash = '1171d39a2baeb8b81012e04ded262814', file_hash = '8e66955e8444f2b4957fb2db80a678a9', thumbnail_path = '', thumbnail_hash = '', last_update = '2022-04-12 06:04:38', date_purchased = '2022-04-12 11:29:14', date_installed = '2022-08-14 08:16:34', release_date = '1970-01-01 00:00:00', group_num = 0, hide = 'f', is_vendor = 't', has_cloud_meta_data = 't', is_installed = 'f', is_cloud_owned = 't', needs_update = 'f', normalized_name = 'dforce night dress outfit for genesis 81 female', normalized_name_first_letter = 'D' WHERE id = 1001) failed with exception: ERROR: duplicate key value violates unique constraint "product_guid_key"
DETAIL: Key (guid)=(6825d37c-4871-43b8-93a1-b8ad966979f8) already exists.
2022-08-14 10:16:34.250 [WARNING] :: ..\..\..\..\..\src\sdksource\cloud\dzcloudtasknotifier.cpp(178): ERROR: duplicate key value violates unique constraint "product_guid_key"
DETAIL: Key (guid)=(6825d37c-4871-43b8-93a1-b8ad966979f8) already exists.
Afterwards I tried to reinstall the content provided with DAZ Studio and got as far as "Secret Garden Shaded Heaven"... The log just states:
2022-08-14 10:49:06.256 [INFO] :: Beginning Install of "Secret Garden Shaded Haven"
2022-08-14 10:49:06.261 [INFO] :: "Secret Garden Shaded Haven": Install Failed.
What am I missing?
Best regards
Michael
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Did you wipe the database? That would not be touched, beyond any uninsalling you had done through all applications that had insalled, by reinstaling the applications. Of coruse if you do do that you will lost any user data (e.g. your own saved files' metadata, not the fiels themselves, and any catgorisation you had done) unless you export the userdata first.
It's possible that there is an error in the product metadata, or was an error that isn't being cleared. I don't have the particular product you mention to check.
I did wipe the complete installation, including the database. The only things left were the backup of my user defined categories I made previously, hoping I might be able to restore them.
After installation DAZ Studio did reload the metadata. I got two messages about duplicate product_keys, which was a bit surprising since I believe I had an empty database. Unfortunately it tells which product it tries to register, but not the product it is in conflict with. So I don't know if this is an software error, or two of the products from daz3d.com have the same guid, which I think would be a bit surprising since I neither tried to create my own product nor did I try to add something from another source than daz3d.com yet.
I am currently in the process of re-installing all products through DAZ Studio. However I noticed that some of the products from my content library are missing in the content manager pane. So I do have to install them differently. Any advice, since I won't let DAZ Central back onto my hard disk?
Have you checked to see if the products that are throwing the duplicate ID error are not actually installed twice? The Filelist for that product states the config file is "DAZ_3D_84805_dForce_Night_Dress_Outfit_for_Genesis_8_1_Female.dsx" - have you checked that a file with this name only exists once amoungst all your mapped library folders?
Yes, I am quite sure, since I wiped all installed content, uninstalled the applications and deleted the content management database. After re-installation of DAZ Studio it connected to postgres:
2022-08-14 02:08:40.864 [INFO] :: Connected to CMS: PostgreSQL 9.3.4, compiled by Visual C++ build 1600, 64-bit 2022-08-14 02:08:41.433 [INFO] :: Creating Cloud Manager... 2022-08-14 02:08:41.758 [WARNING] :: ..\..\..\..\..\src\sdksource\cloud\dzcloudtasknotifier.cpp(178): Query (SELECT * FROM "customer") failed with exception: ERROR: relation "customer" does not exist LINE 1: SELECT * FROM "customer" ^ 2022-08-14 02:08:41.765 [INFO] :: Upgrading CMS 2022-08-14 02:08:41.797 [WARNING] :: ..\..\..\..\..\src\sdksource\cloud\dzcloudtasknotifier.cpp(178): Upgrading database from version 1 to 48and after plugin handling it updated metadata:
Another update reduced the number of fails to 1 (surprise), but since then it has been pretty consistent.
And just to be sure I did scan all my hard drives. No file DAZ_3D_84805_dForce_Night_Dress_Outfit_for_Genesis_8_1_Female.dsx has been found. Still I don't say I did everything correct, only as best as I knew. And of course DAZ Content Manager isn't offering the product for installation, nor does it list it as installed content.
Reason for the collision found!
What I did was to export all metadata stored in the content database, which provided me with a bunch of dsx-files. I scanned them for the offending key and found another product with the same GUID for the metadata. Product sku 84805 "dForce Night Dress Outfit for Genesis 8.1 Females" and 85331 "Black Long Dress Outfit dForce Dress for Genesis 8.1 Females" do have the same GUID for their metadata.
I would have expected DAZ to have a quality procedure in place which prevents such conflicts. That would have spared me much work where I encountered many pitfalls in using the combination of their different installation tools. I guess now I will raise a ticket and see how they do respond.
If it's the Global ID not the token/SKU then I think that is an auto-generated ID. Technically there is a chance of two products being assigned the same ID - an incredibly infinitesimally tiny remote chance but a chance none-the-less. However, if you used a "combination of different installation tools" then that may have increased the chance of that happening. I believe you'll find that most people agree that you should pick one method that works for you and stick with it.
True, I found out that the hard way. Since I don't know how and when those id's are generated it might be due to my use of different tools or a very rare occasion. I decided for myself now to use DAZ Install manager only. Then I am able to seperate download from installation in the content library, which safes bandwith should I mangle the database again.
However, since the collision appears every time now, it appears to be already in the downloaded data, at least the manifest file contains a field MetadataGlobalID with the offending value, which is different from the ProductFileID. The latter might be used in the product library so I can see both products there, but that's only my guess. However, since the metadata GUID is part of the downloaded dsx-file I can't install both packages now independent of any installation method I am using. Without help from technical support I don't see a chance to get rid of the problem.
Well, hopefully you'll get a response back on your ticket that will help.
If you don't want to wait then you *MIGHT* be able to fix it yourself. If you right-click on one of the products in the Smart Content tab and select "Edit Metadata" there is a button next to the GUID to "Generate". I have never clicked that button but presumably it would generate a new unique GUID for that product. Then when you click Accept it will write the relevant files which could fix your issue. I have used this dialog before - although not to change the GUID.
Important Note: for some odd reason this saves the updated dsx & dsa in the first listed base directory you have configured instead of updating the files that already exist. This will not be an issue if you have only one base directory assigned but if you have more than one this may make the situation worse by creating more duplicates. This has reminded me I should submit a ticket about that because it seems like a bug to me.
@SofaCitzen: Thx for your advice. Since I am prone to press randomly buttons in software (why they are there if they are not doing something useful
) I will try that out. However I am still downloading the stuff from DAZ and my connection is slow, so it will take some days. I will update the thread with the results then...
DAZ acknowledged the problem ten days ago, since then I have heard nothing from them. However the number of collisions is increasing. I do have 4 colisisions now.
Btw, no mixing of installation methods anymore, just downloading through DIM and installing manually after successful download. Doesn't stop them from generating duplicates.
It's now twenty-five days after DAS support acknowledged the metaddata collision in the downloaded metadata. And although the person who received my ticket seems friendly and willing enough to help, there was no further reaction from the DAZ Support team responsible for solving the ticket. Even after additional metadata conflicts, there were no requests for more information, or any advice how to solve the problem. If you're from DAZ let me tell you, I think that's not very user friendly.
Although I promised myself to wait for them to come up with a solution I lost my patience today and investigated myself. I think I caused the additional conflicts myself. Previously I made heavy use of custom categories. The default categories just did not seem to be concise to me. In order to save me some work I reimported those metadata into the new metadata db. I might have not had all my content reinstalled, so when I imported my custom categories (no other metadata, since I had wiped them from disk keeping only the user data) the program did not skip those entries it could not associate with an installed product, but created a dummy product, which led to conflicts with the metadata update. It's probably the same situation as if you install the same product twice using different installation methods.
Since I am no DAZ developer, this is only an assumption, which I am trying to verify now. I exported all metadata from the db and then did a reset. That was 4 hours ago, it's still working. When it's done I will reimport the saved metadata minus those of the offending products. If my assumption is correct, this should solve the additional conflicts.
So in the end for the additional conflicts the problem might sit between my chair and my monitor. Although I might claim, the software does make it quite too easy to cause metadata problems (and offers not enough to solve them) .
Now for the initial metadata conflict, this seems to be in the downloaded metadata, and even if I might be responsible for the collision somehow, I can't think of a way to solve it myself. I would have expected it to be quite easy to assign a new metadata id to the metadata associated with my account. And if it is not that easy, an explanation and any advice how to procede might be welcome and soothe my annoyance. Even a statement like cut your losses would be better than no response.
If the conflict is indeed the GUID, why not simply change it? You are aware that there is a button to generate a new one in the DB Editor (from the Products view). Do that, export (if not set to Auto-Export), and Accept the changes. (Ensure no custom categories have been applied to that product yet; if so, remove them). Either way will create a new metadata support file in the runtime/support folder with the new GUID. Re-import the metadata for that product in DIM (right-click on the installed product, Re-Import Metadata). If ever you need to re-install the product, copy that metadata file into the product package, replacing the original so that the changes are included the next time. The GUID is only used a key in the database, nothing else that I can see. Maybe, someday, there might an update that corrects it, which would probably just change the GUID.
Thx for the comment. Of course I got no response until now, but I can be anything but persistent.
Thx for the suggestion, actually I did change the GUID for the not installed product and DIM was happy. Still it's annoying that in case there should be an update for the package I will have to redo the change. OK it won't probably happen at all, but still...
If this problem was good for anything then to give me a good grasp of the structure of their metadata, I actually made installation packages for items from other sources without an installer and so far had no further problems.
Of all the thousands of products I have, that has never happened, so it really is a fluke. Oh, well...
This is happening to me a lot. I currently have 6 files that have conflicting GUIDs which prevents me from even seeing them in my smart content. I've downloaded the DSX files and am now searching through them to find the conflicting GUIDs, but it's file by file of 3k files, so it's going to take me a minute. It would seriously help if this error could either be solved by a popup with like "reassign GUID" or telling us which file it's conflicting with.
The GUID appears once in the manifest inside the zip archive, and once in the metadata support file. Hopefully they are the same...
Which products are involved? If you have the conflict, anyone esle with the same products will have the same GUID conflicts.
https://www.daz3d.com/cc-angelic-accoutrements
GUID: 01c2990f-6003-4fe1-b0e4-9fafca217f14
https://www.daz3d.com/only-dress-for-genesis-8-female
GUID: 6572a82b-1103-4b4e-9912-d781b8297b02
79531 - Reindeer irradiator and poses (no link, it doesn't exist anymore in the store)
GUID: 869ce964-48cb-4179-b65a-5fe334e2bc86
79534 - Sbibb kitten poses and morph (no link, it doesn't exist anymore in the store)
GUID: 5dd744e4-dd3a-45b2-bc31-2c3ff527b07c
https://www.daz3d.com/better-together-poses-for-genesis-8
GUID: 2f0b2a05-1cad-4508-944a-936b52b5e8e0
83933 - Half-timbered house (no link, it doesn't exist anymore in the store)
GUID: 65722092-e1a8-49e8-84eb-7a1f436dde12
Edit to add: I have checked my user installed content and none of them conflict. Will update once I'm done searching the DSX files!
I have five of those, and I can see them all in Smart Content. No conflicts. I searched my Manifest Archive, each GUID appears twice (GlobalID and MetadataGlobalID) in the (single) corresponding manifest, as expected.
And, which .dsx files were you downloading? They come with the package when downloading with DIM, but the GUID in those is different, it is for the zip package itself, not the Product.
I did the tip from mpadubrin, export all metadata stored in the content database.
Edit: Finished searching and have found no GUIDs that match the ones I'm having problems with. I did make a work-around, though, to at least let me access the products.
1. Navigate to your Daz 3D/My DAZ 3D Library/Runtime/Support folder
2. Open the DSX file for whichever product you're having trouble with. Modify the GUID number in that file, I just went up a number on the last digit, then save.
3. With DAZ closed, open the DAZ Script 2 file of the broken product, it will open DAZ and let you import the product with the altered GUID number you just did. I went a step further and edited the metadata to generate a new GUID for it. If you're doing multiple products, you have to close DAZ each time so it's opening for the import. Don't know why but it doesn't work if DAZ is already open.
Now the broken products are imported into DAZ so I can use them, but it hasn't fixed the issue internally.
How do you search your Manifest Archive?
Personally, I use notepad++ for all things text. In this case you can use Find In Files to search for a specific string in a set of files in any folder hierarchy. The Manifest Archive is a folder (I use DIM, configured in the settings) with a record of every product you have installed, and each .dsx file has all sorts of information about the product, including GlobalID and MetadataID.
Exporting all the metadata from Content DB Maintenance only saves the UserData, and that does not have any GUIDs. To search the vendor data, just go to the Runtime/Support folder and search the metadata .dsx files. That or the manifest archive has all the info you need to find the GUID. Together they have the info that gets entered into the database to begin with, and maybe some other behind-the-scenes data, when installing a product.
If you click "Export content metadata" while selecting every smart product, it saves all of their dsx files wherever you specify, and they have GUIDs.
I tried your advice and used notepad++ to search my Manifest folder and it only pulled up the GUIDs from the product files of the ones that are broken. So nothing else has a conflicting GUID that I can find. I searched my custom content folder as well and found nothing. I'm not sure where else I can search. I've had the CC Angelic product for a long time and it has literally never worked right. It's so frustrating.
I really appreciate all your help!
According to the OP, a duplicate GUID will prevent the product from installing, so it would not even be available to not work, so for you they must have installed successfully. Not working in Smart Content is more likely another metadata problem, since that is what drives the whole system.
When you say it doesn't work, what exactly are the symptoms? For example, I can see all assets of the CC Angelic Accoutrements in Smart Content, in several default categories, as expected, with, say, a G8 character in the scene and selected, with Filter by Context checked (along with a bunch of other compatible items from other products, an "angelic" filter helps narrow it down).
Before I did my fix, it was just literally not there. At all. Not in smart content, not in the content library, it just did not exist. There was nothing to install. I wonder if DAZ is mistaking the product itself as a duplicate, not realizing it's... itself?
I should have first asked how you normally install your products. Sounds like you are using Connect, but I use DIM. So, you did not see the product in the Available tab (or any others) in Smart Content?
If using Connect, it will overwrite the metadata you just changed the next time you log in, as I understand it. If using DIM or Central, copy the metadata file back into the zip in case you need to re-install at some point.
Either way, best to write a ticket to get the root problem corrected, if it is just a matter of changing the GlobalID. Not finding the conflict could prove fruitless, though.
(...and I now see that exporting metadata from a Product view will include the GUID, but still only User Data.)
I install with DAZ Central.
Using Central, Studio won't see anything until the product is installed. Connect will know if and when you log in, which is to be avoided if using DIM or Central. It just might "update" the metadata whether you want it to or not.