What are the memory limits on String manipulation ?

jstephens

Hello everyone!

I'm actually trying to make an export script and I'm facing some limits of Daz DSA Scripts.

Basically the format need to be a single JSON containing various properties and objects, so I retrieve geometry, materials, etc into a single object then "JSON.stringify" it.

 

The string holding the JSON is actually around 4 Go, that is what "String.length" is reporting : 

    2022-11-09 01:56:18.623 [DEBUG] :: str.length: 4189042444

 

To save this huge string into a file, i've tried various methods :

1) A single DZFile; but "fwrite" report "0" and the file is empty too, so no contents have been written :

    var fwrite = fp.write(str);

 

2) Chunk it into multiple 32Kb parts ; but again, fp.write report "0" and the file is still empty :

    var str = "// big json //";    var size = 32 * 1000;    var numChunks = Math.ceil(str.length / size);    var fp = new DzFile(filepath);    fp.open(DzFile.WriteOnly);    for (var i = 0, o = 0; i < numChunks; ++i, o += size) {        var chunk = str.substr(o, size);        var fwrite = fp.write(chunk);    }

 

In this case, RAM is not a problem and i don't care about optimisation, I just want to save this huge string into a file

Is there a limitation in how DAZ handles Strings or file writes ? I'm not sure i've found such information in the documentations.

 

Thanks for your help!

Richard Haseltine

have yopu checked that the code works with a shorter string?

jstephens

Richard Haseltine said:

have yopu checked that the code works with a shorter string?

Thanks for your reply!

Yes, the "String.substr" function (used chunk the string) run fine with a string of ~15Mo, ~60Mo, ~250Mo or ~1.2Go but apparently not 4Go.

That's why I were asking if maybe there were some king of limitation ? Like how the Javascript stack memory is handled, for example.

Richard Haseltine

It could be, Zip certainly has a limit (can't recall if it's 2 or 4) so it may be determined by the same factors - 32 bit count, for example - but I am not certain.

jstephens

Richard Haseltine said:

It could be, Zip certainly has a limit (can't recall if it's 2 or 4) so it may be determined by the same factors - 32 bit count, for example - but I am not certain.

It surprises me that the string can hold 4Go of data, but some functions just does not seems to work correctly at that size.

You may be right that there is some kind of limitation for some functions in particular.

After some trials, I saw that "String.charAt" worked for the 4Go string, so I implemented a similar "substr". It is slow, but it work!