Unity 5.3 scripts
Just a heads up - after a week of pretty solid reverse engineering and scripting, my scripts are now working with Unity 5.3, and I've updated The Long Dark to prove it ;-) This version of Unity changed the shader format in several ways - DX9 shaders are no longer stripped, but DX11 shaders are still stripped as they were before. For DX9 this means that all the hashes will have changed, breaking the fixes for any games that update to this version (if this happens, it may be worth trying UseAlternateCRC to see if it can get the old fix working before spending time updating it). For DX11 the hashes won't necessarily change as a result of updating to Unity 5.3, though of course they may change for the usual reasons they do on updates (e.g. the directional lighting vertex shader hash has changed). For DX11 the headers need to be extracted from the Unity asset files as before, and I have updated my scripts to enable this. For DX9 I would still recommend extracting the Unity headers even though the DX headers are no longer stripped - the Unity headers contain more information than is found in the DX shader headers - most importantly, they indicate which shaders are used as shadow casters (and therefore should be avoided when copying matrices). The second change they made was to their shader file format, which broke my scripts. Traditionally Unity has stored shaders in the asset files in either plain text (DX9, OpenGL) or using a trivial encoding scheme (DX11) embedded inside some plain-text data structures with info about the shaders - this is the info my scripts extract and attach to the shaders. This plain text format still exists in Unity 5.3, but almost everything of interest in it has been removed and moved to a new compressed binary section of the shader file, which I've spent most of the last week deciphering and adapting my scripts to process. There's still a few bytes that I have not deciphered, but they don't seem very important. unity_asset_extractor will now extract .shader.decompressed files (along with the .shader files that it used to extract) from the Unity .asset files (which includes the files in the resources folder despite the missing extension). extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand. I've updated my Unity 5 DX11 template and added an autofix53.sh script to it (later I might combine that into autofix.sh). The DX9 template has not been updated, but it should only be a few small tweaks to the script to make it work (e.g. using extract_unity53_shaders instead of extract_unity_shaders). Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them. As usual, all the scripts can be found in my 3d-fixes repository linked from my signature.
Just a heads up - after a week of pretty solid reverse engineering and scripting, my scripts are now working with Unity 5.3, and I've updated The Long Dark to prove it ;-)

This version of Unity changed the shader format in several ways - DX9 shaders are no longer stripped, but DX11 shaders are still stripped as they were before.

For DX9 this means that all the hashes will have changed, breaking the fixes for any games that update to this version (if this happens, it may be worth trying UseAlternateCRC to see if it can get the old fix working before spending time updating it).

For DX11 the hashes won't necessarily change as a result of updating to Unity 5.3, though of course they may change for the usual reasons they do on updates (e.g. the directional lighting vertex shader hash has changed).

For DX11 the headers need to be extracted from the Unity asset files as before, and I have updated my scripts to enable this. For DX9 I would still recommend extracting the Unity headers even though the DX headers are no longer stripped - the Unity headers contain more information than is found in the DX shader headers - most importantly, they indicate which shaders are used as shadow casters (and therefore should be avoided when copying matrices).



The second change they made was to their shader file format, which broke my scripts. Traditionally Unity has stored shaders in the asset files in either plain text (DX9, OpenGL) or using a trivial encoding scheme (DX11) embedded inside some plain-text data structures with info about the shaders - this is the info my scripts extract and attach to the shaders. This plain text format still exists in Unity 5.3, but almost everything of interest in it has been removed and moved to a new compressed binary section of the shader file, which I've spent most of the last week deciphering and adapting my scripts to process. There's still a few bytes that I have not deciphered, but they don't seem very important.


unity_asset_extractor will now extract .shader.decompressed files (along with the .shader files that it used to extract) from the Unity .asset files (which includes the files in the resources folder despite the missing extension).

extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand.

I've updated my Unity 5 DX11 template and added an autofix53.sh script to it (later I might combine that into autofix.sh). The DX9 template has not been updated, but it should only be a few small tweaks to the script to make it work (e.g. using extract_unity53_shaders instead of extract_unity_shaders).


Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them.


As usual, all the scripts can be found in my 3d-fixes repository linked from my signature.

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD
Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword
Contact & PayPal: darkstarsword@gmail.com

#1
Posted 04/30/2016 07:08 PM   
Thanks for this. Out of necessity I've had to learn to build macros to implement offline fixes to hundreds of shaders, and I always find that an interesting challenge, but that's a drop in the bucket compared to these awesome tools you provide. Would love to refresh my decade old learning in programming to be able to implement more advanced scripts and tools like you do (been toying with the idea of going back to school, but kinda hard to figure out how to make that work). Anyway, thanks again. I'll go ahead and test this out on one of the games that updated to 5.3 that broke a fix I was working on with your previous version of the tool and confirm it's success.
Thanks for this. Out of necessity I've had to learn to build macros to implement offline fixes to hundreds of shaders, and I always find that an interesting challenge, but that's a drop in the bucket compared to these awesome tools you provide. Would love to refresh my decade old learning in programming to be able to implement more advanced scripts and tools like you do (been toying with the idea of going back to school, but kinda hard to figure out how to make that work). Anyway, thanks again. I'll go ahead and test this out on one of the games that updated to 5.3 that broke a fix I was working on with your previous version of the tool and confirm it's success.

3D Gaming Rig: CPU: i5 6600K | Mobo: Asus Maximus Hero VIII | RAM: Corsair Dominator 16GB | GPU: GTX 1080 Ti | 2xSSDs for OS and Apps | PSU: Seasonic X-1250 M2| Case: Corsair C70 | Cooling: Hyper 212+ (Air) | Displays: BenQ XL2420TX & BenQ HT1075 | OS: Windows 10 Pro

Like my fixes? Dontations can be made to: www.paypal.me/DShanz or rshannonca@gmail.com
Like electronic music? Check out: www.soundcloud.com/dj-ryan-king

#2
Posted 04/30/2016 07:54 PM   
Great work DSS, Thanks alot for your dedication, really appreciate it. I have bought NightCry recently wich is Unity 5.3.3, so I'm very exited about your update.. I'll look into it next week, when I hopefully get some sparetime :)
Great work DSS, Thanks alot for your dedication, really appreciate it.

I have bought NightCry recently wich is Unity 5.3.3, so I'm very exited about your update..
I'll look into it next week, when I hopefully get some sparetime :)

#3
Posted 04/30/2016 08:09 PM   
Awesome, this is great news. TY For those like me that hadn't heard of The Long Dark or NightCry, here's the Steam links. http://store.steampowered.com/app/305620/ https://www.youtube.com/watch?v=dmAYWj58gKI http://store.steampowered.com/app/427660/ https://www.youtube.com/watch?v=vnseGvK-9aY
Awesome, this is great news. TY

For those like me that hadn't heard of The Long Dark or NightCry, here's the Steam links.


http://store.steampowered.com/app/305620/



http://store.steampowered.com/app/427660/

#4
Posted 05/01/2016 12:39 AM   
Thank you so much for the updates, DSS! [quote="DarkStarSword"]extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand.[/quote] Where can I locate "cmd_Decompiler"? I did a search but I was unable to find it. If I try to use "extract_unity53_shaders" without it, I only get Dx9 & Dx11 headers. Edit: I think I found it but I'm not quite sure how to use it-- https://github.com/bo3b/3Dmigoto/tree/master/HLSLDecompiler/cmd_Decompiler I suppose Visual C++ is needed?
Thank you so much for the updates, DSS!

DarkStarSword said:extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand.


Where can I locate "cmd_Decompiler"? I did a search but I was unable to find it. If I try to use "extract_unity53_shaders" without it, I only get Dx9 & Dx11 headers.

Edit: I think I found it but I'm not quite sure how to use it-- https://github.com/bo3b/3Dmigoto/tree/master/HLSLDecompiler/cmd_Decompiler

I suppose Visual C++ is needed?

#5
Posted 05/12/2016 06:58 AM   
You can grab it from the 3DMigoto releases page here: https://github.com/bo3b/3Dmigoto/releases It will be updated anytime we make an improvement to the 3DMigoto decompiler, assembler or disassembler. Let me know if you have any trouble with the scripts - I expect there might still be a few combinations in some games they aren't handling yet, and I've made them err on the side of failing if they see something they aren't sure about rather than risk missing something.
You can grab it from the 3DMigoto releases page here:
https://github.com/bo3b/3Dmigoto/releases

It will be updated anytime we make an improvement to the 3DMigoto decompiler, assembler or disassembler.


Let me know if you have any trouble with the scripts - I expect there might still be a few combinations in some games they aren't handling yet, and I've made them err on the side of failing if they see something they aren't sure about rather than risk missing something.

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD
Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword
Contact & PayPal: darkstarsword@gmail.com

#6
Posted 05/12/2016 09:53 AM   
Cool. I got it working now. Thanks Again! [quote="DarkStarSword"]Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them.[/quote] So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case. https://www.dropbox.com/s/zh7gzjqyftn29l4/DepthOfFieldDX11.shader.decompressed?dl=0
Cool. I got it working now. Thanks Again!

DarkStarSword said:Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them.

So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case.

https://www.dropbox.com/s/zh7gzjqyftn29l4/DepthOfFieldDX11.shader.decompressed?dl=0

#7
Posted 05/15/2016 05:42 AM   
BTW, Unity has Single Pass stereo in their Engine now. I was curious if you found any thing interesting, provided that you had a chance to give it a once over. Edit: here's the post where I first saw mention of it after googling single pass stereo as soon as they mentioned it in the pascal live event feed. robinb-u3d Jan 14, 2016 Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye. If you look at your current VR project in the Unity profiler you should see that on the Main Thread each camera ("Camera.Render") that is enabled for VR will generate two sets of "Drawing" markers one for each eye. The second of those markers should go away with the optimizations enabled so reducing the VR "Drawing" time by 50%. We are also seeing a large reduction in the time spent on the GfxWorker Render Thread as a result of this optimizations as well. Currently we are targeting Oculus Rift and PlayStation VR with this round of optimizations, we'll be looking into optimizations for GearVR a little further down the line. Hope this is of some use. Robin http://forum.unity3d.com/threads/upcoming-vr-optimisations.379138/
BTW, Unity has Single Pass stereo in their Engine now. I was curious if you found any thing interesting, provided that you had a chance to give it a once over.

Edit: here's the post where I first saw mention of it after googling single pass stereo as soon as they mentioned it in the pascal live event feed.

robinb-u3d Jan 14, 2016

Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye.

If you look at your current VR project in the Unity profiler you should see that on the Main Thread each camera ("Camera.Render") that is enabled for VR will generate two sets of "Drawing" markers one for each eye. The second of those markers should go away with the optimizations enabled so reducing the VR "Drawing" time by 50%. We are also seeing a large reduction in the time spent on the GfxWorker Render Thread as a result of this optimizations as well.

Currently we are targeting Oculus Rift and PlayStation VR with this round of optimizations, we'll be looking into optimizations for GearVR a little further down the line.

Hope this is of some use.

Robin

http://forum.unity3d.com/threads/upcoming-vr-optimisations.379138/

#8
Posted 05/15/2016 12:26 PM   
[quote="4everAwake"]So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case.[/quote]Thanks for that - I've added support for gs_5_0 so that file should extract now. Stranded Deep also gave me the IDs for ds_5_0, hs_5_0 and several more exotic data types (VectorBool, ScalarInt, 3D textures, Buffers, textures without samplers) so the coverage should be pretty good now. I'm still missing the shader model 4 variant of geometry shaders (guessing it will be 19 since gs_5_0 was 20, but I'll wait until I see one to be sure), and I have not yet come across a Unity game using compute shaders, so that's missing too. 1D textures are also missing - 99% certain I can guess that their ID will be '1' given the IDs used by 2D, 3D and cube textures were 2, 3 and 4 respectively, but I'll also wait until I see one in the field just in case it turns out to be a 0.
4everAwake said:So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case.
Thanks for that - I've added support for gs_5_0 so that file should extract now.

Stranded Deep also gave me the IDs for ds_5_0, hs_5_0 and several more exotic data types (VectorBool, ScalarInt, 3D textures, Buffers, textures without samplers) so the coverage should be pretty good now. I'm still missing the shader model 4 variant of geometry shaders (guessing it will be 19 since gs_5_0 was 20, but I'll wait until I see one to be sure), and I have not yet come across a Unity game using compute shaders, so that's missing too. 1D textures are also missing - 99% certain I can guess that their ID will be '1' given the IDs used by 2D, 3D and cube textures were 2, 3 and 4 respectively, but I'll also wait until I see one in the field just in case it turns out to be a 0.

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD
Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword
Contact & PayPal: darkstarsword@gmail.com

#9
Posted 05/15/2016 04:32 PM   
[quote="D-Man11"]Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye.[/quote]Interesting info, but it doesn't really help us since we aren't using Unity's stereo renderer. I do however wonder if that is what 3D Vision Automatic does internally, as it might be lighter weight than creating (and constantly switching between) two independent render targets, would easily make their reverse stereo blit API work *exactly the way it does work*, may work better with GPU caching effects (perhaps why it outperforms Tridef?), and a few other observations/haunches would suggest that might be the case. Can't be sure though - they hide all those details from us.
D-Man11 said:Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye.
Interesting info, but it doesn't really help us since we aren't using Unity's stereo renderer.

I do however wonder if that is what 3D Vision Automatic does internally, as it might be lighter weight than creating (and constantly switching between) two independent render targets, would easily make their reverse stereo blit API work *exactly the way it does work*, may work better with GPU caching effects (perhaps why it outperforms Tridef?), and a few other observations/haunches would suggest that might be the case. Can't be sure though - they hide all those details from us.

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD
Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword
Contact & PayPal: darkstarsword@gmail.com

#10
Posted 05/15/2016 04:43 PM   
What a coincidence that [u][b]The Long Dark is on sale. 10 EUR[/b][/u] http://store.steampowered.com/app/305620/
What a coincidence that The Long Dark is on sale. 10 EUR


http://store.steampowered.com/app/305620/

Intel Core i7-3820, 4 X 3,60 GHz overclocked to 4,50 GHz ; EVGA Titan X 12VRAM ; 16 GB Corsair Vengeance DDR-1600 (4x 4 GB) ; Asus VG278H 27-inch incl. 3D vision 2 glasses, integrated transmitter ; Xbox One Elite wireless controller ; Windows 10
HTC VIVE 2,5 m2 roomscale
3D VISION GAMERS - VISIT ME ON STEAM and feel free to add me:
http://steamcommunity.com/profiles/76561198064106555/edit
Image

#11
Posted 05/15/2016 06:16 PM   
I saw this post [url]https://forums.geforce.com/default/topic/920481/3d-vision/the-gate-of-firmament-3d-fix-shader-help-please-/post/4824423/#4824423[/url] [quote="tonka"]Here you will find the templates: https://github.com/DarkStarSword/3d-fixes If you install python you can extract all the relevant unity assets and then extract all shader headers.[/quote] Which made me think of seeing this on the Unity doc site. "By default to optimise the size of DirectX11 shaders, debugging information is stripped out. This means that constants and resources will have no names, and the shader source will not be available. To include this debugging information in your shader, include #pragma enable_d3d11_debug_symbols in your shader’s CGPROGRAM block." https://docs.unity3d.com/Manual/RenderDocIntegration.html I've no idea if this is something that can be implemented after the fact or if it's something that only the developer can do? I also have no idea if it's there is a similar function for dx9. Anyways, it might be totally useless, but just in case... It does seem to suggest that it can be used with Visual Studio? Debugging DirectX 11 shaders with Visual Studio https://docs.unity3d.com/Manual/SL-DebuggingD3D11ShadersWithVS.html
I saw this post
https://forums.geforce.com/default/topic/920481/3d-vision/the-gate-of-firmament-3d-fix-shader-help-please-/post/4824423/#4824423

tonka said:Here you will find the templates: https://github.com/DarkStarSword/3d-fixes

If you install python you can extract all the relevant unity assets and then extract all shader headers.


Which made me think of seeing this on the Unity doc site.

"By default to optimise the size of DirectX11 shaders, debugging information is stripped out. This means that constants and resources will have no names, and the shader source will not be available. To include this debugging information in your shader, include #pragma enable_d3d11_debug_symbols in your shader’s CGPROGRAM block."

https://docs.unity3d.com/Manual/RenderDocIntegration.html

I've no idea if this is something that can be implemented after the fact or if it's something that only the developer can do? I also have no idea if it's there is a similar function for dx9.

Anyways, it might be totally useless, but just in case...


It does seem to suggest that it can be used with Visual Studio?

Debugging DirectX 11 shaders with Visual Studio
https://docs.unity3d.com/Manual/SL-DebuggingD3D11ShadersWithVS.html

#12
Posted 07/06/2016 03:25 AM   
That's something the developers would need to do, but my scripts make it a non-issue for us.
That's something the developers would need to do, but my scripts make it a non-issue for us.

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD
Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword
Contact & PayPal: darkstarsword@gmail.com

#13
Posted 07/06/2016 08:03 AM   
Scroll To Top