Unity 5.3 scripts (Updated for Unity 5.5)
  1 / 2    
Just a heads up - after a week of pretty solid reverse engineering and scripting, my scripts are now working with Unity 5.3, and I've updated The Long Dark to prove it ;-) This version of Unity changed the shader format in several ways - DX9 shaders are no longer stripped, but DX11 shaders are still stripped as they were before. For DX9 this means that all the hashes will have changed, breaking the fixes for any games that update to this version (if this happens, it may be worth trying UseAlternateCRC to see if it can get the old fix working before spending time updating it). For DX11 the hashes won't necessarily change as a result of updating to Unity 5.3, though of course they may change for the usual reasons they do on updates (e.g. the directional lighting vertex shader hash has changed). For DX11 the headers need to be extracted from the Unity asset files as before, and I have updated my scripts to enable this. For DX9 I would still recommend extracting the Unity headers even though the DX headers are no longer stripped - the Unity headers contain more information than is found in the DX shader headers - most importantly, they indicate which shaders are used as shadow casters (and therefore should be avoided when copying matrices). The second change they made was to their shader file format, which broke my scripts. Traditionally Unity has stored shaders in the asset files in either plain text (DX9, OpenGL) or using a trivial encoding scheme (DX11) embedded inside some plain-text data structures with info about the shaders - this is the info my scripts extract and attach to the shaders. This plain text format still exists in Unity 5.3, but almost everything of interest in it has been removed and moved to a new compressed binary section of the shader file, which I've spent most of the last week deciphering and adapting my scripts to process. There's still a few bytes that I have not deciphered, but they don't seem very important. unity_asset_extractor will now extract .shader.decompressed files (along with the .shader files that it used to extract) from the Unity .asset files (which includes the files in the resources folder despite the missing extension). extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand. I've updated my Unity 5 DX11 template and added an autofix53.sh script to it (later I might combine that into autofix.sh). The DX9 template has not been updated, but it should only be a few small tweaks to the script to make it work (e.g. using extract_unity53_shaders instead of extract_unity_shaders). Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them. As usual, all the scripts can be found in my 3d-fixes repository linked from my signature.
Just a heads up - after a week of pretty solid reverse engineering and scripting, my scripts are now working with Unity 5.3, and I've updated The Long Dark to prove it ;-)

This version of Unity changed the shader format in several ways - DX9 shaders are no longer stripped, but DX11 shaders are still stripped as they were before.

For DX9 this means that all the hashes will have changed, breaking the fixes for any games that update to this version (if this happens, it may be worth trying UseAlternateCRC to see if it can get the old fix working before spending time updating it).

For DX11 the hashes won't necessarily change as a result of updating to Unity 5.3, though of course they may change for the usual reasons they do on updates (e.g. the directional lighting vertex shader hash has changed).

For DX11 the headers need to be extracted from the Unity asset files as before, and I have updated my scripts to enable this. For DX9 I would still recommend extracting the Unity headers even though the DX headers are no longer stripped - the Unity headers contain more information than is found in the DX shader headers - most importantly, they indicate which shaders are used as shadow casters (and therefore should be avoided when copying matrices).



The second change they made was to their shader file format, which broke my scripts. Traditionally Unity has stored shaders in the asset files in either plain text (DX9, OpenGL) or using a trivial encoding scheme (DX11) embedded inside some plain-text data structures with info about the shaders - this is the info my scripts extract and attach to the shaders. This plain text format still exists in Unity 5.3, but almost everything of interest in it has been removed and moved to a new compressed binary section of the shader file, which I've spent most of the last week deciphering and adapting my scripts to process. There's still a few bytes that I have not deciphered, but they don't seem very important.


unity_asset_extractor will now extract .shader.decompressed files (along with the .shader files that it used to extract) from the Unity .asset files (which includes the files in the resources folder despite the missing extension).

extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand.

I've updated my Unity 5 DX11 template and added an autofix53.sh script to it (later I might combine that into autofix.sh). The DX9 template has not been updated, but it should only be a few small tweaks to the script to make it work (e.g. using extract_unity53_shaders instead of extract_unity_shaders).


Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them.


As usual, all the scripts can be found in my 3d-fixes repository linked from my signature.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#1
Posted 04/30/2016 07:08 PM   
Thanks for this. Out of necessity I've had to learn to build macros to implement offline fixes to hundreds of shaders, and I always find that an interesting challenge, but that's a drop in the bucket compared to these awesome tools you provide. Would love to refresh my decade old learning in programming to be able to implement more advanced scripts and tools like you do (been toying with the idea of going back to school, but kinda hard to figure out how to make that work). Anyway, thanks again. I'll go ahead and test this out on one of the games that updated to 5.3 that broke a fix I was working on with your previous version of the tool and confirm it's success.
Thanks for this. Out of necessity I've had to learn to build macros to implement offline fixes to hundreds of shaders, and I always find that an interesting challenge, but that's a drop in the bucket compared to these awesome tools you provide. Would love to refresh my decade old learning in programming to be able to implement more advanced scripts and tools like you do (been toying with the idea of going back to school, but kinda hard to figure out how to make that work). Anyway, thanks again. I'll go ahead and test this out on one of the games that updated to 5.3 that broke a fix I was working on with your previous version of the tool and confirm it's success.

3D Gaming Rig: CPU: i5 6600K | Mobo: Asus Maximus Hero VIII | RAM: Corsair Dominator 16GB | GPU: GTX 1080 Ti | 2xSSDs for OS and Apps | PSU: Seasonic X-1250 M2| Case: Corsair C70 | Cooling: Hyper 212+ (Air) | Displays: BenQ XL2420TX & BenQ HT1075 | OS: Windows 10 Pro

Like my fixes? Dontations can be made to: www.paypal.me/DShanz or rshannonca@gmail.com
Like electronic music? Check out: www.soundcloud.com/dj-ryan-king

#2
Posted 04/30/2016 07:54 PM   
Great work DSS, Thanks alot for your dedication, really appreciate it. I have bought NightCry recently wich is Unity 5.3.3, so I'm very exited about your update.. I'll look into it next week, when I hopefully get some sparetime :)
Great work DSS, Thanks alot for your dedication, really appreciate it.

I have bought NightCry recently wich is Unity 5.3.3, so I'm very exited about your update..
I'll look into it next week, when I hopefully get some sparetime :)

Win7 64bit Pro
CPU: 4790K 4.8 GHZ
GPU: Aurus 1080 TI 2.08 GHZ - 100% Watercooled !
Monitor: Asus PG278QR
And lots of ram and HD's ;)

#3
Posted 04/30/2016 08:09 PM   
Awesome, this is great news. TY For those like me that hadn't heard of The Long Dark or NightCry, here's the Steam links. http://store.steampowered.com/app/305620/ https://www.youtube.com/watch?v=dmAYWj58gKI http://store.steampowered.com/app/427660/ https://www.youtube.com/watch?v=vnseGvK-9aY
Awesome, this is great news. TY

For those like me that hadn't heard of The Long Dark or NightCry, here's the Steam links.


http://store.steampowered.com/app/305620/



http://store.steampowered.com/app/427660/

#4
Posted 05/01/2016 12:39 AM   
Thank you so much for the updates, DSS! [quote="DarkStarSword"]extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand.[/quote] Where can I locate "cmd_Decompiler"? I did a search but I was unable to find it. If I try to use "extract_unity53_shaders" without it, I only get Dx9 & Dx11 headers. Edit: I think I found it but I'm not quite sure how to use it-- https://github.com/bo3b/3Dmigoto/tree/master/HLSLDecompiler/cmd_Decompiler I suppose Visual C++ is needed?
Thank you so much for the updates, DSS!

DarkStarSword said:extract_unity53_shaders takes the .shader.decompressed files generated with unity_asset_extractor. If cmd_Decompiler is placed next to the script it will use it to extract binary and assembly versions of DX9 shaders; binary, assembly and HLSL versions of DX11 shaders and GLSL versions of OpenGL shaders. The DX9 and DX11 hashes will match those used by Helix Mod and 3DMigoto. It decodes the bind info from the compressed shader blob in Unity and attaches it to the headers, using the same format that previous versions of Unity used and that shadertool/hlsltool/asmtool understand.


Where can I locate "cmd_Decompiler"? I did a search but I was unable to find it. If I try to use "extract_unity53_shaders" without it, I only get Dx9 & Dx11 headers.

Edit: I think I found it but I'm not quite sure how to use it-- https://github.com/bo3b/3Dmigoto/tree/master/HLSLDecompiler/cmd_Decompiler

I suppose Visual C++ is needed?

#5
Posted 05/12/2016 06:58 AM   
You can grab it from the 3DMigoto releases page here: https://github.com/bo3b/3Dmigoto/releases It will be updated anytime we make an improvement to the 3DMigoto decompiler, assembler or disassembler. Let me know if you have any trouble with the scripts - I expect there might still be a few combinations in some games they aren't handling yet, and I've made them err on the side of failing if they see something they aren't sure about rather than risk missing something.
You can grab it from the 3DMigoto releases page here:
https://github.com/bo3b/3Dmigoto/releases

It will be updated anytime we make an improvement to the 3DMigoto decompiler, assembler or disassembler.


Let me know if you have any trouble with the scripts - I expect there might still be a few combinations in some games they aren't handling yet, and I've made them err on the side of failing if they see something they aren't sure about rather than risk missing something.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#6
Posted 05/12/2016 09:53 AM   
Cool. I got it working now. Thanks Again! [quote="DarkStarSword"]Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them.[/quote] So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case. https://www.dropbox.com/s/zh7gzjqyftn29l4/DepthOfFieldDX11.shader.decompressed?dl=0
Cool. I got it working now. Thanks Again!

DarkStarSword said:Currently extract_unity53_shaders will fail if it comes across a shader using hull, domain, geometry or compute shaders, certain rare data types, etc. So far I've only used shaders from The Long Dark to test, which did not do anything too exotic. I will need samples of any files it fails on to determine what values correspond to these to add support for them.

So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case.

https://www.dropbox.com/s/zh7gzjqyftn29l4/DepthOfFieldDX11.shader.decompressed?dl=0

#7
Posted 05/15/2016 05:42 AM   
BTW, Unity has Single Pass stereo in their Engine now. I was curious if you found any thing interesting, provided that you had a chance to give it a once over. Edit: here's the post where I first saw mention of it after googling single pass stereo as soon as they mentioned it in the pascal live event feed. robinb-u3d Jan 14, 2016 Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye. If you look at your current VR project in the Unity profiler you should see that on the Main Thread each camera ("Camera.Render") that is enabled for VR will generate two sets of "Drawing" markers one for each eye. The second of those markers should go away with the optimizations enabled so reducing the VR "Drawing" time by 50%. We are also seeing a large reduction in the time spent on the GfxWorker Render Thread as a result of this optimizations as well. Currently we are targeting Oculus Rift and PlayStation VR with this round of optimizations, we'll be looking into optimizations for GearVR a little further down the line. Hope this is of some use. Robin http://forum.unity3d.com/threads/upcoming-vr-optimisations.379138/
BTW, Unity has Single Pass stereo in their Engine now. I was curious if you found any thing interesting, provided that you had a chance to give it a once over.

Edit: here's the post where I first saw mention of it after googling single pass stereo as soon as they mentioned it in the pascal live event feed.

robinb-u3d Jan 14, 2016

Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye.

If you look at your current VR project in the Unity profiler you should see that on the Main Thread each camera ("Camera.Render") that is enabled for VR will generate two sets of "Drawing" markers one for each eye. The second of those markers should go away with the optimizations enabled so reducing the VR "Drawing" time by 50%. We are also seeing a large reduction in the time spent on the GfxWorker Render Thread as a result of this optimizations as well.

Currently we are targeting Oculus Rift and PlayStation VR with this round of optimizations, we'll be looking into optimizations for GearVR a little further down the line.

Hope this is of some use.

Robin

http://forum.unity3d.com/threads/upcoming-vr-optimisations.379138/

#8
Posted 05/15/2016 12:26 PM   
[quote="4everAwake"]So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case.[/quote]Thanks for that - I've added support for gs_5_0 so that file should extract now. Stranded Deep also gave me the IDs for ds_5_0, hs_5_0 and several more exotic data types (VectorBool, ScalarInt, 3D textures, Buffers, textures without samplers) so the coverage should be pretty good now. I'm still missing the shader model 4 variant of geometry shaders (guessing it will be 19 since gs_5_0 was 20, but I'll wait until I see one to be sure), and I have not yet come across a Unity game using compute shaders, so that's missing too. 1D textures are also missing - 99% certain I can guess that their ID will be '1' given the IDs used by 2D, 3D and cube textures were 2, 3 and 4 respectively, but I'll also wait until I see one in the field just in case it turns out to be a 0.
4everAwake said:So far I've tested 3 games and they all failed on one file: DepthOfFieldDX11.shader.decompressed . Once removed, the shaders extract normally. You probably already know about this file, but I'll link it here just in case.
Thanks for that - I've added support for gs_5_0 so that file should extract now.

Stranded Deep also gave me the IDs for ds_5_0, hs_5_0 and several more exotic data types (VectorBool, ScalarInt, 3D textures, Buffers, textures without samplers) so the coverage should be pretty good now. I'm still missing the shader model 4 variant of geometry shaders (guessing it will be 19 since gs_5_0 was 20, but I'll wait until I see one to be sure), and I have not yet come across a Unity game using compute shaders, so that's missing too. 1D textures are also missing - 99% certain I can guess that their ID will be '1' given the IDs used by 2D, 3D and cube textures were 2, 3 and 4 respectively, but I'll also wait until I see one in the field just in case it turns out to be a 0.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#9
Posted 05/15/2016 04:32 PM   
[quote="D-Man11"]Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye.[/quote]Interesting info, but it doesn't really help us since we aren't using Unity's stereo renderer. I do however wonder if that is what 3D Vision Automatic does internally, as it might be lighter weight than creating (and constantly switching between) two independent render targets, would easily make their reverse stereo blit API work *exactly the way it does work*, may work better with GPU caching effects (perhaps why it outperforms Tridef?), and a few other observations/haunches would suggest that might be the case. Can't be sure though - they hide all those details from us.
D-Man11 said:Currently when Unity performs a render from a VR enabled camera in Unity it will internally perform one set of culling and then in turn render the left eye and then the right eye. The optimizations we are working on internally at the moment move most of Unity over to performing only one rendering pass that will render to both eyes at the same time. This packs both eye targets into one double wide target and when it gets to Unity's render thread we will at the last possible moment double up the render call again binding appropriate matrices and setting up the viewport to render to the correct half of the double wide target for each eye.
Interesting info, but it doesn't really help us since we aren't using Unity's stereo renderer.

I do however wonder if that is what 3D Vision Automatic does internally, as it might be lighter weight than creating (and constantly switching between) two independent render targets, would easily make their reverse stereo blit API work *exactly the way it does work*, may work better with GPU caching effects (perhaps why it outperforms Tridef?), and a few other observations/haunches would suggest that might be the case. Can't be sure though - they hide all those details from us.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#10
Posted 05/15/2016 04:43 PM   
What a coincidence that [u][b]The Long Dark is on sale. 10 EUR[/b][/u] http://store.steampowered.com/app/305620/
What a coincidence that The Long Dark is on sale. 10 EUR


http://store.steampowered.com/app/305620/

Intel Core i7-3820, 4 X 3,60 GHz overclocked to 4,50 GHz ; EVGA Titan X 12VRAM ; 16 GB Corsair Vengeance DDR-1600 (4x 4 GB) ; Asus VG278H 27-inch incl. 3D vision 2 glasses, integrated transmitter ; Xbox One Elite wireless controller ; Windows 10
HTC VIVE 2,5 m2 roomscale
3D VISION GAMERS - VISIT ME ON STEAM and feel free to add me:
http://steamcommunity.com/profiles/76561198064106555/edit
Image

#11
Posted 05/15/2016 06:16 PM   
I saw this post [url]https://forums.geforce.com/default/topic/920481/3d-vision/the-gate-of-firmament-3d-fix-shader-help-please-/post/4824423/#4824423[/url] [quote="tonka"]Here you will find the templates: https://github.com/DarkStarSword/3d-fixes If you install python you can extract all the relevant unity assets and then extract all shader headers.[/quote] Which made me think of seeing this on the Unity doc site. "By default to optimise the size of DirectX11 shaders, debugging information is stripped out. This means that constants and resources will have no names, and the shader source will not be available. To include this debugging information in your shader, include #pragma enable_d3d11_debug_symbols in your shader’s CGPROGRAM block." https://docs.unity3d.com/Manual/RenderDocIntegration.html I've no idea if this is something that can be implemented after the fact or if it's something that only the developer can do? I also have no idea if it's there is a similar function for dx9. Anyways, it might be totally useless, but just in case... It does seem to suggest that it can be used with Visual Studio? Debugging DirectX 11 shaders with Visual Studio https://docs.unity3d.com/Manual/SL-DebuggingD3D11ShadersWithVS.html
I saw this post
https://forums.geforce.com/default/topic/920481/3d-vision/the-gate-of-firmament-3d-fix-shader-help-please-/post/4824423/#4824423

tonka said:Here you will find the templates: https://github.com/DarkStarSword/3d-fixes

If you install python you can extract all the relevant unity assets and then extract all shader headers.


Which made me think of seeing this on the Unity doc site.

"By default to optimise the size of DirectX11 shaders, debugging information is stripped out. This means that constants and resources will have no names, and the shader source will not be available. To include this debugging information in your shader, include #pragma enable_d3d11_debug_symbols in your shader’s CGPROGRAM block."

https://docs.unity3d.com/Manual/RenderDocIntegration.html

I've no idea if this is something that can be implemented after the fact or if it's something that only the developer can do? I also have no idea if it's there is a similar function for dx9.

Anyways, it might be totally useless, but just in case...


It does seem to suggest that it can be used with Visual Studio?

Debugging DirectX 11 shaders with Visual Studio
https://docs.unity3d.com/Manual/SL-DebuggingD3D11ShadersWithVS.html

#12
Posted 07/06/2016 03:25 AM   
That's something the developers would need to do, but my scripts make it a non-issue for us.
That's something the developers would need to do, but my scripts make it a non-issue for us.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#13
Posted 07/06/2016 08:03 AM   
This should probably have been included in this thread: [quote]If anyone's using my scripts and unity template be aware that there have been a few updates recently. You should use 3DMigoto 1.2.65 or later - it has some important updates that the templates make use of. - autofix53.sh now uses assembly for the lighting fix, which I found performed 5-15fps better in Dreamfall Chapters depending on the location. Note that you should enable "assemble_signature_comments = 1" in the d3dx.ini to make sure that this is done safely (this option was added in the latest 3DMigoto - it goes in the [Rendering] section if you are updating an existing fix). - The inverse shader now pre-multiplies two matrices together to get the inverse view-projection matrix, which can save instructions in all the shaders that use it. It will still return the inverse model-view-projection matrix as well for compatibility with existing fixes. - The resulting matrix is now copied into a constant buffer once after the inverse matrix shader has run, and from there is copied by reference into each of the shaders that need it. This saved an additional 5fps in Dreamfall Chapters. - The template includes a new [ClearDepthStencilView] section to work with games that render multiple scenes per frame. - Fixes for Cinematic Ambient Occlusion and soft shadows are included in the template. - asmtool's auto halo fix will now work on more shaders (especially distortion effects in recent Unity versions) - asmtool will no longer patch some shaders that didn't need a fix (harmless to leave them in) [/quote]
This should probably have been included in this thread:

If anyone's using my scripts and unity template be aware that there have been a few updates recently.

You should use 3DMigoto 1.2.65 or later - it has some important updates that the templates make use of.

- autofix53.sh now uses assembly for the lighting fix, which I found performed 5-15fps better in Dreamfall Chapters depending on the location. Note that you should enable "assemble_signature_comments = 1" in the d3dx.ini to make sure that this is done safely (this option was added in the latest 3DMigoto - it goes in the [Rendering] section if you are updating an existing fix).

- The inverse shader now pre-multiplies two matrices together to get the inverse view-projection matrix, which can save instructions in all the shaders that use it. It will still return the inverse model-view-projection matrix as well for compatibility with existing fixes.

- The resulting matrix is now copied into a constant buffer once after the inverse matrix shader has run, and from there is copied by reference into each of the shaders that need it. This saved an additional 5fps in Dreamfall Chapters.

- The template includes a new [ClearDepthStencilView] section to work with games that render multiple scenes per frame.

- Fixes for Cinematic Ambient Occlusion and soft shadows are included in the template.

- asmtool's auto halo fix will now work on more shaders (especially distortion effects in recent Unity versions)

- asmtool will no longer patch some shaders that didn't need a fix (harmless to leave them in)

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#14
Posted 01/12/2018 10:15 PM   
My Unity scripts are now updated for Unity 5.5 (and possibly 5.6 and 2017, but I haven't used them for a full fix on those versions yet to check for any other new problems they may have introduced). Unity 5.5 made major changes to their shader file format, including completely replacing their old textual metadata with a custom binary format that was... pretty challenging to reverse engineer. [list] [.] There's a new autofix55.sh script for Unity 5.5+ to automate the process[/.] [.] unity_asset_extractor.py now supports Unity 5.5 asset files, and will extract the new shader assets as raw files. Unity 5.5 no longer includes the shader name in a place this script can find, so the extracted files will have generic hex code names instead.[/.] [.] There's a new extract_unity55_shaders.py to parse the new binary format Unity 5.5 introduced to replace their old textual format, extracts the shaders and attaches headers. Pass -vv to this script to see the massive amount of data that this script has to be able to decode to find the shaders (deciphering these was a huge challenge, and necessary to even be able to find the shaders that were after them). Not all fields it decodes are added to the headers, but those used by the other scripts are.[/.] [.] There's a new unity_asset_bundle_extractor.py script that can extract shader assets from asset bundle files that are used in some games like Life is Strange Before the Storm. This only supports the newer UNITYFS version of these files, so may not work with asset bundles from older versions of Unity. The autofix script will *not* call this script, because that can be quite slow, there isn't a standard location for these (they might be under *_Data/StreamingAssets/Bundles, but LiSBtS also had a bunch of them under DLC), and only a few games use them. If you find a shader in a Unity game that wasn't extracted by the regular extract_unity_shaders script you may need to use this one as well.[/.] [.]Includes Unity 5.5 lighting shaders from LiSBtS[/.] [.]Includes one variant of a highly unusual lighting shader that does not follow the usual pattern (PCF_5x5_FORCE_INV_PROJECTION_IN_PS)[/.] [.]Copy limits are now reset from the ClearRenderTargetView call instead of ClearDepthStencilView - the later was resulting in bogus matrices in LiSBtS leading to some effects (such as reflections and certain fog effects) being broken.[/.] [/list]
My Unity scripts are now updated for Unity 5.5 (and possibly 5.6 and 2017, but I haven't used them for a full fix on those versions yet to check for any other new problems they may have introduced). Unity 5.5 made major changes to their shader file format, including completely replacing their old textual metadata with a custom binary format that was... pretty challenging to reverse engineer.

  • There's a new autofix55.sh script for Unity 5.5+ to automate the process

  • unity_asset_extractor.py now supports Unity 5.5 asset files, and will extract the new shader assets as raw files. Unity 5.5 no longer includes the shader name in a place this script can find, so the extracted files will have generic hex code names instead.

  • There's a new extract_unity55_shaders.py to parse the new binary format Unity 5.5 introduced to replace their old textual format, extracts the shaders and attaches headers. Pass -vv to this script to see the massive amount of data that this script has to be able to decode to find the shaders (deciphering these was a huge challenge, and necessary to even be able to find the shaders that were after them). Not all fields it decodes are added to the headers, but those used by the other scripts are.

  • There's a new unity_asset_bundle_extractor.py script that can extract shader assets from asset bundle files that are used in some games like Life is Strange Before the Storm. This only supports the newer UNITYFS version of these files, so may not work with asset bundles from older versions of Unity. The autofix script will *not* call this script, because that can be quite slow, there isn't a standard location for these (they might be under *_Data/StreamingAssets/Bundles, but LiSBtS also had a bunch of them under DLC), and only a few games use them. If you find a shader in a Unity game that wasn't extracted by the regular extract_unity_shaders script you may need to use this one as well.

  • Includes Unity 5.5 lighting shaders from LiSBtS

  • Includes one variant of a highly unusual lighting shader that does not follow the usual pattern (PCF_5x5_FORCE_INV_PROJECTION_IN_PS)

  • Copy limits are now reset from the ClearRenderTargetView call instead of ClearDepthStencilView - the later was resulting in bogus matrices in LiSBtS leading to some effects (such as reflections and certain fog effects) being broken.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#15
Posted 01/12/2018 10:16 PM   
  1 / 2    
Scroll To Top