UserTool: Open Sauce multifunctional command line tool
By TheFieryScythe
Usertool is a command line program that Open Sauce developers can use to provide a familar tool-like way to manipulate tags, or for any other purpose they may need.
Using UserTool:
If you know tool then you already know how to use it. Open up a command prompt and navigate to wherever usertool is located. Type 'usertool' and hit enter to list all of the commands that are available. If you are unsure as to what a particular command will do, type 'usertool *the command*' and it will print the command with a short description about what it does. Once you know what command you want to use type 'usertool *the command* *arguments*' and the command will do its stuff.
Obviously you will need to change the include directories to match your file setup.
I've tried to keep the coding for usertool as similar to OS code as I can so its simpler to switch between the two. To add a command to usertool you simply add your own namespace and then put the function that your command will call inside it. You then add a COMPONENT macro to the components array in Main.cpp and fill in the function, name, description, etc. sections.
There are a few namespaces included that handle some things for you:
ErrorPool - holds a list of error strings that you add to when your code has a problem, which then get output to the console and saved to debug.txt before the program closes.
DebugOutput - Use this to output messages to the console and to manually add messages to debug.txt.
Direct3D - Use this namespace to create and destroy a Direct3D device should you need it.
As this is mainly intended for manipulating custom tags, i've put together a basic tag interface namespace that you used to build, load and save your tags. An example of which can be found in the shader_post_process tag included. There may be circumstances where it might not work with more complex tags but those bridges will be crossed when we jump off them.
Readme says it all really, its a tool-like command line tool for OS developers to add commands to. If your not developing anything for OS and not putting any post processing effects into your maps then this isn't going to be of any interest to you :downs:.
test-post-process - Takes a shader_post_process tag as its argument and check it to see if it's HLSL is valid and compiles correctly. Does not change the tag.
compile-post-process - Takes a shader_post_process tag as its argument and if the HLSL is valid, it will compile it to a binary format and save that to the tag. This WILL remove the HLSL code from the tag, so use it at your own risk!!
January 27th, 2010, 08:47 AM
FireScythe
Re: UserTool - OpenSauce command line tool
Update 27-01-10
New build available with some changes and some new commands :downs: >>Download<<
Changes:
Replaced ErrorPool and DebugOutput with a single Console namespace which records all text put to the console and writes it to debug.txt if a function fails.
Removed handle checking from test\compile-post-process functions as it was causing a crash.
Added:
New function "os-bitmaps". os_bitmap is a new tag i've made to provide an easy way of getting bitmaps from the map cache, and this function creates either one bitmap from one image, many bitmaps from a directory of images, or a single bitmap from a directory of images.
New functions "compile-custom-ps" and "compile-custom-vs". I have decompiled and saved all of halo's vertex and pixel shaders to a number of text files. These functions allow you to edit those shaders and compile them into two files, "vs_replacements.vsr" and "ps_replacements.psr", which when used in conjunction with the current post processing build will replace Halo's shaders with your edited ones. The decompiled shaders can be found >>here<<. Extract the folders to your shaders directory.
Added a console command "-listfull" which will list all commands and their descriptions.
January 27th, 2010, 01:30 PM
Kornman00
Re: UserTool - OpenSauce command line tool
In PostProcess.cpp, I think the CheckHandles error may stem from the tag_string pointer. Verify that pointer magic you use for the pName parameter. Since tag_string is really just a "char value[31+1];" type you should just be able to pass as is (except with just a single dereference since it is a tag_string* in your CheckHandles code).
Also, you may want to verify your tag_string TagInterface code. Make sure that it properly terminates the string data (ie, some_tag_string[32] = '\0') after loading and before saving. I think I may have originally annotated that tag_string was 32 characters but later updated code to treat it as 31 characters (thats the true count, it uses a zero-terminator). I failed to go back and update the documentation though.
The documentation says that the GetParam function is suppose to return NULL if the shit is invalid, but that's probably under the assumption that pName is a proper string pointer :P
Still looking at the rest of the code
January 27th, 2010, 02:51 PM
FireScythe
Re: UserTool - OpenSauce command line tool
The GetParameter function exceptions no matter what gets passed to it, even plain old "foobar". So i'm not sure whats wrong there :saddowns:.
Fair point on the tag_string IO, code now forces the null terminator when saving and loading.
On an unrelated note, i've made a change to the custom shader compiler that will allow replacement shaders to be written in HLSL rather than just shader assembly. So that will be coming soon.
January 27th, 2010, 03:20 PM
Kornman00
Re: UserTool - OpenSauce command line tool
Have you tried setting your DX setup to debug and see if it provides any extra information? That sounds really odd.
I'm currently scheduling the next 4 to 5 months in terms of Kornner Software developments. Depending on how that goes, how far you're able to get with this, and when I'll get a decent test computer again, maybe I'll get to finalize OpenSauce 2 and include this. Since this is adding to the engine, be sure to plan your designs accordingly for any possible future endeavors as this entire (OpenSauce) setup is far from version friendly. Personally, I'd try to keep most things tag and cache based, but when it comes to user configuration, it's kind of hard to ignore letting the user define their own skeeters and tweeters :-3
January 29th, 2010, 06:29 AM
FireScythe
Re: UserTool - OpenSauce command line tool
Turns out that crash only occurs when linking to the d3dx9d.lib, so it would have been fine in release :gonk:.
Which project are you looking to include? Whichever one (or both), i'm not looking to add much more functionality at the moment, just trying fo fix bugs and polish things up, although I haven't planned anything so far so i'll have to see how it goes.
January 29th, 2010, 08:34 AM
Kornman00
Re: UserTool - OpenSauce command line tool
Any and all of your code if possible. I'll contact you a month before things are bolted down on my end to leave room for integration.
Tag field values are now references, not pointers. So dereferencing is no longer required when using the values.
Post process component has been updated to account for changes to the shader_post_process tag definition.
Shader replacement component can now compile shaders that are in either assembly format or HLSL.
D3DX errors are now parsed better.
Boost::filesystem::path is now used whenever file paths are used because its cool shit.
Component descriptions have been moved to the component cpp files since Main was getting a bit crowded.
The shader replacement component no longer uses the .txt extension. For assembly shaders the file extension is .sasm and HLSL file use .hlsl. All exported shaders are now included in the post processing build >>here<<, with an example shader thats been converted to HLSL included. Because of the HLSL inclusion the compile-custom-** commands now require a default shader profile which can be found in the command descriptions, however to define a different shader profile for each shader you can put a comment at the top of the shader to define which shader profile you want to use, "//--*shader profile*--//".
Tag structs now require a char string id so that should an error occur the block type can be printed to the console/debug for easier debugging.
TagBase now requires a tag version;
No longer necessary to declare pads in the struct, they are now added in the constructor.
Added:
DynamicArray's and tag block elements can now be accessed with [index]'s.
Fully mapped out scenario_structure_bsp definition.
PackedFile object to create a data file with a generic struture that can be easily read by OS.
Fixed:
Data fields were being loaded incorrectly when in nested blocks. Data is now loaded correctly.
Two new commands have been added this build:
pack-files - Takes a directory of files and packs them into one file that OS can use.
import-lightmap-uv - Replaces the lightmap UV's of a single bsp with those read from a matching obj file.
For the lightmap UV replacement you should wait until you have completely finished your BSP as tool can create different lightmap surface arrangements each time you run lightmaps so an obj created with an older bsp might not match a newer one, and you don't want to re-uv the bsp everytime you make a change. Also, you should ONLY change the UV's as changing the mesh can break the surface ordering and will cause problems. The workflow for using this with Aether would be:
Finish the BSP completely.
Export lightmaps with Aether.
Re-UV the lightmaps and import.
Recreate your Aether project and export everything again.
Do your Max/Maya lighting as normal.
I can't guarantee every obj file format will work so if you have problems let me know.
February 26th, 2010, 04:03 PM
Dwood
Re: UserTool - OpenSauce command line tool
Dees dreams have come true.
March 2nd, 2010, 05:58 PM
Dwood
Re: UserTool - OpenSauce command line tool
I redid a number of vertex shaders in hlsl. I have yet to figure out how pixel shaders will work (mainly because scythe didn't redo any one that takes more than 5 sasm lines :P) these may be all the ones that I rewrote however I don't think so.... Anyways; here:
If someone figures out what those registers are holding, by all means, let us know as for now I'm just shooting in the dark trying to fish out what they do.
Add each one individually and compile with usertool.
March 3rd, 2010, 12:20 AM
Kornman00
Re: UserTool - OpenSauce command line tool
Halo1 Mac and Stubbs the Zombie Mac/Xbox have the source shaders included in the install. Looking at Hal1 Mac's VSH sources would probably help. I'm not sure if I would be allowed to post them or not
March 3rd, 2010, 05:45 AM
Dwood
Re: UserTool - OpenSauce command line tool
Quote:
Originally Posted by Kornman00
Halo1 Mac and Stubbs the Zombie Mac/Xbox have the source shaders included in the install. Looking at Hal1 Mac's VSH sources would probably help. I'm not sure if I would be allowed to post them or not
I don't think that the devs would care... my main problem is figuring out what values are being held in the registers anyways.
Last night because I was bored doing homework I took turns doing the water reflection shader and Pre-Cal I did transparent_water_reflection. Gives the water a neat effect as-is however removing my modifications makes it look normal so dun u worry that hlsl screwed it up. :)
I also tried the shader changes in Revelations, and the water looks p. cool, hope ya'll like:
OUT.Tex1.w = temp_r5.x;
OUT.Tex2.w = 0 * temp_r5.y; //Nulls effect to isolate what each does
OUT.Tex3.w = 0 * temp_r5.z; //If you want it to look normal, remove the multiplication
Must spread rep. lol. Anyways I completely failed in my attempt to rewrite the active camo. Well it works, but it works a little... too well. The player is totally invisible.
March 4th, 2010, 06:49 AM
FireScythe
Re: UserTool - OpenSauce command line tool
Thanks kornman, those shaders should provide some good insight. Just to be sure poeple know, the shaders are written for OpenGL(?) so won't directly compile with usertool, some conversion is still necessary :).
Quote:
Originally Posted by Dwood
Anyways I completely failed in my attempt to rewrite the active camo. Well it works, but it works a little... too well. The player is totally invisible.
When rewriting rigged model vertex shaders i'd advise using a map that is basically a box at point 0,0,0 as thats where the verts end up (in object space) if the skinning code and/or WVP are incorrect.
March 4th, 2010, 08:36 AM
Kornman00
Re: UserTool - OpenSauce command line tool
np
There's also the Halo PC Beta vsh\fx shaders which I don't think changed until retail...but uhhhhh yeah. Not gonna post those ;x.
March 9th, 2010, 02:13 PM
Sasc
Re: UserTool - OpenSauce command line tool
Quote:
Originally Posted by FireScythe
Update 25\02\10[*]
import-lightmap-uv - Replaces the lightmap UV's of a single bsp with those read from a matching obj file.
For the lightmap UV replacement you should wait until you have completely finished your BSP as tool can create different lightmap surface arrangements each time you run lightmaps so an obj created with an older bsp might not match a newer one, and you don't want to re-uv the bsp everytime you make a change. Also, you should ONLY change the UV's as changing the mesh can break the surface ordering and will cause problems. The workflow for using this with Aether would be:
Finish the BSP completely.
Export lightmaps with Aether.
Re-UV the lightmaps and import.
Recreate your Aether project and export everything again.
Do your Max/Maya lighting as normal.
I can't guarantee every obj file format will work so if you have problems let me know.
Do the custom lightmaps still work if the d3d9.dll is not used? Similar to how the maximum BSP increase still works without the DLL? I ask because I'd love to have custom lightmaps for a map, but I don't really want to have every user install the d3d9.dll. And I also don't want to install Boost, open sauce, and the DirectX redist just to find out that I need the DLL to use custom lightmaps.
March 9th, 2010, 03:10 PM
FireScythe
Re: UserTool - OpenSauce command line tool
Yes. The lightmap UV replacement doesn't have anything to do with Open Sauce, it changes the lightmap UV's in the BSP tag itself so there is no dependence on extra files. You also don't need boost, DirectX SDK, etc. if your just using UserTool. You only need those if you are developing for it.
March 9th, 2010, 07:59 PM
Sasc
Re: UserTool - OpenSauce command line tool
That's perfect! Thanks!
March 14th, 2010, 11:52 AM
DEElekgolo
Re: UserTool - OpenSauce command line tool
I'm making a lightmapping tut, permission to include this in the tutorial files?
March 14th, 2010, 01:32 PM
FireScythe
Re: UserTool - OpenSauce command line tool
You should link to this thread since usertool gets updated/changed on occasion.
March 14th, 2010, 01:46 PM
DEElekgolo
Re: UserTool - OpenSauce command line tool
ok.
May 5th, 2010, 05:57 PM
CodeBrain
Re: UserTool - OpenSauce command line tool
usertool - version 00.01.03
Unable to locate HEK directory.
failed to find tags and/or data directories.
Could that be because it cannot find my registry for Halo CE?
May 5th, 2010, 06:02 PM
Dwood
Re: UserTool - OpenSauce command line tool
It should be in your hce root directory.
May 5th, 2010, 06:06 PM
CodeBrain
Re: UserTool - OpenSauce command line tool
It is. How else would I be running it lol
May 5th, 2010, 10:38 PM
Kornman00
Re: UserTool - OpenSauce command line tool
User tool uses the HEK registry entries so that's probably your issue. Add them or re-install
August 27th, 2010, 08:53 AM
Siliconmaster
Re: UserTool - OpenSauce command line tool
Okay, so I'm running into the same problem CodeBrain had. He theorized that the tool was looking at the x64 path, not the x86 path. Both of us are running 64 bit systems, so that would make sense. However, I have no clue how to fix this issue. Any ideas? Would be really nice to be able to import custom uvs.
My registry path was default: "SOFTWARE\\Wow6432Node\\Microsoft\\Microsoft Games\\Halo HEK". I am running Windows 7. Regardless of where your Halo Custom Edition directory is installed or whether it's x86 or x64, the registry entry it opens may be incorrect. Since UserTool must be placed in the Halo CE directory anyway, may I suggest using GetCurrentDirectory or GetModuleFileName to obtain the path? Otherwise you may need to check multiple registry paths.
My registry path was default: "SOFTWARE\\Wow6432Node\\Microsoft\\Microsoft Games\\Halo HEK". I am running Windows 7. Regardless of where your Halo Custom Edition directory is installed or whether it's x86 or x64, the registry entry it opens may be incorrect. Since UserTool must be placed in the Halo CE directory anyway, may I suggest using GetCurrentDirectory or GetModuleFileName to obtain the path? Otherwise you may need to check multiple registry paths.
This is actually quite simple to fix, however the fix I am about to give really only changes the registry usertool tries to find: the x32 Halo HEK registry, and instead makes it look for the x64 directory.
I was going to make a small tutorial on how to do this yourself, but because of how FireSythe has his project folders and directories set up (He has the directories/folders on another hard drive of some kind, I had to use my flash drive to make it look there. I even placed all the files exactly the way he had it, and Visual C++ still told me it couldnt find the files, even though it was sitting in front of it's stupid face.) I cannot accomplish that. I did however make a successful exe a while ago, and I have no idea what caused it to not work anymore.
Anyway, enough rambling, here is the exe I promised.
The odd thing is it was developed and tested on x64 since that's what I use. When I get the chance i'll change it to use the current working directory rather than the registry path.
August 28th, 2010, 08:16 AM
Siliconmaster
Re: UserTool - OpenSauce command line tool
Yeah, the usertool_x64.exe gets the same error for me, thanks though CodeBrain. Guess I'll have to wait for Firescythe's fix.
August 28th, 2010, 12:06 PM
Skarma
Re: UserTool - OpenSauce command line tool
Add some more error checking so the app can spit out the exact error codes from failed functions, might help work out the kinks
August 28th, 2010, 12:19 PM
CodeBrain
Re: UserTool - OpenSauce command line tool
Oh derp I forgot to give you guys the .reg file as well. <_>
If you install the Halo Editing Kit on a x64 bit machine, I think usertool tries to look in the x32 registry instead of the x64 registry. Therefore, the solution is to copy what the x64 registry for "Halo HEK" has and import it into the x32 bit. The file above is the solution. (I think)
August 28th, 2010, 12:46 PM
Siliconmaster
Re: UserTool - OpenSauce command line tool
Drat. Still gives that error. Does it work on your end?
And I'm testing it by typing in usertool_x64.exe, then seeing if it gives a list of commands, like tool does. Instead, it gives back that error. If I need to type an actual command, let me know. I couldn't figure out what exactly that command was, honestly.
August 30th, 2010, 11:49 AM
Skarma
Re: UserTool - OpenSauce command line tool
@Siliconmaster: Try this fix I cooked up, works great for me now. It uses a different method other than the registry to get your Halo CE working directories. Just don't forget to put it in your Halo CE folder before running it. You may need to run as admin, but it worked without that for me.
and not only that, the app is also 20 bytes faster congratz :allears:
September 1st, 2010, 10:08 AM
Siliconmaster
Re: UserTool - OpenSauce command line tool
All right, Skarma fixed the issue for me, though he initially fixed the earlier version of usertool, not the most updated one. However, now the app itself is giving me crap lol. Max exports the 22 lightmaps as a single .obj file. But when the app tries to import the uvs, it says that the bsp has 295 objects. Wat. How do I fix this?
September 1st, 2010, 10:23 AM
Skarma
Re: UserTool - OpenSauce command line tool
This line of code is just backwards.
obj group count = 295
lightmap count = 22
Code:
Console::AddFormattedText("obj group count (%i) does not match bsp lightmap count (%i)", bsp.lightmaps.element_count - 1, obj_file.groups.Count);
I think your obj format is not compatible with UserTool.
September 1st, 2010, 11:05 AM
Siliconmaster
Re: UserTool - OpenSauce command line tool
Yeah, turns out max was exporting everything, not just the lightmaps. This time I used the "export selected" option. :downs: Imported the uvs fine, just need to make sure they match the lightmaps ingame.
And they do. Yay. Pics in a bit.
September 1st, 2010, 12:34 PM
Siliconmaster
Re: UserTool - OpenSauce command line tool
Doublepost for importance, also posted this in the Precipice thread.:
Thanks to Skarma, usertool now works on my computer. The first import tests have been very successful, but I've been seeing a few very odd glitches. I think the importer may be modifying the level uvs as well, resulting in bizarre uv errors that were NOT there before. I managed to recover it from the ingame build, but just barely.
Edit: Confirmed. Prior to the import, the uv error was not there.