OpenSMACX v0.2.3

[scient]

Initial release posted here including source:
https://github.com/b-casey/OpenSMACX

Let me know if you run into any issues, questions or comments about code. Or if you notice any mistakes or realize what some of the unknown structure values are. I've done pretty rigorous tests and countless rechecks to make sure the source mirrors original code except in areas I've improved logic or fixed bugs (as well as a couple auxiliary functions that are WIP).

You can copy over the exe and dll to any existing installation. The exe is a vanilla binary patched only with redirects to dll (ie. it does not include any additional bug fixes from my previous patches).

8/18/20: Added v0.2.3 notes
5/17/20: Added v0.2.2 notes
2/10/20: Added v0.2.1 notes
1/16/20: Added v0.2.0 notes
3/26/19: Init

v0.2.3
* Lots of additional analysis, clean up, refactoring, and optimization of the existing code base.
* Fixed four bugs within OpenSMACX code.
* Created new probe related source files (two net new functions, shifted one existing).
* Implemented one of the probe related fixes from my unofficial patch.
* Additional work on Path class (almost complete, 3 functions left).
* 55 net new functions across multiple areas.
* Cumulative decompiled and redirected function count: 420

Bug Fixes:
* Fixed a bug with OpenSMACX where a small change to how the Random class was reseeded from the
  default behavior caused rivers globally to shift. This happened whenever a Borehole or Condenser
  was built by any faction. I'm still unsure as to why. Something to review again in the future when
  more of the world code is decompiled.
* Fixed a bug with OpenSMACX where after selecting a random faction for a game, afterward on game
  start the game would throw a faction parsing error related to JENN282. This was due to the
  JENN282 code not being implemented in read_factions(). Originally, it looked like this was left
  over debug code so it was skipped. As it turns out, JENN282 is the internal faction identifier
  that SMACX uses for a random faction. Thankfully, this was the only place in the project where
  code wasn't implemented like this (excluding obvious optimizations). Multiple improvements to the
  code's performance and fixes for fringe cases were applied to this parsing code.
* Fixed an issue with OpenSMACX where two functions (tech_ai, rnd) weren't having rand() reseeded
  properly. This was caused by my_srand() only having the old built-in version of srand(). Both need
  to be initialized while the project straddles new and old code base.
* Fixed a preference parsing bug with OpenSMACX where "Time Controls" default value could have been
  set as "0" rather than "1". This was due to an oversight in how the return value was processed
  by prefs_get(LPCSTR, int, BOOL) when the useDefault parameter was set to true. This code is part
  of some of the earliest decompiled functions prior to my including comprehensive regression tests
  into the decompile process.
* Fixed a bug inside success_rates() where a probe wouldn't receive a penalty to its survival rate
  when it targeted a faction that has the Hunter-Seeker Algorithm secret project. Instead, its
  success rate would be given an erroneous 2nd penalty. This issue was somewhat masked because the
  2nd penalty happened after the success rate had already been written for display by the UI. This
  bug caused a lower success rate and increased survival rate of probes targeting an HSA faction.
* Fixed various issues related to the Scenario Editor undo/redo mechanic. Undo auto-saves weren't
  properly saved outside of the 1st one because auto_undo() didn't include the ".SAV" extension
  to process previous auto-saves. Undo auto-saves weren't properly removed because wipe_undo()
  didn't include the ".SAV" extension. Added in logic to prevent being able to attempt a redo before
  an undo has taken place. Fixed a map redraw issue after loading an undo or redo where units and
  tiles wouldn't be properly redrawn.

v0.2.2
* Lots of clean up and optimization of the existing code base.
* Initial work on the Path class.
* Added some Council related functions.
* 50 new functions across almost every game related area with a focus on Map and Technology.
* Decompiled and redirected function count: 365

Bug Fixes:
* Fixed a bug under certain conditions that the end game function to determine whether a faction is nearing a diplomatic win (Supreme Leader) would return incorrect results. The function aah_ooga() is called with the 2nd parameter set to -1 in certain instances. The original code would then attempt to use this value to check the pact status within the player_data structure of the 1st faction parameter. In these instances, it would actually be trying to do the diplomacy check against tutorialMoreBases value. There was a check to skip the pact check if the 2nd parameter was 0. The fix now accepts -1 or 0 to skip the pact check.

v0.2.1
* Some additional map functions to get ready for working on Path class.
* Miscellaneous clean up.
* Fixed an issue with the patcher script incorrectly patching best_specialist() and base_making().
* Decompiled and redirected function count: 315

Bug Fixes:

* Factions with the FREEPROTO flag (Spartans) will gain free retooling in their bases as long as the production switch is within the same category (unit to unit, base facility to base facility) and they've discovered the necessary tech for Skunkworks (default is "Advanced Subatomic Theory"). This is to resolve the issue with FREEPROTO factions never being able to gain the undocumented retooling ability of a Skunkworks found in base_making(). There is good indication this was likely an oversight and that the FREEPROTO flag should be the equivalent to a free Skunkworks for the faction's bases.

v0.2.0
* Veh, Map and Base related code that sets the groundwork to break down more complex functions.
* Engine classes: Font, Spot, Time.
* Decompiled and redirected function count: 308

Bug Fixes:
* Added an additional check to facility_avail() that prevents the Caretakers from being given the
  ability of selecting the secret project "Ascent to Transcendence". This goes against their core
  philosophy and would cause them to declare war on themselves if initiated.
* Sealurks no longer receive a movement penalty when moving through Sea Fungus. They are now treated
  the same as an "Isle of the Deep" as intended. This was likely an oversight to add the specific
  check to hex_cost().
* Added a new check inside facility_avail() that prevents building Paradise Garden if you have
  Punishment Sphere built or in queue. There is a check to prevent Punishment Sphere from being
  constructed if you have Paradise Garden but not vise versa. Paradise Garden and Punishment Sphere
  are antithetical facilities where you can build either one in any given base, but you were never
  suppose to be able to build both.
* Fixed a bug in DirectPlay multiplayer that likely caused a performance hit when moving units due
  to a faulty coordinate check. Whenever the stack_fix() function was called by moving a unit or
  other actions, it would cause every stack of units on the map to be sorted in a certain way (at
  least temporarily) as well as redundantly sorting the same stack once for every unit on the map.
* Removed an extra space displayed for certain prototype statistics (Ex. Transport Foil) shown in
  the Military Command Nexus via say_stats() function.
* Fixed logic inside offense_proto() and armor_proto() where under certain conditions the game
  would try to compare an arbitrary memory value against the Spore Launcher basic unit id. If they
  matched, non-PSI units would display incorrect offense and defense values. This happened any time
  you clicked on a non-PSI unit when calculating its offensive value. It was just incredibly rare
  the memory address value would match the Spore Launcher id. While it was unlikely this would be
  triggered by armor_proto() due to logic flow, added in a preventative bound check anyway.
* Fixed a bug in Time::pulse(void(__cdecl *)(int), int, uint32_t, uint32_t) where the Timer event
  could have persisted and executed continuously when it should have executed only once. It seems
  affected code branch was never used by the original game.
* Miscellaneous additional error handling or bound checks to various functions (see source).

Enhancement:
* Added the ability to set a reactor value (1-4) for #UNITS inside alpha/x.txt. To do so, add a
  comma after the Abil field with the value of the reactor you want for the unit. If no value or a
  value of 0 is set, it defaults to 1 (Fission Plant) like the original code.  For SMACX only, there
  are two exceptions where the default isn't 1 but 2 (Battle Ogre MK2) and 3 (Battle Ogre MK3). The
  Ogre defaults can still be overridden.
  Ex. "Colony Pod,..., 00000000000000000000000000,4" will give Colony Pods a Singularity Engine.
* Added some basic randomization for sea base name order. This mostly affects Pirates since
  the randomization method isn't great for small lists. Base name count increases once
  all names inside faction files and basename.txt are exhausted rather than stopping.

v0.1
* Parsing in alpha/x.txt and faction text files complete

Bug fixes:
* Fixed crash if BaseButton::set_name(LPCSTR input) parameter was NULL.
* Fixed bug with TextIndex having carriage return at end of "HEADER\r" breaking compare. This could
  have had significant performance increases back in 1999. Not so much in the time of cheap SSDs.
* Fixed bug if Heap::get() realloc ended up shifting memory to a different location (unlikely).
* Fixed crash in proto_cost() found via fuzzing/testing. Could be triggered by mods.
* Fixed Retool strictness in alpha/x.txt to accept a value of 3 (never free).
* Fixed setting alpha/x.txt probe teams can steal technologies now only to accept 0 or 1.
* Fixed setting alpha/x.txt humans always contact in net games now only to accept 0 or 1.
* Fixed setting alpha/x.txt humans always contact in hotseat/email games now only to accept 0 or 1.
* Fixed setting alpha/x.txt obliterating a base counts as an atrocity now only to accept 0 or 1.
* Fixed socIdeologyEffect overriding faction text files with -1, disabling AI faction "Emphasis".
* Fixed reactor power value being ignored when parsing. Game still won't use this value until
  more code is decompiled since hardcoded values are being used throughout.

Changes:
* Improved how Random::reseed sets new seed value.
* Improved performance of proto_cost() without changes to original logic.
* Removed debug code related to non-existent faction JENN282. Nothing of value. SMACX binary only.
* Various optimizations without changes to original logic.

Enhancement:
* The original code explicitly sets facility freetech value to disabled (-2) overriding alpha/x.txt. 
  It states in #FACILITIES alpha/x.txt: "Free  = No longer supported". This mechanic could have been
  removed for balance reasons or dropped due to time constraints. There is code that checks this
  value and sets the free facility only for new bases built after discovering the tech. It looks
  like existing bases do not get it. This will have to be reviewed more. For now, this mechanic is
  included as is. You can revert to vanilla behavior by modifying the four entries below in
  alpha/x.txt #FACILITIES with free tech parameter set to "Disabled".

   Recycling Tanks, 4, 0, Biogen, EcoEng2,   > free with "Adv.Ecological Engineering"
   Recreation Commons, 4, 1, Psych, SentEco, > free with "Sentient Econometrics"
   Energy Bank, 8, 1, IndEcon, QuanMac,      > free with "Quantum Machinery"
   Network Node, 8, 1, InfNet, HAL9000,      > free with "Self - Aware Machines"

[Vidsek]

    Congratulations on the release!!!!!

  It looks good as far as I could understand the codespeak  .

  Would combining this with your earlier bugfixes (and maybe other folks' stuff) be in the future?

  Someday, I'd love to see a patch that simply cleaned up as many bugs as possible without any frills, that might be  generally accepted by our community as a new standard baseline for all .txt and .exe mods to start from.

[dino]

Out of curiosity, what % of the code is dissassembled ?

The idea of having full smac source code to play with makes me want go back to learning C++ and programming, I've abandoned 15 years ago.

[scient]

  Would combining this with your earlier bugfixes (and maybe other folks' stuff) be in the future?

  Someday, I'd love to see a patch that simply cleaned up as many bugs as possible without any frills, that might be  generally accepted by our community as a new standard baseline for all .txt and .exe mods to start from.

My next goal is to work on decompiling functions where I've applied bug fixes to in previous binary patches. This may take some time due to size of some of the individual functions patched. I plan to push updates to the git regularly now that I have my build environment pretty much set up. So it will be a slow process of attrition until I've incorporated my original patches into the dll.

I'm hoping the community will review the logic to some degree to see if I've made any mistakes (as meticulous as I am, it can happen) or potential design/logic bugs I totally missed since I'm not a SMACX expert. Or just poor coding practices/choices since I'm not in dev space.

I agree with the sentiment regarding having new baseline standard but trying to manage the decompiling project on top of keeping my binary patches straight was too messy. Someone with some basic ASM knowledge could (fairly) easily do a binary compare between the two EXEs and apply my patches to the exe provided OpenSMACX dll. I would be happy to provide my own internal document related to patch offsets. I've already shared this with Induktio which I believe was incorporated with Thinker.

Out of curiosity, what % of the code is dissassembled ?

Hard to quantify. There are about 5500 functions related to engine classes and game logic. Some of those are just a few lines, some are massive. I've redirected about 132 functions. Now, not all of these functions are actually used including some of the ones I've redirected. Example, two functions A and B are pretty similar. B is never used. I will still add B to the dll for potential future use if it's only a few simple changes to A.

For comparison, read_rules() is 96/5500 and read_faction(Player, int) is 99/5500 sorted by largest functions. The largest function, enemy_move() is 14 times larger code base than read_rules().

That is a good idea to eventually put together a metric of # of functions decompiled and total bytes redirected from exe to dll.

[Induktio]

SMACX binary is 3 megabytes so unless a large part of that was statically linked libraries, it implies the original codebase was pretty big. There's so many implementation details in these game engines, the programs have to be pretty large. An equivalent of 5% of the original codebase reversed is probably not far off the target.

Now it might be a good idea to document how the patching process works in more detail. I have a bunch of different terranx.exe versions and none of them match the size of terranx_opensmacx.exe so I cannot run any binary diffs on it. With Thinker, the binary diffs are always stored in the repository. They can be obtained by diffing the default GOG binary and terranx_mod.exe, so that way one doesn't have to store large binary blobs in a source code repository. The majority of that patching is actually Scient's patch because Thinker only needs an entry in the dll import table to load with the game.

[scient]

The size difference with terranx_opensmacx.exe is the automated way I'm bulk updating the import table. Essentially each time I add a new set of imports from mangled function definitions, it adds 4kb to the file size. Eventually, I need to look into tweaking it to keep a few things consistent like address of imports and streamlining updating import table process

The code section should only have diffs with calls to new imports. So any decent hex editor you should be able to compare the two without any issue. The opcodes for redirects are as follows:
JMP DWORD PTR DS:[ADDR]

[Induktio]

It's somewhat of a problem if currently nobody else understands how the patching process works or how that code is generated. Apparently it doesn't have a lot changes on the binary level but the point is that it's impossible to review it with normal development tools. A big part of this project is how the program interfaces with the binary-only engine. Long term solution could be to make the patching process work in a way it can be described in plain text source code. For example Pracx has it's own patching tool that is compiled from normal source code. Just be aware Pracx's implementation relies on Visual Studio assembler semantics that are NOT portable to other compilers like gcc.

Regarding coding style I've been following this guide pretty closely. For example "The names of all types — classes, structs, type aliases, enums, and type template parameters — have the same naming convention. Type names should start with a capital letter and have a capital letter for each new word."

[scient]

I agree. I'll have to do some brain storming about best method to go about automating adding imports as well as making a guide on how you go about patching each function to redirect it to the dll.

[bvanevery]

https://github.com/b-casey/OpenSMACX

Your readme says you used Visual Studio 2017.  I do not see any .sln file or project files in the GIT repository.  That means there's no way to replicate your build process.

[Mart]

I took a look on github, it seems files are incomplete. I checked 2 .h files.

[scient]

Your readme says you used Visual Studio 2017.  I do not see any .sln file or project files in the GIT repository.  That means there's no way to replicate your build process.

I'm using default settings for a dll. You should be able to copy all source files into a default dll project and they'll compile. While I do have some different settings for my own project, they are purely there to aid with validating code (turning off optimizations, inline functions). I have tested this process and didn't run into any issues. If you do, let me know and I'll look into resolving them. Eventually I'll test compiling source with gcc and make any fix ups as needed. But for now at least, having project files are superfluous.

I took a look on github, it seems files are incomplete. I checked 2 .h files.

A lot of the interface code are struct place holders that I'll add to over time. The engine interface code isn't a priority for me except when needed to fix bugs. I have completely finished some of the low level classes related to memory/file IO and text but ones like the button and window classes will be slowly added to as needed. The reason for adding them all in was because I wanted to completely finish the function that parses alpha/x and faction files in. This required setting text for a couple buttons. Bulk of code for this release is in alpha.h/cpp.

I'm working towards knocking out some of the smaller prototype and unit functions that feed into larger combat functions. My goal for the next build is to have a couple bug fixes like the air scrambling crash resolved via the dll.

[bvanevery]

I would strongly suggest that you not regard a replicable build system as superfluous, even if it seems trivial to you.  I do not know what you are doing on your box.  I also suspect it's going to be more work to produce a working DLL than you seem to think it is.

[scient]

I agree that eventually when this project becomes more fleshed out that I will include some kind of project files. But for now, everything compiles with default settings in Visual Studio. I've tested this multiple times and tried to make it as simple as possible bar including the default project files. Part of the reason for not uploading my copies of project files is because I am doing environment specific stuff (removing optimizations, disabling inline functions, auto copying dll to my game directories, auto generating definition files) that would break on other peoples systems.

I just downloaded latest iteration of source, copied it to a separate Win7 Visual Studio 2015 VM I use for exploit development, created an empty dll project, copied source into it and compiled without a single warning or error. Then I copied the dll back over to my Win10 Visual Studio 2017 environment and it ran with v0.1 of patched exe without any problems. So it should really work with any modern version of Visual Studio. Compiling with Mingw-w64 is another story and for now isn't support. But if someone cannot compile the source as a dll with Visual Studio 2015+ I will eat my metaphorical hat and shoe horn in project files. 

[scient]

I added additional language into the README to make it explicitly clear that the project is using Visual C++ and compiled as an x86 (default) dll. I did make some assumptions that someone looking at code or familiar with SMAC/X would get this but I made that explicitly clear now. If anyone does run into issues compiling their own version let me know. For now, it is pretty pointless unless you also understand how to add to import table / patch exe to redirect to dll. That is something I'll definitely either figure out how to cleanly automate or put together guide so someone could theoretically roll their own copy of exe.

[bvanevery]

I do wonder about different versions of Windows SDK biting you in the ass.