Alpha Centauri 2

Sid Meier's Alpha Centauri & Alien Crossfire => Modding => Bug/Patch Discussion => Topic started by: scient on March 27, 2019, 04:42:34 AM

Title: OpenSMACX v0.2.3
Post by: scient on March 27, 2019, 04:42:34 AM
Initial release posted here including source:
https://github.com/b-casey/OpenSMACX

Let me know if you run into any issues, questions or comments about code. Or if you notice any mistakes or realize what some of the unknown structure values are. I've done pretty rigorous tests and countless rechecks to make sure the source mirrors original code except in areas I've improved logic or fixed bugs (as well as a couple auxiliary functions that are WIP).

You can copy over the exe and dll to any existing installation. The exe is a vanilla binary patched only with redirects to dll (ie. it does not include any additional bug fixes from my previous patches).

8/18/20: Added v0.2.3 notes
5/17/20: Added v0.2.2 notes
2/10/20: Added v0.2.1 notes
1/16/20: Added v0.2.0 notes
3/26/19: Init

v0.2.3
* Lots of additional analysis, clean up, refactoring, and optimization of the existing code base.
* Fixed four bugs within OpenSMACX code.
* Created new probe related source files (two net new functions, shifted one existing).
* Implemented one of the probe related fixes from my unofficial patch.
* Additional work on Path class (almost complete, 3 functions left).
* 55 net new functions across multiple areas.
* Cumulative decompiled and redirected function count: 420

Bug Fixes:
* Fixed a bug with OpenSMACX where a small change to how the Random class was reseeded from the
  default behavior caused rivers globally to shift. This happened whenever a Borehole or Condenser
  was built by any faction. I'm still unsure as to why. Something to review again in the future when
  more of the world code is decompiled.
* Fixed a bug with OpenSMACX where after selecting a random faction for a game, afterward on game
  start the game would throw a faction parsing error related to JENN282. This was due to the
  JENN282 code not being implemented in read_factions(). Originally, it looked like this was left
  over debug code so it was skipped. As it turns out, JENN282 is the internal faction identifier
  that SMACX uses for a random faction. Thankfully, this was the only place in the project where
  code wasn't implemented like this (excluding obvious optimizations). Multiple improvements to the
  code's performance and fixes for fringe cases were applied to this parsing code.
* Fixed an issue with OpenSMACX where two functions (tech_ai, rnd) weren't having rand() reseeded
  properly. This was caused by my_srand() only having the old built-in version of srand(). Both need
  to be initialized while the project straddles new and old code base.
* Fixed a preference parsing bug with OpenSMACX where "Time Controls" default value could have been
  set as "0" rather than "1". This was due to an oversight in how the return value was processed
  by prefs_get(LPCSTR, int, BOOL) when the useDefault parameter was set to true. This code is part
  of some of the earliest decompiled functions prior to my including comprehensive regression tests
  into the decompile process.
* Fixed a bug inside success_rates() where a probe wouldn't receive a penalty to its survival rate
  when it targeted a faction that has the Hunter-Seeker Algorithm secret project. Instead, its
  success rate would be given an erroneous 2nd penalty. This issue was somewhat masked because the
  2nd penalty happened after the success rate had already been written for display by the UI. This
  bug caused a lower success rate and increased survival rate of probes targeting an HSA faction.
* Fixed various issues related to the Scenario Editor undo/redo mechanic. Undo auto-saves weren't
  properly saved outside of the 1st one because auto_undo() didn't include the ".SAV" extension
  to process previous auto-saves. Undo auto-saves weren't properly removed because wipe_undo()
  didn't include the ".SAV" extension. Added in logic to prevent being able to attempt a redo before
  an undo has taken place. Fixed a map redraw issue after loading an undo or redo where units and
  tiles wouldn't be properly redrawn.

v0.2.2
* Lots of clean up and optimization of the existing code base.
* Initial work on the Path class.
* Added some Council related functions.
* 50 new functions across almost every game related area with a focus on Map and Technology.
* Decompiled and redirected function count: 365

Bug Fixes:
* Fixed a bug under certain conditions that the end game function to determine whether a faction is nearing a diplomatic win (Supreme Leader) would return incorrect results. The function aah_ooga() is called with the 2nd parameter set to -1 in certain instances. The original code would then attempt to use this value to check the pact status within the player_data structure of the 1st faction parameter. In these instances, it would actually be trying to do the diplomacy check against tutorialMoreBases value. There was a check to skip the pact check if the 2nd parameter was 0. The fix now accepts -1 or 0 to skip the pact check.

v0.2.1
* Some additional map functions to get ready for working on Path class.
* Miscellaneous clean up.
* Fixed an issue with the patcher script incorrectly patching best_specialist() and base_making().
* Decompiled and redirected function count: 315

Bug Fixes:

* Factions with the FREEPROTO flag (Spartans) will gain free retooling in their bases as long as the production switch is within the same category (unit to unit, base facility to base facility) and they've discovered the necessary tech for Skunkworks (default is "Advanced Subatomic Theory"). This is to resolve the issue with FREEPROTO factions never being able to gain the undocumented retooling ability of a Skunkworks found in base_making(). There is good indication this was likely an oversight and that the FREEPROTO flag should be the equivalent to a free Skunkworks for the faction's bases.

v0.2.0
* Veh, Map and Base related code that sets the groundwork to break down more complex functions.
* Engine classes: Font, Spot, Time.
* Decompiled and redirected function count: 308

Bug Fixes:
* Added an additional check to facility_avail() that prevents the Caretakers from being given the
  ability of selecting the secret project "Ascent to Transcendence". This goes against their core
  philosophy and would cause them to declare war on themselves if initiated.
* Sealurks no longer receive a movement penalty when moving through Sea Fungus. They are now treated
  the same as an "Isle of the Deep" as intended. This was likely an oversight to add the specific
  check to hex_cost().
* Added a new check inside facility_avail() that prevents building Paradise Garden if you have
  Punishment Sphere built or in queue. There is a check to prevent Punishment Sphere from being
  constructed if you have Paradise Garden but not vise versa. Paradise Garden and Punishment Sphere
  are antithetical facilities where you can build either one in any given base, but you were never
  suppose to be able to build both.
* Fixed a bug in DirectPlay multiplayer that likely caused a performance hit when moving units due
  to a faulty coordinate check. Whenever the stack_fix() function was called by moving a unit or
  other actions, it would cause every stack of units on the map to be sorted in a certain way (at
  least temporarily) as well as redundantly sorting the same stack once for every unit on the map.
* Removed an extra space displayed for certain prototype statistics (Ex. Transport Foil) shown in
  the Military Command Nexus via say_stats() function.
* Fixed logic inside offense_proto() and armor_proto() where under certain conditions the game
  would try to compare an arbitrary memory value against the Spore Launcher basic unit id. If they
  matched, non-PSI units would display incorrect offense and defense values. This happened any time
  you clicked on a non-PSI unit when calculating its offensive value. It was just incredibly rare
  the memory address value would match the Spore Launcher id. While it was unlikely this would be
  triggered by armor_proto() due to logic flow, added in a preventative bound check anyway.
* Fixed a bug in Time::pulse(void(__cdecl *)(int), int, uint32_t, uint32_t) where the Timer event
  could have persisted and executed continuously when it should have executed only once. It seems
  affected code branch was never used by the original game.
* Miscellaneous additional error handling or bound checks to various functions (see source).

Enhancement:
* Added the ability to set a reactor value (1-4) for #UNITS inside alpha/x.txt. To do so, add a
  comma after the Abil field with the value of the reactor you want for the unit. If no value or a
  value of 0 is set, it defaults to 1 (Fission Plant) like the original code.  For SMACX only, there
  are two exceptions where the default isn't 1 but 2 (Battle Ogre MK2) and 3 (Battle Ogre MK3). The
  Ogre defaults can still be overridden.
  Ex. "Colony Pod,..., 00000000000000000000000000,4" will give Colony Pods a Singularity Engine.
* Added some basic randomization for sea base name order. This mostly affects Pirates since
  the randomization method isn't great for small lists. Base name count increases once
  all names inside faction files and basename.txt are exhausted rather than stopping.

v0.1
* Parsing in alpha/x.txt and faction text files complete

Bug fixes:
* Fixed crash if BaseButton::set_name(LPCSTR input) parameter was NULL.
* Fixed bug with TextIndex having carriage return at end of "HEADER\r" breaking compare. This could
  have had significant performance increases back in 1999. Not so much in the time of cheap SSDs.
* Fixed bug if Heap::get() realloc ended up shifting memory to a different location (unlikely).
* Fixed crash in proto_cost() found via fuzzing/testing. Could be triggered by mods.
* Fixed Retool strictness in alpha/x.txt to accept a value of 3 (never free).
* Fixed setting alpha/x.txt probe teams can steal technologies now only to accept 0 or 1.
* Fixed setting alpha/x.txt humans always contact in net games now only to accept 0 or 1.
* Fixed setting alpha/x.txt humans always contact in hotseat/email games now only to accept 0 or 1.
* Fixed setting alpha/x.txt obliterating a base counts as an atrocity now only to accept 0 or 1.
* Fixed socIdeologyEffect overriding faction text files with -1, disabling AI faction "Emphasis".
* Fixed reactor power value being ignored when parsing. Game still won't use this value until
  more code is decompiled since hardcoded values are being used throughout.

Changes:
* Improved how Random::reseed sets new seed value.
* Improved performance of proto_cost() without changes to original logic.
* Removed debug code related to non-existent faction JENN282. Nothing of value. SMACX binary only.
* Various optimizations without changes to original logic.

Enhancement:
* The original code explicitly sets facility freetech value to disabled (-2) overriding alpha/x.txt. 
  It states in #FACILITIES alpha/x.txt: "Free  = No longer supported". This mechanic could have been
  removed for balance reasons or dropped due to time constraints. There is code that checks this
  value and sets the free facility only for new bases built after discovering the tech. It looks
  like existing bases do not get it. This will have to be reviewed more. For now, this mechanic is
  included as is. You can revert to vanilla behavior by modifying the four entries below in
  alpha/x.txt #FACILITIES with free tech parameter set to "Disabled".

   Recycling Tanks, 4, 0, Biogen, EcoEng2,   > free with "Adv.Ecological Engineering"
   Recreation Commons, 4, 1, Psych, SentEco, > free with "Sentient Econometrics"
   Energy Bank, 8, 1, IndEcon, QuanMac,      > free with "Quantum Machinery"
   Network Node, 8, 1, InfNet, HAL9000,      > free with "Self - Aware Machines"
Title: Re: OpenSMACX v0.1
Post by: Vidsek on March 27, 2019, 06:32:37 AM
    Congratulations on the release!!!!!

  It looks good as far as I could understand the codespeak  ;).

  Would combining this with your earlier bugfixes (and maybe other folks' stuff) be in the future?

  Someday, I'd love to see a patch that simply cleaned up as many bugs as possible without any frills, that might be  generally accepted by our community as a new standard baseline for all .txt and .exe mods to start from.
Title: Re: OpenSMACX v0.1
Post by: dino on March 27, 2019, 08:26:05 AM
Out of curiosity, what % of the code is dissassembled ?

The idea of having full smac source code to play with makes me want go back to learning C++ and programming, I've abandoned 15 years ago.
Title: Re: OpenSMACX v0.1
Post by: scient on March 27, 2019, 05:42:21 PM

  Would combining this with your earlier bugfixes (and maybe other folks' stuff) be in the future?

  Someday, I'd love to see a patch that simply cleaned up as many bugs as possible without any frills, that might be  generally accepted by our community as a new standard baseline for all .txt and .exe mods to start from.

My next goal is to work on decompiling functions where I've applied bug fixes to in previous binary patches. This may take some time due to size of some of the individual functions patched. I plan to push updates to the git regularly now that I have my build environment pretty much set up. So it will be a slow process of attrition until I've incorporated my original patches into the dll.

I'm hoping the community will review the logic to some degree to see if I've made any mistakes (as meticulous as I am, it can happen) or potential design/logic bugs I totally missed since I'm not a SMACX expert. Or just poor coding practices/choices since I'm not in dev space.

I agree with the sentiment regarding having new baseline standard but trying to manage the decompiling project on top of keeping my binary patches straight was too messy. Someone with some basic ASM knowledge could (fairly) easily do a binary compare between the two EXEs and apply my patches to the exe provided OpenSMACX dll. I would be happy to provide my own internal document related to patch offsets. I've already shared this with Induktio which I believe was incorporated with Thinker.

Out of curiosity, what % of the code is dissassembled ?

Hard to quantify. There are about 5500 functions related to engine classes and game logic. Some of those are just a few lines, some are massive. I've redirected about 132 functions. Now, not all of these functions are actually used including some of the ones I've redirected. Example, two functions A and B are pretty similar. B is never used. I will still add B to the dll for potential future use if it's only a few simple changes to A.

For comparison, read_rules() is 96/5500 and read_faction(Player, int) is 99/5500 sorted by largest functions. The largest function, enemy_move() is 14 times larger code base than read_rules().

That is a good idea to eventually put together a metric of # of functions decompiled and total bytes redirected from exe to dll.
Title: Re: OpenSMACX v0.1
Post by: Induktio on March 27, 2019, 09:15:16 PM
SMACX binary is 3 megabytes so unless a large part of that was statically linked libraries, it implies the original codebase was pretty big. There's so many implementation details in these game engines, the programs have to be pretty large. An equivalent of 5% of the original codebase reversed is probably not far off the target.

Now it might be a good idea to document how the patching process works in more detail. I have a bunch of different terranx.exe versions and none of them match the size of terranx_opensmacx.exe so I cannot run any binary diffs on it. With Thinker, the binary diffs are always stored in the repository. They can be obtained by diffing the default GOG binary and terranx_mod.exe, so that way one doesn't have to store large binary blobs in a source code repository. The majority of that patching is actually Scient's patch because Thinker only needs an entry in the dll import table to load with the game.
Title: Re: OpenSMACX v0.1
Post by: scient on March 27, 2019, 11:14:38 PM
The size difference with terranx_opensmacx.exe is the automated way I'm bulk updating the import table. Essentially each time I add a new set of imports from mangled function definitions, it adds 4kb to the file size. Eventually, I need to look into tweaking it to keep a few things consistent like address of imports and streamlining updating import table process

The code section should only have diffs with calls to new imports. So any decent hex editor you should be able to compare the two without any issue. The opcodes for redirects are as follows:
JMP DWORD PTR DS:[ADDR]
Title: Re: OpenSMACX v0.1
Post by: Induktio on March 28, 2019, 01:48:39 PM
It's somewhat of a problem if currently nobody else understands how the patching process works or how that code is generated. Apparently it doesn't have a lot changes on the binary level but the point is that it's impossible to review it with normal development tools. A big part of this project is how the program interfaces with the binary-only engine. Long term solution could be to make the patching process work in a way it can be described in plain text source code. For example Pracx has it's own patching tool that is compiled from normal source code. Just be aware Pracx's implementation relies on Visual Studio assembler semantics that are NOT portable to other compilers like gcc.

Regarding coding style I've been following this guide (https://google.github.io/styleguide/cppguide.html) pretty closely. For example "The names of all types — classes, structs, type aliases, enums, and type template parameters — have the same naming convention. Type names should start with a capital letter and have a capital letter for each new word."
Title: Re: OpenSMACX v0.1
Post by: scient on March 28, 2019, 02:59:34 PM
I agree. I'll have to do some brain storming about best method to go about automating adding imports as well as making a guide on how you go about patching each function to redirect it to the dll.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on April 01, 2019, 05:28:57 AM
https://github.com/b-casey/OpenSMACX

Your readme says you used Visual Studio 2017.  I do not see any .sln file or project files in the GIT repository.  That means there's no way to replicate your build process.
Title: Re: OpenSMACX v0.1
Post by: Mart on April 01, 2019, 10:20:51 AM
I took a look on github, it seems files are incomplete. I checked 2 .h files.
Title: Re: OpenSMACX v0.1
Post by: scient on April 01, 2019, 05:14:52 PM
Your readme says you used Visual Studio 2017.  I do not see any .sln file or project files in the GIT repository.  That means there's no way to replicate your build process.

I'm using default settings for a dll. You should be able to copy all source files into a default dll project and they'll compile. While I do have some different settings for my own project, they are purely there to aid with validating code (turning off optimizations, inline functions). I have tested this process and didn't run into any issues. If you do, let me know and I'll look into resolving them. Eventually I'll test compiling source with gcc and make any fix ups as needed. But for now at least, having project files are superfluous.

I took a look on github, it seems files are incomplete. I checked 2 .h files.
A lot of the interface code are struct place holders that I'll add to over time. The engine interface code isn't a priority for me except when needed to fix bugs. I have completely finished some of the low level classes related to memory/file IO and text but ones like the button and window classes will be slowly added to as needed. The reason for adding them all in was because I wanted to completely finish the function that parses alpha/x and faction files in. This required setting text for a couple buttons. Bulk of code for this release is in alpha.h/cpp.

I'm working towards knocking out some of the smaller prototype and unit functions that feed into larger combat functions. My goal for the next build is to have a couple bug fixes like the air scrambling crash resolved via the dll.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on April 01, 2019, 06:03:40 PM
I would strongly suggest that you not regard a replicable build system as superfluous, even if it seems trivial to you.  I do not know what you are doing on your box.  I also suspect it's going to be more work to produce a working DLL than you seem to think it is.
Title: Re: OpenSMACX v0.1
Post by: scient on April 01, 2019, 08:05:04 PM
I agree that eventually when this project becomes more fleshed out that I will include some kind of project files. But for now, everything compiles with default settings in Visual Studio. I've tested this multiple times and tried to make it as simple as possible bar including the default project files. Part of the reason for not uploading my copies of project files is because I am doing environment specific stuff (removing optimizations, disabling inline functions, auto copying dll to my game directories, auto generating definition files) that would break on other peoples systems.

I just downloaded latest iteration of source, copied it to a separate Win7 Visual Studio 2015 VM I use for exploit development, created an empty dll project, copied source into it and compiled without a single warning or error. Then I copied the dll back over to my Win10 Visual Studio 2017 environment and it ran with v0.1 of patched exe without any problems. So it should really work with any modern version of Visual Studio. Compiling with Mingw-w64 is another story and for now isn't support. But if someone cannot compile the source as a dll with Visual Studio 2015+ I will eat my metaphorical hat and shoe horn in project files.  :)
Title: Re: OpenSMACX v0.1
Post by: scient on April 01, 2019, 08:24:50 PM
I added additional language into the README to make it explicitly clear that the project is using Visual C++ and compiled as an x86 (default) dll. I did make some assumptions that someone looking at code or familiar with SMAC/X would get this but I made that explicitly clear now. If anyone does run into issues compiling their own version let me know. For now, it is pretty pointless unless you also understand how to add to import table / patch exe to redirect to dll. That is something I'll definitely either figure out how to cleanly automate or put together guide so someone could theoretically roll their own copy of exe.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on April 01, 2019, 08:41:49 PM
I do wonder about different versions of Windows SDK biting you in the ass.
Title: Re: OpenSMACX v0.1
Post by: scient on April 01, 2019, 08:59:13 PM
I do wonder about different versions of Windows SDK biting you in the ass.
I did run into a similar issue with redirecting mem_get() and env_open() between current Windows SDK and the one that SMACX is compiled with. There was an incompatibility that would cause the game to crash on start up. I tracked it down to free() in section of code I hadn't decompiled. That's why I have two additional "_old" versions of those functions inside general.cpp. The classes I've fully decompiled, including destruction, use the non-old versions. I then redirect the exe to call the old version that does a direct call to old SDK version of _fopen and malloc.

I'm not sure what version of SDK I have on my VS 2015 VM but it is not up to date so likely one that is a couple years old at this point. I don't think I'm using anything that is version specific that would require setting hard SDK limits.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on April 01, 2019, 10:57:20 PM
Nevertheless standardizing on your compiler and your SDK, even if other people "could" build it some other way by changing VS version and .sln settings, is good for your community development.  When someone comes to you with a weird crash, you get to ask them, "Does it work with VS 2017 and SDK version X" ?  Then everyone has to meet the same threshold of yep, it's sane, there's actually a problem other than me borking my dev environment.

Since you are doing weird things to dlls, I think this kind of engineering discipline does actually matter.  You aren't just writing "portable ANSI C code that runs everywhere".

Title: Re: OpenSMACX v0.1
Post by: scient on April 24, 2019, 02:42:50 AM
Added CodeBlocks project file thanks to induktio and cleared up applicable warnings when using gcc. Still chipping away at Veh/unit functions.
Title: Re: OpenSMACX v0.1
Post by: scient on May 16, 2019, 12:33:08 AM
I realized to continue working on Veh/unit functions that I needed a better understanding of how the underlying map code works. There are various inaccuracies in all the previous analysis that I've come across (mainly 63456_SMAX-MP.txt) in how it is handled internally. For those interested, have a look at map.h/map.cpp. They really did like to pack everything into as little bits possible. There is still one byte per tile (offset +6) that doesn't seem to be used. I think some of the default .mp files have it set but I haven't looked into it more. Otherwise, the only part I haven't fully documented are some flags in bit2.

Still chugging along with more Veh functions. Getting closer to being able to work on more combat related code that I know there are issues with.
Title: Re: OpenSMACX v0.1
Post by: vidyajames on May 28, 2019, 09:39:55 PM
A source port would be amazing, hopefully it can implemented with a cross-platform library like SDL to bring support to Linux and others.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 28, 2019, 11:36:24 PM
Feel free to get started on that.  I mean, a bunch of people looking at each other to do stuff, never got anything done in Open Source.

Myself, I'm not going to lead such a thing.  I have 25 years of $0 Open Source experience.  I know why things get done and why they don't.  I just spent 6 full time person months, 1 calendar year, coughing up my mod.  It doesn't even touch code! yet that's how much work it was in the real world.  And I did it.  All of it.  But man that will give you burn out.  Not throwing another year of my life away on something that makes me $0.  This is hardly the 1st time I've done this sort of thing either.  At least this time, I crossed the finish line and got people to use it and it looks like they will keep using it.  Next step for me is getting money for something.
Title: Re: OpenSMACX v0.1
Post by: vidyajames on May 29, 2019, 03:36:53 AM
I would be willing to assist, I do have experience with SDL and some comparable libraries. It's unclear if this code is supposed to be a pure vanilla reverse engineering project, or something that could be ported to SDL itself. Do you think it would be worth it to port this code as-is or should that come later?
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 29, 2019, 06:35:52 AM
It's unclear if this code is supposed to be a pure vanilla reverse engineering project, or something that could be ported to SDL itself.

Scient wrote in his readme.md:
Quote
A project to decompile SMAC/X to C++ with the long term goal of creating a full open source clone.

I think you can see that in principle, SDL can be applicable.  Particularly if you somehow manage to apply it.

In practice, you have a hell problem on your hands.  The 3D voxel game artwork is in a proprietary format that nobody has ever completely reversed engineered.  Some person(s) got started on that, and made enough progress to be able to insert a texture map into a model.  But that's hardly the same as being able to render or edit those models.  It's a real showstopper piece of work, jousting at crap like that.  It's where I definitely begged off.  For the effort to be spent, I'd be far better off writing a rendering display for a 4X TBS game from scratch.  In which case, why should I spend years messing with SMAC?  I'd rather spend years messing with a spiritual successor to SMAC, that I can make money selling.

Rather than trying to display the original artwork, it might be easier to just create similar artwork from scratch, in a modern open source usable 3D graphics format.  I did start researching what format would be applicable.  glTF (https://en.wikipedia.org/wiki/GlTF) seems like the leading candidate, particularly for an open source crowd.

I'm not clear on what open source 3D engines can display glTF, nor how well various 3D editors handle or export glTF.  I have been resisting learning Blender for years, because to me it looks like a real turd.  So much complexity to do things.  Not that I really know any ways to produce 3D art assets easily.  I'm rather much a dunce at that, even though I've written parts of 3D libraries and 3D device drivers before.  I'm even a visual artist in traditional media such as acrylics, but I have no digital art skills whatsoever.  It just looks like such a godawful chore that I simply never want to get started on learning Blender, the thing makes me want to puke.  I've wondered whether the upcoming Blender 2.80 will alleviate any of that.  I tend to research "Blender things" every several months when I remember I have this core problem of how to produce game assets.  Then I tend to get distracted by real life, like fixing cars, or making small amounts of money to keep my head above water.

Also I don't personally believe in Linux development, as there's no road to money that way.  I did Linux recently for 3 years solid, waiting for the Steam Machines to turn into something.  They didn't.  I had hoped for a better development environment under Linux, with better tools and ecosystem, and I didn't get it.  It was crap, at least for consumer software and game development.  I bet it's like the greatest network and embedding platform ever, but for consumer stuff, it's crap.  I begrudgingly went back to Windows, which is the epicenter of the 4X TBS game market.  Real customers use those systems, and real development will surely be in some kind of DirectX.

Don't get me started too much on OpenGL, it's a nightmare.  It's also dying and people would be fools to start any new project with it.  If some 3D engine already did OpenGL well enough and had the job done, then that might work ok, but it would also need to do glTF.  I can't tell you if anyone in open source has risen to that bar.  I knew more about such things about 6 to 9 months ago, at which time I said, to hell with this ain't gonna happen.

I was also deeply spending piles of time on my SMACX AI Growth mod back then.  I had a reason to avoid any and all pointless technical distractions.

Vulkan is future looking in the Linux world at least.  Problem is, nobody else gives a damn about it.  Apple went with Metal.  Windows has had and will continue to have DirectX.

If WebGL is actually usable for anything, and someone's got a 3D engine for it, and someone's got glTF support, that would be worth knowing.

The strongest open source game development engine out there right now seems to be Godot, but it is 2D oriented more than 3D.  Also what that mainly buys you is smartphones, which I really don't think is the point of the 4X TBS genre.

Anyways this is a very long way of saying, I can see why people would be motivated to simply hack on the game binary, rather than try to tackle the difficult problem of making an open source clone.  But hey, if your research and due diligence actually yields some kind of viable answer, and you produce some kind of proof of concept like 1 piece of unit artwork up on a screen, then you da man (or woman, or whatever).
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 29, 2019, 08:06:17 AM
I am now remembering that Godot is doing glTF, they are proponents of it.
Title: Re: OpenSMACX v0.1
Post by: vidyajames on May 30, 2019, 09:31:35 PM
While I'm a proponent of Godot due to it being FOSS, I don't think it'd be a good fit for a source port. It's a full on engine/workflow that would be more trouble to get everything working within it's "node system".

I completely overlooked the 3D models in the game, which would make SDL less than great for this use-case since SDL is 2D-centric unless you write raw OpenGL, which would hurt future Vulkan support. There are other libraries that have less experience with, but do lend itself towards mixing 2D and 3D within it's framework (ie. SFML, Allegro, raylib).

All of these libraries support OpenGL ES, and using tools like emscripten can be transpiled to WebAssembly and played in a browser. Easily portable to mobile and consoles, etc.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 30, 2019, 10:57:11 PM
But the point is Godot does glTF, their core developer is into it.  A solution is needed for the 3d art assets.  I seriously don't think you or anyone else is going to manage to reverse engineer the proprietary voxel format.  Unless that kind of grungy problem really really turns you on, and you're fully prepared to spend a really really long time banging your head on that, just because you're that kind of person.  Semi-sane people would make new artwork from scratch.  Completely sane people would move along and not take on any of this SMAC open source idea at all.

If you have some other way to come up with proof of concept on the 3D art assets that doesn't use glTF and/or Godot, power to you.  But it's the major roadblock.
Title: Re: OpenSMACX v0.1
Post by: vidyajames on May 30, 2019, 11:00:18 PM
I do think replacement 3D assets would be the way to go. raylib supports glTF: https://github.com/raysan5/raylib/pull/842
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 30, 2019, 11:12:06 PM
Those comments on the raylib glTF implementation say it's pretty bare bones.  Only a diffuse texture, no animations.  That means someone has to do the work of coming up with all of that in the real world.  You know of anyone out there with a more complete glTF implementation?  You may not be thrilled about the organizational pain Godot would cause you, but raylib would clearly cause you glTF pain at this point.  I think it's too much to expect that all that kind of stuff is just going to get done in open source land and be good / robust.  That feature landed on raylib 13 days ago.
Title: Re: OpenSMACX v0.1
Post by: vidyajames on May 30, 2019, 11:23:29 PM
It is very early, but we're also talking about a project that is also bare-bones, and hypothetical. I'm unsure why glTF specifically is important, other than to be especially bleeding edge in context to a 20 year old game. Godot is certainly not the only implementation of glTF 2.0, and is available as a stand-alone library: https://github.com/syoyo/tinygltf

I've worked a bit with Godot, but I've never tried GDNative (their method using C/C++ code) so I can't vouch for how well it works. The idea of re-implementing the whole game using their Node system seems to me like creating something from scratch using basic game logic derived from reverse engineering, instead of repackaging the reverse engineer code into a cross-platform graphics/audio/input library.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 31, 2019, 01:23:12 AM
I'm unsure why glTF specifically is important,

Because I just did due diligence on candidate 3D file formats and that's what I came up with as far as what would be a good idea nowadays.  Look you can read a wiki entry about glTF if you don't know much about it, why it might be important, and what its value might be.  You're free to come up with any other proof of concept option especially if you personally are going to do the work.  That's how open source goes, the tools that some author likes are what get used because they actually did the work.  I can't quote you chapter and verse of glTF right now, I did that research 6 to 9 months ago and I'm going on memory.  I know I'm not stupid and there's a reason I was considering that one.

Quote
other than to be especially bleeding edge in context to a 20 year old game.

Look if you just got the memo on SMAC having a proprietary voxel format that you're not going to get working in the real world, don't cast aspersions on other people's research or motives.  If you need to convince yourself that it's not actually possible or wise to reverse engineer SMAC's voxel format, do so.  The problem is obvious to any 3D person with experience, and I am one of those.  I've been paid to do it, if a long time ago.  I went bankrupt trying to write my own SMAC-like game 20 years ago, and this isn't my 1st open source rodeo, not by a long shot.

Quote
Godot is certainly not the only implementation of glTF 2.0,

I'm sure.  The question is, what's out there that's real world right now, not some moon mission of additional work before one has got any results to show for it.  I didn't get as far as researching every glTF option available.  Also 6 to 9 months ago, there were very few.  Things may have changed since then.

Quote
The idea of re-implementing the whole game using their Node system seems to me like creating something from scratch using basic game logic derived from reverse engineering, instead of repackaging the reverse engineer code into a cross-platform graphics/audio/input library.

I don't know why you think you're going to be able to get a front end for no work.  Maybe if you dig around in the SMAC binary you'll find some kind of old school dumb frame buffer implementation.  Maybe.  Maybe PRACX has some clues about how such things are put together.
Title: Re: OpenSMACX v0.1
Post by: vidyajames on May 31, 2019, 08:27:36 AM
I appreciate your desire to use a bleeding-edge file format for 3D scenes, despite the actual requirement being simple 3D models that would work fine in an OBJ file. I think the goal should be to build something standards compliant, and with maximum support. There are of course many paths to achieve glTF 2.0 support, at least there will, over time, once glTF 2.0 becomes more accepted as a supported standard among tools and frameworks.

I also appreciate your desire to reverse engineer a proprietary 3D model format. I think it would be wasted effort and it would be preferable if a standard file format (like an OBJ file) could be used which would allow for greater extensibility and modding support. The models included in the vanilla game are extremely low-poly (voxels!), it's not exactly something that would take a lot of effort to replicate.

The idea of using Godot as the basis of a source port is frankly laughable, and the justification for it being due to glTF 2.0 support (the dev is a believer!) is embarrassing. The node system is just a full-stop non-starter for this type of project. It's not the right tool for the job, it's not even the right type of tool, like using a chainsaw to unfasten a screw.

I don't understand your point about a front-end. you mean the menus? you mean the actual rendering? If you have an actual source port of the game logic, why would you be hacking around with the binary? You'd be re-implementing it into a rendering framework. PRACX is a hack that would become redundant.

I kinda get the feeling you don't really know what you're talking about, and you've never used any of these things before. When I speak, I'm not just talking out of my ass about the latest meme tech, I've actually implemented these things before. I have to assume you've used Godot before, but frankly it seems questionable based on some of the things you've said.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 31, 2019, 12:32:10 PM
I also appreciate your desire to reverse engineer a proprietary 3D model format.

Somehow communication between us is not working.  Please reread what I've written before.  I have no desire whatsoever to reverse engineer a proprietary 3D model format.  My point was to warn you how hard it would be to do that.  What you personally want to do, is not at all clear.

Quote
The idea of using Godot as the basis of a source port is frankly laughable,

Whatever dude.  Do what you want, nobody's stopping you.

Quote
If you have an actual source port of the game logic, why would you be hacking around with the binary?

If you want to use the original 3D art assets, in the real world you'll need to use the game binary to display them.  PRACX might inform you how it works.  Or it might not.

If you want to recreate the 3D art assets in some common file format, and you want it to be cross-platform, then you're going to need some kind of 3D engine to display stuff.  It might as well be a 3D engine that has got "legs" and isn't going to go the way of the dodo bird when someone gets tired of working on it.  Godot is one such engine.

Quote
When I speak, I'm not just talking out of my ass about the latest meme tech,

If you're "all that", you don't need to be talking.  You can be doing.
Title: Re: OpenSMACX v0.1
Post by: vidyajames on May 31, 2019, 02:33:04 PM
It's now overwhelming obviously you've never worked with any of these technologies. Maybe you should stick with editing txt files while refusing to adopt the community consensus patches. ;lol

As for working on this project, that leads me back to my original question about the state pure reverse engineering effort. This effort that you suggest should be abstracted and re-interpreted to fit in Godot's node system, because... glTF 2.0?  ???
Title: Re: OpenSMACX v0.1
Post by: bvanevery on May 31, 2019, 06:30:10 PM
The only thing overwhelmingly obvious is I can't imagine helping you with any project you might lead.  Which I don't think you're actually going to do anyways.
Title: Re: OpenSMACX v0.1
Post by: scient on June 18, 2019, 12:45:47 AM
Trying to manage manually patching the function redirects to the dll has started to become cumbersome now that I'm redirecting over 250 functions. I put together a python script that will handle all the patching after imports are added in. That process is also automated (C++ code), although I'll have to modify the program slightly to take input parameters rather than hardcoded import list. I plan on hooking that into the python script so everything is done in one shot. There are a few things I need to work out but I don't think it will be too difficult.

So far it is working smoothly! I finished validating everything got patched correctly. I even found a mistake in my manually patched version (unused function) that was off by 1 byte. I applied the same script/process to the last version of my bug fix exe and that seems to run without any issues. I'm going to cut off working on new functions and wrap up all the existing ones I've started or that need testing. After that I'll put together a 2nd release, this time with a script that will hopefully make it easier to utilize the dll.
Title: Re: OpenSMACX v0.1
Post by: Induktio on June 18, 2019, 09:57:31 PM
Trying to manage manually patching the function redirects to the dll has started to become cumbersome now that I'm redirecting over 250 functions. I put together a python script that will handle all the patching after imports are added in. That process is also automated (C++ code), although I'll have to modify the program slightly to take input parameters rather than hardcoded import list.

So does it mean this build script will be able to take the unpatched game binary and make it usable with OpenSMACX's dll? I'm not sure what build steps you are referring to here. Although I agree it's very important to create a reproducible build process, so that for example one can take the GOG binary and then patch it to be usable with OpenSMACX. Otherwise the source code for this project would be incomplete unfortunately.
Title: Re: OpenSMACX v0.1
Post by: scient on June 19, 2019, 12:12:21 AM
That's correct. As long as the GOG version is using the latest SMACX binary (patch 2.0 I think) unpacked. If the .text code section lines up you're good. My script is using file offsets to patch rather than a search and replace patch which would take a lot more work to put together and manage. I believe the GOG version is just the unpacked exe with a no cd crack applied.

Minimally, I'll verify offset patches are being applied correctly to the vanilla exe, last iteration of my patched exe and GOG version. I've never fully looked at kyrub's or yitzi's patch code but if they're messing with functions or redirecting code a lot this method will likely not work smoothly. It would also remove any additional functionality since I'm only applying bug fixes and minor mod-ability enhancements.
Title: Re: OpenSMACX v0.1
Post by: DrazharLn on August 15, 2019, 11:58:05 PM
Re: vidyajames,

Making a new UI for SMAC would be cool. I started making one in Godot (in GDScript) several months ago but didn't get far, partly because games engineering is new to me, partly because it was the first time I was using Godot and it took me a while to find a development workflow that worked for me.

You can use Godot without faffing with the UI and the nodes. Just instantiate all your nodes in code as you need them from one root node. This also makes the project much easier to meaningfully version control.

I'm not sure you'd want to graft a feature-complete UI on to an opensmacx backend (What does the game do that isn't interaction code and isn't pretty trivial? The terrain generation?), but I think it could save a lot of effort and be valuable to do something like replacing the map renderer and compositing the other UI over the top. PRACX has already reversed most of the relevant calls as well.
Title: Re: OpenSMACX v0.1
Post by: Induktio on August 17, 2019, 10:31:46 AM
It is fairly useless to use Godot as a basis for recreating any art assets from the original game because of at least one reason. Importing and redistributing art assets from the game is not possible because they are copyrighted content. Open source game ports cannot do that.

Instead these kinds of projects usually build compatibility to support reading the original files while playing with an open source engine. I don't think there's many successful OS game ports that tried to recreate every asset from scratch to avoid copyright problems, it doesn't work like that. Just implementing binary compatibility with the original game assets is much faster and avoids all the licensing problems. You might or might not use Godot for something but it does zero for reversing engineering.
Title: Re: OpenSMACX v0.1
Post by: DrazharLn on August 17, 2019, 06:17:47 PM
All this stuff about Godot is very confused.

As I see it, there are a variety of reasons for building OpenSMACX:

- For the fun of it
- To eventually build an open-source clone of the SMAC engine for posterity
- To make modding the game easier (because more of it can be done without reverse engineering knowledge)
  - e.g. complete control over the AI code
  - Finally fixing stuff like the MP art/faction issue
  - Fixing TCP/IP multiplayer
  - Having a better map renderer
  - Better UI generally
  - Pluggable/composable mods

I think something like Godot could be used if the project gets far enough. In that scenario OpenSMACX is just being used to give us texture buffers of the UI, the map state, etc.

Anyway, there are lots of other potential choices and Godot isn't a particularly sensible option. I used it because I wanted to see if I could make a simple version of SMACX from scratch as a learning exercise, but it was hard and I am way too busy ;)

If someone did make a new renderer, I think it would be a good idea to redo some of the map art and sprites, especially to have a higher colour depth.

I wouldn't be very concerned about distributing this publicly despite it being derivative work. "High resolution texture packs" rarely attract legal notice and I really doubt that EA or Firaxis care any more.

We should really find someone to talk to at Firaxis. We could ask them for the source code (lol) and ask if they kept high resolution copies of the movies and other artwork. It would be really nice to get high def version of the SP movies. Obviously the CGI would still be dated, but I think that's fine.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on August 19, 2019, 04:24:51 PM
Just implementing binary compatibility with the original game assets is much faster and avoids all the licensing problems.

Except in the case of SMAC.  The voxel format is proprietary and in the real world, you are unlikely to reverse engineer it.  Someone somewhere did make a start on it, there's at least one GitHub project.  They didn't get anywhere remotely near to functional proof of concept.  It is far faster in SMAC's case to recreate the game's voxel artwork from scratch.  YMMV.  I'm not going to attempt this.  I already looked into the feasibility of this a year ago or more.  For the effort to be spent, I will work on a new commercial title and sell it for money.

Godot is a red herring.  I will go so far as to say that vidyajames objections to using Godot, are baseless.  Node architectures don't make or break open source projects.

The reason I brought up Godot at all, is it currently has support for glTF (https://en.wikipedia.org/wiki/GlTF).  The latter is a modern 3D file format.  It has merit for long term sustainability of recreated art assets.  Even Microsoft is into it.  They contributed some kind of DirectX viewer sample.  IMO that's saying something.

I never said one must do anything in Godot.  I said Godot has support for glTF now, and that that's advantageous.  Other open source engines may have glTF support later but Godot has it now.  And Godot itself isn't going to vanish as a project any time soon.  It has legs, it's got the buzz, people are using it.
Title: Re: OpenSMACX v0.1
Post by: DrazharLn on August 19, 2019, 08:28:19 PM
I don't see any reason to favor glTF in particular. In any case, I don't think this is relevant to OpenSMACX.

OpenSMACX doesn't need to make a decision about 3D assets now or perhaps ever (if it does not replace the 3D renderer). If someone does want to make new assets they can use whatever open standard they like because they're reasonably easy to convert and converters will be available for a long time.

You may be right that redrawing the 3D models would be easier than reversing the vector file format (though as I recall Ford actually reversed the whole format and it just needs someone to export it to a more common vector format or write their own marching cubes code), but it's also not really relevant to what Scient is working on now.

I think at this early stage of the process the discussion should be led by those who actually want to contribute code to the project.
Title: Re: OpenSMACX v0.1
Post by: scient on August 21, 2019, 01:51:09 AM
My own internal priorities:

1. Game logic code. Within this, I've been heavily focusing on Veh code to fix bugs with combat (among other things). I'll eventually turn my focus to Bases, Map, Tech, etc. Sometimes I'll get distracted and do few random functions here or there.
2. Generic utility functions, parsing of game text files. I've done most of these already. Save files are probably next in this area so I can take over loading/saving of game state.
3. Core game engine code (memory management, strings, logging, etc). The big one here will be Buffer class that is used by everything UI related and most of complex non-UI classes (~160 functions for this class alone). I'd include voxel parsing here as well. Honestly, voxel classes (Caviar/CaviarData) aren't that complex in the grand scheme of things (from my cursory look through functions). It won't be too difficult to break them down. Trying to reverse engineer the entire format is something else entirely and I personally won't be doing that.
4. Core UI engine code > Once you get base classes out of the way (BaseWin, BaseButton, BasePop) it should make it easier and have cascading effect on smaller derived classes.
5. Sound classes, ambiance > hooking into sound.dll. This would probably tie with #4.
6. Non-PBEM multiplayer

Also, per release my aim is to complete a few non-game logic classes. These are the ones I don't enjoy doing as much so pushes me to actually get through them. Slated for 0.2 are Font, Spot and Time. I've actually completed all of these and just need to test Time.

On that note, haven't had as much time to work on project juggling training for work certifications. Also, it's summer. But as cert stuff is winding down, getting some more bandwidth to work on project again. :)
Title: Re: OpenSMACX v0.1
Post by: DrazharLn on August 21, 2019, 09:27:31 AM
Sounds great, scient :)

I'm particularly interested in save files and multiplayer. If save files are reversed we can possibly fix the faction graphics bug and if the multiplayer code is accessible we can make it so players don't need to faff loads to play online and could also support e.g. Thinker in multiplayer.
Title: Re: OpenSMACX v0.1
Post by: Induktio on August 24, 2019, 04:10:19 PM
The main thing that prevents Thinker from being played in multilplayer is the data synchronization across networks. Network desyncs are also one of the worst areas to debug without good tools, so it's not implemented at the moment. Objectively it probably wouldn't require much new code to implement but it would be very difficult to debug. The majority of that stuff deals with the unit movement code.

In the long term, if this project were to become a standalone game engine, importing the art assets some way or another would become unavoidable. Before that it's not really important though if Scient just keeps patching the old game engine.
Title: Re: OpenSMACX v0.1
Post by: bvanevery on August 25, 2019, 04:04:31 PM
importing the art assets some way or another would become unavoidable.

Recreating 3D unit art assets from scratch isn't importing them.  If they were sufficiently different from the originals, they would even be free of copyright restriction.

But this is a project for someone motivated to do it.  I myself have serious misgivings about all current forms of 3D unit creation.  It might spur me to finally create a programming language to do it.  Every time I stare at the blank screen of something like Blender, I basically can't stand it.  Maybe Blender 2.80 alleviates that pain, but to this noob, it still looks like screenfuls of junk and some huge learning curve, that I simply have never wanted to do.
Title: Re: OpenSMACX v0.2
Post by: scient on January 16, 2020, 07:31:12 PM
I've pushed v0.2 changes to git plus a patching script for import/redirects. I've tested the script on the GOG version and latest exe of my unofficial patch without any issue. There will be one regression with using my patch related to FREEPROTO flag and Spartan retooling that I haven't implemented. While reviewing it, I don't remember original justification for change so need to look into it more.
Title: Re: OpenSMACX v0.2.1
Post by: scient on February 10, 2020, 09:09:16 PM
Released a new minor version since I noticed a mistake in my patcher script used to create the release exe. Re-implemented the FREEPROTO/Spartans retool strictness via code rather than binary patch.

Also, finished a few map related functions used to determine underlying rockiness, resource bonus and pods. I don't quite fully understand some of the logic but it correctly mirrors the game code. If anyone has an idea what is going on, let me know. Could just be a way to randomize things a little.

https://github.com/b-casey/OpenSMACX/blob/master/src/map.cpp#L489
https://github.com/b-casey/OpenSMACX/blob/master/src/map.cpp#L538
https://github.com/b-casey/OpenSMACX/blob/master/src/map.cpp#L577
Title: Re: OpenSMACX v0.2.1
Post by: MercantileInterest on February 21, 2020, 10:57:26 PM
I'm very happy with this project, Scient, largely in the hope of getting more features. Please, keep us updated regularly. I don't really understand it, though. What do we have to learn to actually edit the decompiled code?

There are a few things I'd like to incorporate in my next mod. Will list them here in order to talk about feasibility.


1. Support in excess of social engineering depletes energy rather than minerals.

2. All formers deplete 1 mineral from home base per turn, independently of social engineering. (One of Yitzi's ideas before he disappeared.)

3. Gaians gain +1 nutrient in forest squares.


Again, thanks!
Title: Re: OpenSMACX v0.2.1
Post by: bvanevery on February 21, 2020, 11:29:02 PM
3. Gaians gain +1 nutrient in forest squares.

That's seriously overpowered.  In my mod, Tree Farms are the basic midgame method of boosting minerals production, as I don't allow factories, supply crawlers, and thermal boreholes until early late game.  Ergo, I'm rather familiar with the value of the food the Tree Farm gives.  You're proposing to start a faction off with that advantage at the beginning of the game.  The reason that a fungus nutrient advantage isn't a big deal, is because fungus doesn't produce anything else useful for quite some time.  The Gaians' basic advantage is that when other land is compromised, ruined, pillaged, squatted upon, etc., they won't starve.  Whereas, the minerals of forests are incredibly useful to production.

Unless your mod already took minerals away from forests?

While on this sort of subject, removing the Pirates' +1 minerals on ocean shelf squares would be a good idea, replacing it with something else I suppose.  Not sure what.  But their +1 is overpowered.

Title: Re: OpenSMACX v0.2.1
Post by: MercantileInterest on February 21, 2020, 11:55:59 PM
My current mod is Binary Dawn--essentially a rebalancing of SMAX intended to keep players from snowballing in MP.

My next mod (still very much in progress) aims to restructure the core experience to feel less like CivII reskinned for space and more like a desperate colony of Earth's finest stranded on a distant and hostile world. Nutrients and minerals will be relatively easy to procure whereas shortage of energy will be the defining struggle of early game. In early human history, populations expanded slowly, limited by disease and primitive farming techniques. The colonists on Chiron, in contrast, will be largely immune to endemic diseases evolved to interact with an entirely different form of life and will access to sufficient supplies through advanced agriculture. As such, bases will grow quickly but the resources required to found new bases with all their complex technical systems and life support will be enormous. My tests so far have made me pretty happy with the colony module costing 25 rows of minerals. However, I've reduced the default number of nutrients per line from 10 to 8 and increased the amount of nutrients granted by a farm to a total of +2. Thus, a farmed rainy tile with a solar collector on low ground gives 4-1-1. Have made a number of other relevant changes as well. By default, base squares don't grant any resources, until boosted by econ, recycling tanks, rivers, etc.

With all of this, you can see how +1 nutrient per forest square fits in quite nicely.
Title: Re: OpenSMACX v0.2.1
Post by: bvanevery on February 22, 2020, 12:27:28 AM
So it seems you're interested in serious bigpox.  25 * 10 = 250 minerals for a human player, making a Colonist cost more than an early Secret Project in the stock game.  That might be ok in MP, but if you're expecting the stock AI to handle this, it's suicide.  The AI gets plenty of Colonists killed as is, with much cheaper costs to take the colonization gamble.  Its ability to successfully navigate fungus is really bad.  I've even  contemplated giving all factions 4 colonists to try to cut down on the deaths.  But, my mod makes Clean Reactors available from the beginning of the game, which at least allows superfluous "milling" Colonists not to drag down SUPPORT and cripple the AI players.

You would need a binary patched game with much better colonization AI, for an expensive Colonist to have any chance of working.  Even if an AI has a 50% production offset, it's a 125 mineral Colonist.  Thinker Mod etc. might almost be that, but it surely doesn't assume a 125 mineral Colonist, so YMMV.
Title: Re: OpenSMACX v0.2.1
Post by: scient on February 25, 2020, 06:55:16 AM
I'm very happy with this project, Scient, largely in the hope of getting more features. Please, keep us updated regularly. I don't really understand it, though. What do we have to learn to actually edit the decompiled code?

Glad to hear! Me too. Eventually the end goal is to have the source that can be expanded upon. C++, I'm converting it back into the original (albeit updated) language it was coded in.

There are a few things I'd like to incorporate in my next mod. Will list them here in order to talk about feasibility.


1. Support in excess of social engineering depletes energy rather than minerals.

This was easy to find. It's in a function for initializing a base.

(https://i.imgur.com/Cd0yks3.png)

You would then change the mineral base value to one of energy related base variables. Not 100% on which from quick look.

2. All formers deplete 1 mineral from home base per turn, independently of social engineering. (One of Yitzi's ideas before he disappeared.)

This would likely be in base_upkeep() or base_support(). It would be easy to determine former's tied to home base and put in some calculation once these are done.

3. Gaians gain +1 nutrient in forest squares.
This would be in crop_yield() which is used to calculate nutrients. There is already logic branch for tree tiles related to Hybrid Forest. So adding in an additional check for specific faction would be easy.


These all sound doable. None of functions mentioned have been decompiled yet but once they are I'd be happy to give you some help in forking the existing project to make mods.

Title: Re: OpenSMACX v0.2.1
Post by: MercantileInterest on February 26, 2020, 06:04:46 AM
Hey, thanks, man!
Title: Re: OpenSMACX v0.2.1
Post by: MercantileInterest on February 26, 2020, 08:59:15 PM
Another thought: scripting in additional starting units, so factions begin with extra colony pods and free formers. This is currently possible with txt but each unit added in this way takes up limited faction customization slots.
Title: Re: OpenSMACX v0.2.1
Post by: bvanevery on February 26, 2020, 11:51:42 PM
I don't agree with free formers.  In fact it's a perfectly viable opening strategy to not obsess about terraforming right away.  I often just scout first and rely on resource specials to flip from size 1 to size 2, chucking out the colony pods I want.  Only worry about terraforming once there's no unoccupied land next to core cities anymore.  This is working particularly well with the Pirates as of late.  I go pop sea colony pods and typically get completions for all kinds of stuff.

Giving everyone 4 colony pods, like I said, I've considered that.  But my mod version 1.40 seems pretty solid, and needs public feedback to be shown otherwise.

There are so many factors when modding!  We can get into an idea of "well I'd like this, and this, and this and this and this," but as we change factors around, what we need changes too.  Everything is interconnected.  One ends up working "the whole canvas".

Most of my factions don't get free starting techs.  I did that to slow down the early tech economy.  What's the point if everyone's got one?  Only exception is the Pirates, they do get Doctrine: Flexibility.  And the Aliens, since you can't take Progenitor Psych or Field Modulation away from them.  I did reassign what they give you.  PrPsych gives Hypnotic Trance.  Field Modulation gives the Cloaking Device, which I think is useless in practice.  AI just cheats and spots your unit.

So a side effect of my tech policy, is my customization slots have a little less pressure on them.
Title: Re: OpenSMACX v0.2.1
Post by: scient on February 27, 2020, 02:49:50 AM
Another thought: scripting in additional starting units, so factions begin with extra colony pods and free formers. This is currently possible with txt but each unit added in this way takes up limited faction customization slots.

Design decisions aside, this is doable. There is a setup_game() function that at the end either calls time_warp() for an accelerated games or balance() for a standard game. These in turn call veh_init() to create units. I'll keep in mind to let you know once these are decompiled.
Title: Re: OpenSMACX v0.2.1
Post by: MercantileInterest on February 27, 2020, 04:15:32 AM
After asking you to think about this, it occurs to me that making the UNIT feature an unlimited faction bonus would be a much more precise and useful way of doing this.
Title: Re: OpenSMACX v0.2.1
Post by: MercantileInterest on February 27, 2020, 04:57:43 AM
I once had the opportunity to observe elite cloaked hovertanks in a multi-player match. You can't imagine the horror such a weapon instills without having seen it with your own eyes. In my last game, I made extensive use of submarines--not nearly so good, since foils stop their move after attacking, but they still gave the ability for my navy to strike at its pleasure rather than on an equal playing field.

As you can see, the multiplayer experience is much different than what you're familiar with. This is what the extra colony pods are about. Modded gameplay tests are fun so far but early gameplay would be somewhat sluggish when played by PBEM over two months. Remember that colony pods cost 70 minerals (this is what a cost of 25 in txt comes out to.) Formers, for their part, can only build farms and roads, with other improvements scattered across early techs. Without good terrain, farms are essential because bases provide 0-0-0 resources as a default. Without the ability to immediately farm, a bad landing zone could cripple a faction.

As a side note, razing a base suggests that each pop equals 10,000 citizens. This means that the Unity carried 160,000 citizens. If that number is reduced to a mere 1,000, the total carrying capacity of the Unity then becomes a much more reasonable 32,000, even with each faction starting with 4 colony pods.

Scient, if you like, I'll stop cluttering up your thread with my reasonings and restrict talk to features.

Title: Re: OpenSMACX v0.2.1
Post by: bvanevery on February 27, 2020, 06:29:36 AM
Without good terrain, farms are essential because bases provide 0-0-0 resources as a default.

That's not stock behavior.  Why change it?
Code: [Select]
#RESOURCEINFO
Ocean Square,     1, 0, 0, 0,
Base Square,      2, 1, 1, 0,

I gave the AI Cloaked Recon Rovers and such.  They're annoying but not deadly.  Cloaked units are only a problem if they have weapons big enough to hurt you.  It's not so easy to get ahold of bigger weapons in my mod.  I don't think anyone has tried to play my mod in multiplayer though.  I think there are lots of ways to sneak around.  Cloaking is just another one.


Title: Re: OpenSMACX v0.2.1
Post by: mmmniple on March 31, 2020, 12:56:06 PM
Hello,great to see the proyect go on. :danc:
I think than it is better avoid enter on opengl programing,and use SDL as the game is 2d (if we excluded the voxel , but they are render using software) as it happens with the great Blade Runner which has been reverse enginer and supported by scummvm developers.
(On their forum they discuss the 3d slice model for add on scummvm or on residual vm.)

I tried RE caviar voxel but sadly i was no succesfull.Curiously just before i saw this thread, I had just post some request on two places where they are some experts reverse enginering 3d formats :

https://zenhax.com/viewtopic.php?f=5&t=13381

and

https://forum.xentax.com/viewtopic.php?f=18&t=21948

I hope we have any answer soon.

In case it did no happens,we always could make a bounty ,as you can see on Messiah case by MrAdults:

https://forum.xentax.com/viewtopic.php?t=6763

"
I happened to snag this one on a bounty earlier this week. Because the guy offering the bounty is offering it to have the format and specs opened up to the community, as he isn't a massive dickchugger like everyone who's ever offered a "bounty" here. That means Messiah will be supported in the next build of Noesis, and loader code will be forthcoming."

what people thinks about it?

a question : have you the part of caviar render library extracted/exported ?

Thanks
Title: Re: OpenSMACX v0.2.1
Post by: bvanevery on March 31, 2020, 05:23:10 PM
Bounties mean paying someone money, right?  Leaving aside the question of where one would get the money, there's the question of how much money, from the standpoint of the person who does the work.  I foresee a lot of labor hours, for this voxel format reverse engineering problem.  That's based on the various open source efforts to reverse engineer it, that have existed in the past, but that stalled out and didn't get very far.  Someone demonstrated the ability to texture the voxel artwork, they managed to figure the texturing out.  But... if it didn't get solved already by the usual open source means, then it doesn't smell like an easy problem.  So I really have to wonder, if anyone can offer a bounty high enough, to get that job done in the real world.  YMMV.

This is why I broached the subject of creating 3D models from scratch.  But... as all of my effort presently will go into a commercial title, and not this timesink, all I can say is "good luck" and hope one of you folks figures out what you want to do someday.  All I can do is give the advice I gave.  I've given 2 years of my life to "the cause" of SMAC and that's enough.
Title: Re: OpenSMACX v0.2.1
Post by: mmmniple on April 01, 2020, 05:26:53 AM
Hello, first say excuse me : it was a idea I had only thinking on how it could help the proyect after create my posts while I was looking on the website and found the Messiah one.

I understand you plenty , and it would be a shame you stop for my fault : the proyect is, and always will be open source. You has no wasted 2 years.
As you can see on the post I created, I did no write anything about a bounty.
Excuse me
Title: Re: OpenSMACX v0.2.1
Post by: bvanevery on April 01, 2020, 06:34:48 AM
I understand you plenty , and it would be a shame you stop for my fault : the proyect is, and always will be open source. You has no wasted 2 years.

Actually, I meant my own modding project, that I put 2 years into.  Not OpenSMACX.  Never touched binary code, only did *.txt modding.  And it's taken 2 years.  Can't call it open source per the Open Source Definition (https://en.wikipedia.org/wiki/The_Open_Source_Definition) because my license restricts the field of endeavor to noncommercial.  I put it under CC-BY-NC license, figuring if anyone ever stood the remotest chance of being paid for it, it jolly well should be me.  But hey, in the real world, people can mod my mod, long as they give me credit.  Good enough for what people will actually do with this old game.

Quote
As you can see on the post I created, I did no write anything about a bounty.

Yeah, I guess you asked "what people think" about it.  I think bounties are unlikely to work, because it's way more labor than the amount of money probably anybody's going to offer.
Title: Re: OpenSMACX v0.2.2
Post by: scient on May 18, 2020, 01:27:36 AM
New update available. Not much in the way of bug fixes but lots of additional code and clean up.
Title: Re: OpenSMACX v0.2.3
Post by: scient on August 18, 2020, 11:10:32 PM
Created a new release. I've moved to trying to push changes more frequently so code base is generally within a day or two from my local copy. This does fix a couple of bugs in my code. The biggest one thanks to induktio for reporting. A small change in the Random class code caused all sorts of map weirdness. I'd say this release is stable enough to play games with. Definitely let me know if you notice or run into any issues!
Title: Re: OpenSMACX v0.2.3
Post by: EmpathCrawler on September 19, 2020, 04:25:48 PM
Some lazy Saturday morning reading led me to a potential bug fix here:

https://github.com/b-casey/OpenSMACX/blob/defe36bdb1604c738f9cdf842ce1c8caabc456f1/src/faction.cpp#L222

Circumstantial evidence suggests that this Psi Gate prereq check in the climactic_battle function is legacy code left over from when Psi Gates apparently had some role in a Transcendence Victory. I wonder if it makes the AI unduly agitated too early at the arbitrary tech of Matter Transmission?

See this thread on reddit regarding the evidence: https://old.reddit.com/r/alphacentauri/comments/iua8v2/miriams_psiicide_put_in_another_light/
Title: Re: OpenSMACX v0.2.3
Post by: scient on September 20, 2020, 04:46:55 AM
Some lazy Saturday morning reading led me to a potential bug fix here:

https://github.com/b-casey/OpenSMACX/blob/defe36bdb1604c738f9cdf842ce1c8caabc456f1/src/faction.cpp#L222

Circumstantial evidence suggests that this Psi Gate prereq check in the climactic_battle function is legacy code left over from when Psi Gates apparently had some role in a Transcendence Victory. I wonder if it makes the AI unduly agitated too early at the arbitrary tech of Matter Transmission?

See this thread on reddit regarding the evidence: https://old.reddit.com/r/alphacentauri/comments/iua8v2/miriams_psiicide_put_in_another_light/

Hmm, that could be. There is definitely some odd behavior with some of these end game related functions. For example, aah_ooga() basically doing nothing in climactic_battle().

I recently pushed drafts of two net new functions that call climactic_battle().
great_satan(): https://github.com/b-casey/OpenSMACX/blob/master/src/faction.cpp#L148
at_climax(): https://github.com/b-casey/OpenSMACX/blob/master/src/faction.cpp#L241

In tech_val(), it's used to prioritize FAC_ASCENT_TO_TRANSCENDENCE tech:
https://github.com/b-casey/OpenSMACX/blob/master/src/technology.cpp#L388

I've attached a screenshot of the other references to climactic_battle(). I'm planning on doing rankings() soon. The other three (base_build, probe, tech_research) will be in future. I figure once all calls to things like climactic_battle() are done it will be easier to figure out impact and see if any tweaks or fixes are needed. But that does make sense about potential for AI to get prematurely aggressive thinking end game is near.

These is also reference to Psi Gate inside base_hurry where I think AI (and maybe gov controlled bases) will potentially prioritize and hurry building Psi Gates if end game is detected.
Title: Re: OpenSMACX v0.2.3
Post by: EmpathCrawler on September 23, 2020, 10:51:02 PM
https://github.com/b-casey/OpenSMACX/blob/d7cf7140dfa5e24cd3462f7d37b0b65c2c591159/src/game.cpp#L106 (https://github.com/b-casey/OpenSMACX/blob/d7cf7140dfa5e24cd3462f7d37b0b65c2c591159/src/game.cpp#L106)


Are turns 1-indexed? This line leads me to believe they are, as does the fact that games begin in MY 2101 even though alphax and the opening narration suggests 2100. I want to clarify something on the wiki if so.
Title: Re: OpenSMACX v0.2.3
Post by: scient on September 23, 2020, 11:20:45 PM
https://github.com/b-casey/OpenSMACX/blob/d7cf7140dfa5e24cd3462f7d37b0b65c2c591159/src/game.cpp#L106 (https://github.com/b-casey/OpenSMACX/blob/d7cf7140dfa5e24cd3462f7d37b0b65c2c591159/src/game.cpp#L106)


Are turns 1-indexed? This line leads me to believe they are, as does the fact that games begin in MY 2101 even though alphax and the opening narration suggests 2100. I want to clarify something on the wiki if so.

They are. If you look at this this (https://github.com/b-casey/OpenSMACX/blob/d7cf7140dfa5e24cd3462f7d37b0b65c2c591159/src/game.cpp#L32) value on loading a brand new game as the first faction it shows as 1.
Title: Re: OpenSMACX v0.2.3
Post by: Blake00 on October 16, 2020, 06:41:21 AM
This is super cool!
Title: Re: OpenSMACX v0.2.3
Post by: mmmniple on March 26, 2021, 12:56:47 PM
Hello, first say I hope and desire than everyone and relatives are fine as covid is being terrible.

Sadly I had to halt and I could no resume seriously with my two personal projects (both are reverse engineering : one caviar voxel and the other, which is far easier, a program called LiveMove 2  which was used to create "motion files" on wiimote, wiimotion plus and psmove,as I go on studying for a state job (which exams also have delayed by covid).


I would make a request, if you don't care, Scient

Quote
Honestly, voxel classes (Caviar/CaviarData) aren't that complex in the grand scheme of things (from my cursory look through functions). It won't be too difficult to break them down."


Could you please break them down / convert to dll to look them on the little rest times?
or simply but some note on the Ida database and share it.
I would love look it from time to time when I can.
(excuse me if my English has some mistakes)
Thanks to everybody for go on with this wonderful project!
Yours
Title: Re: OpenSMACX v0.2.3
Post by: Blake00 on August 05, 2022, 09:30:20 AM
Sad to see this has gone quiet over the last couple of years. Also surprised that it's not more well known. Have posted a thread over at CivFanatics (https://forums.civfanatics.com/threads/opensmacx-open-source-fan-rebuild-of-sid-meiers-alpha-centauri.678305/). Maybe it'll lure some more volunteers here...
Title: Re: OpenSMACX v0.2.3
Post by: magic9mushroom on July 15, 2023, 06:35:50 AM
Is the random-event code in there somewhere? I can't find it.

Also, regarding genejack factories' effect on mind-control cost, you seem to have the x2 to distance-from-HQ but not the x3/4 to overall cost.
Templates: 1: Printpage (default).
Sub templates: 4: init, print_above, main, print_below.
Language files: 4: index+Modifications.english (default), TopicRating/.english (default), PortaMx/PortaMx.english (default), OharaYTEmbed.english (default).
Style sheets: 0: .
Files included: 31 - 840KB. (show)
Queries used: 15.

[Show Queries]