OpenSMACX v0.2.3

[vidyajames]

I appreciate your desire to use a bleeding-edge file format for 3D scenes, despite the actual requirement being simple 3D models that would work fine in an OBJ file. I think the goal should be to build something standards compliant, and with maximum support. There are of course many paths to achieve glTF 2.0 support, at least there will, over time, once glTF 2.0 becomes more accepted as a supported standard among tools and frameworks.

I also appreciate your desire to reverse engineer a proprietary 3D model format. I think it would be wasted effort and it would be preferable if a standard file format (like an OBJ file) could be used which would allow for greater extensibility and modding support. The models included in the vanilla game are extremely low-poly (voxels!), it's not exactly something that would take a lot of effort to replicate.

The idea of using Godot as the basis of a source port is frankly laughable, and the justification for it being due to glTF 2.0 support (the dev is a believer!) is embarrassing. The node system is just a full-stop non-starter for this type of project. It's not the right tool for the job, it's not even the right type of tool, like using a chainsaw to unfasten a screw.

I don't understand your point about a front-end. you mean the menus? you mean the actual rendering? If you have an actual source port of the game logic, why would you be hacking around with the binary? You'd be re-implementing it into a rendering framework. PRACX is a hack that would become redundant.

I kinda get the feeling you don't really know what you're talking about, and you've never used any of these things before. When I speak, I'm not just talking out of my ass about the latest meme tech, I've actually implemented these things before. I have to assume you've used Godot before, but frankly it seems questionable based on some of the things you've said.

[bvanevery]

I also appreciate your desire to reverse engineer a proprietary 3D model format.

Somehow communication between us is not working.  Please reread what I've written before.  I have no desire whatsoever to reverse engineer a proprietary 3D model format.  My point was to warn you how hard it would be to do that.  What you personally want to do, is not at all clear.

The idea of using Godot as the basis of a source port is frankly laughable,

Whatever dude.  Do what you want, nobody's stopping you.

If you have an actual source port of the game logic, why would you be hacking around with the binary?

If you want to use the original 3D art assets, in the real world you'll need to use the game binary to display them.  PRACX might inform you how it works.  Or it might not.

If you want to recreate the 3D art assets in some common file format, and you want it to be cross-platform, then you're going to need some kind of 3D engine to display stuff.  It might as well be a 3D engine that has got "legs" and isn't going to go the way of the dodo bird when someone gets tired of working on it.  Godot is one such engine.

When I speak, I'm not just talking out of my ass about the latest meme tech,

If you're "all that", you don't need to be talking.  You can be doing.

[vidyajames]

It's now overwhelming obviously you've never worked with any of these technologies. Maybe you should stick with editing txt files while refusing to adopt the community consensus patches.

As for working on this project, that leads me back to my original question about the state pure reverse engineering effort. This effort that you suggest should be abstracted and re-interpreted to fit in Godot's node system, because... glTF 2.0? 

[bvanevery]

The only thing overwhelmingly obvious is I can't imagine helping you with any project you might lead.  Which I don't think you're actually going to do anyways.

[scient]

Trying to manage manually patching the function redirects to the dll has started to become cumbersome now that I'm redirecting over 250 functions. I put together a python script that will handle all the patching after imports are added in. That process is also automated (C++ code), although I'll have to modify the program slightly to take input parameters rather than hardcoded import list. I plan on hooking that into the python script so everything is done in one shot. There are a few things I need to work out but I don't think it will be too difficult.

So far it is working smoothly! I finished validating everything got patched correctly. I even found a mistake in my manually patched version (unused function) that was off by 1 byte. I applied the same script/process to the last version of my bug fix exe and that seems to run without any issues. I'm going to cut off working on new functions and wrap up all the existing ones I've started or that need testing. After that I'll put together a 2nd release, this time with a script that will hopefully make it easier to utilize the dll.

[Induktio]

Trying to manage manually patching the function redirects to the dll has started to become cumbersome now that I'm redirecting over 250 functions. I put together a python script that will handle all the patching after imports are added in. That process is also automated (C++ code), although I'll have to modify the program slightly to take input parameters rather than hardcoded import list.

So does it mean this build script will be able to take the unpatched game binary and make it usable with OpenSMACX's dll? I'm not sure what build steps you are referring to here. Although I agree it's very important to create a reproducible build process, so that for example one can take the GOG binary and then patch it to be usable with OpenSMACX. Otherwise the source code for this project would be incomplete unfortunately.

[scient]

That's correct. As long as the GOG version is using the latest SMACX binary (patch 2.0 I think) unpacked. If the .text code section lines up you're good. My script is using file offsets to patch rather than a search and replace patch which would take a lot more work to put together and manage. I believe the GOG version is just the unpacked exe with a no cd crack applied.

Minimally, I'll verify offset patches are being applied correctly to the vanilla exe, last iteration of my patched exe and GOG version. I've never fully looked at kyrub's or yitzi's patch code but if they're messing with functions or redirecting code a lot this method will likely not work smoothly. It would also remove any additional functionality since I'm only applying bug fixes and minor mod-ability enhancements.

[DrazharLn]

Re: vidyajames,

Making a new UI for SMAC would be cool. I started making one in Godot (in GDScript) several months ago but didn't get far, partly because games engineering is new to me, partly because it was the first time I was using Godot and it took me a while to find a development workflow that worked for me.

You can use Godot without faffing with the UI and the nodes. Just instantiate all your nodes in code as you need them from one root node. This also makes the project much easier to meaningfully version control.

I'm not sure you'd want to graft a feature-complete UI on to an opensmacx backend (What does the game do that isn't interaction code and isn't pretty trivial? The terrain generation?), but I think it could save a lot of effort and be valuable to do something like replacing the map renderer and compositing the other UI over the top. PRACX has already reversed most of the relevant calls as well.

[Induktio]

It is fairly useless to use Godot as a basis for recreating any art assets from the original game because of at least one reason. Importing and redistributing art assets from the game is not possible because they are copyrighted content. Open source game ports cannot do that.

Instead these kinds of projects usually build compatibility to support reading the original files while playing with an open source engine. I don't think there's many successful OS game ports that tried to recreate every asset from scratch to avoid copyright problems, it doesn't work like that. Just implementing binary compatibility with the original game assets is much faster and avoids all the licensing problems. You might or might not use Godot for something but it does zero for reversing engineering.

[DrazharLn]

All this stuff about Godot is very confused.

As I see it, there are a variety of reasons for building OpenSMACX:

- For the fun of it
- To eventually build an open-source clone of the SMAC engine for posterity
- To make modding the game easier (because more of it can be done without reverse engineering knowledge)
  - e.g. complete control over the AI code
  - Finally fixing stuff like the MP art/faction issue
  - Fixing TCP/IP multiplayer
  - Having a better map renderer
  - Better UI generally
  - Pluggable/composable mods

I think something like Godot could be used if the project gets far enough. In that scenario OpenSMACX is just being used to give us texture buffers of the UI, the map state, etc.

Anyway, there are lots of other potential choices and Godot isn't a particularly sensible option. I used it because I wanted to see if I could make a simple version of SMACX from scratch as a learning exercise, but it was hard and I am way too busy

If someone did make a new renderer, I think it would be a good idea to redo some of the map art and sprites, especially to have a higher colour depth.

I wouldn't be very concerned about distributing this publicly despite it being derivative work. "High resolution texture packs" rarely attract legal notice and I really doubt that EA or Firaxis care any more.

We should really find someone to talk to at Firaxis. We could ask them for the source code (lol) and ask if they kept high resolution copies of the movies and other artwork. It would be really nice to get high def version of the SP movies. Obviously the CGI would still be dated, but I think that's fine.

[bvanevery]

Just implementing binary compatibility with the original game assets is much faster and avoids all the licensing problems.

Except in the case of SMAC.  The voxel format is proprietary and in the real world, you are unlikely to reverse engineer it.  Someone somewhere did make a start on it, there's at least one GitHub project.  They didn't get anywhere remotely near to functional proof of concept.  It is far faster in SMAC's case to recreate the game's voxel artwork from scratch.  YMMV.  I'm not going to attempt this.  I already looked into the feasibility of this a year ago or more.  For the effort to be spent, I will work on a new commercial title and sell it for money.

Godot is a red herring.  I will go so far as to say that vidyajames objections to using Godot, are baseless.  Node architectures don't make or break open source projects.

The reason I brought up Godot at all, is it currently has support for glTF.  The latter is a modern 3D file format.  It has merit for long term sustainability of recreated art assets.  Even Microsoft is into it.  They contributed some kind of DirectX viewer sample.  IMO that's saying something.

I never said one must do anything in Godot.  I said Godot has support for glTF now, and that that's advantageous.  Other open source engines may have glTF support later but Godot has it now.  And Godot itself isn't going to vanish as a project any time soon.  It has legs, it's got the buzz, people are using it.

[DrazharLn]

I don't see any reason to favor glTF in particular. In any case, I don't think this is relevant to OpenSMACX.

OpenSMACX doesn't need to make a decision about 3D assets now or perhaps ever (if it does not replace the 3D renderer). If someone does want to make new assets they can use whatever open standard they like because they're reasonably easy to convert and converters will be available for a long time.

You may be right that redrawing the 3D models would be easier than reversing the vector file format (though as I recall Ford actually reversed the whole format and it just needs someone to export it to a more common vector format or write their own marching cubes code), but it's also not really relevant to what Scient is working on now.

I think at this early stage of the process the discussion should be led by those who actually want to contribute code to the project.

[scient]

My own internal priorities:

1. Game logic code. Within this, I've been heavily focusing on Veh code to fix bugs with combat (among other things). I'll eventually turn my focus to Bases, Map, Tech, etc. Sometimes I'll get distracted and do few random functions here or there.
2. Generic utility functions, parsing of game text files. I've done most of these already. Save files are probably next in this area so I can take over loading/saving of game state.
3. Core game engine code (memory management, strings, logging, etc). The big one here will be Buffer class that is used by everything UI related and most of complex non-UI classes (~160 functions for this class alone). I'd include voxel parsing here as well. Honestly, voxel classes (Caviar/CaviarData) aren't that complex in the grand scheme of things (from my cursory look through functions). It won't be too difficult to break them down. Trying to reverse engineer the entire format is something else entirely and I personally won't be doing that.
4. Core UI engine code > Once you get base classes out of the way (BaseWin, BaseButton, BasePop) it should make it easier and have cascading effect on smaller derived classes.
5. Sound classes, ambiance > hooking into sound.dll. This would probably tie with #4.
6. Non-PBEM multiplayer

Also, per release my aim is to complete a few non-game logic classes. These are the ones I don't enjoy doing as much so pushes me to actually get through them. Slated for 0.2 are Font, Spot and Time. I've actually completed all of these and just need to test Time.

On that note, haven't had as much time to work on project juggling training for work certifications. Also, it's summer. But as cert stuff is winding down, getting some more bandwidth to work on project again.

[DrazharLn]

Sounds great, scient

I'm particularly interested in save files and multiplayer. If save files are reversed we can possibly fix the faction graphics bug and if the multiplayer code is accessible we can make it so players don't need to faff loads to play online and could also support e.g. Thinker in multiplayer.

[Induktio]

The main thing that prevents Thinker from being played in multilplayer is the data synchronization across networks. Network desyncs are also one of the worst areas to debug without good tools, so it's not implemented at the moment. Objectively it probably wouldn't require much new code to implement but it would be very difficult to debug. The majority of that stuff deals with the unit movement code.

In the long term, if this project were to become a standalone game engine, importing the art assets some way or another would become unavoidable. Before that it's not really important though if Scient just keeps patching the old game engine.