Alpha Centauri 2

Sid Meier's Alpha Centauri & Alien Crossfire => Modding => Bug/Patch Discussion => Topic started by: Petek on July 21, 2019, 01:37:10 PM

Title: Possible bug in scient patch 2.0
Post by: Petek on July 21, 2019, 01:37:10 PM
I recently bought a new system. Installed SMACX from GOG, then scient's 2.0 patch. In a new game, when I built the Weather Paradigm, the game crashed to the desktop. I tried disabling SP movies and renamed the file (playuv15.exe) that plays the movies. Also tried changing other video settings. I have DirectDraw=1 in my Alphacentauri.ini file. A Google search for Alpha Centauri Weather Paradigm crash returns several hits with the same problem. I only recently started using v2.0 of scient's patch and don't recall such a problem with v1.0.

I then uninstalled the 2.0 patch and the problem disappeared. System specs are OS Windows 10 Home 64-bit (10.0 build 17763), video card is Nvidia geforce RTX 2080 TI with most recent drivers. 16GB system RAM and 11GB video RAM. Other info available upon request.
Title: Re: Possible bug in scient patch 2.0
Post by: bvanevery on July 21, 2019, 03:08:16 PM
This may only be a workaround, not a diagnosis or fix, but wouldn't you prefer DirectDraw=0 ?  That's what allows you to use full screen resolution on modern computers.  Without that you'd be stuck back on 1024x768 I think.
Title: Re: Possible bug in scient patch 2.0
Post by: Petek on July 21, 2019, 03:55:36 PM
Thanks. That was a typo. I am using DirectDraw=0. Also, I forgot to attach a save file. This is from one turn before building the WP.
Title: Re: Possible bug in scient patch 2.0
Post by: bvanevery on July 21, 2019, 06:33:42 PM
Bug confirmed.  Tested on Windows 10 Pro, Version 1809, OS build 17763.165.  First I tried the .sav file in a plain SMAC installation, GOG version 2.0.2.23.  No problems.  Then I installed a brand new copy of SMAC to a new folder, and installed Scient 2.0 into that folder.  Set DirectDraw=0 and loaded the .sav file.  When the game tries to play the Weather Paradigm movie, it crashes.
Title: Re: Possible bug in scient patch 2.0
Post by: Petek on July 21, 2019, 07:28:30 PM
I think I've identified the problem. A thread on reddit (https://www.reddit.com/r/alphacentauri/comments/c9incg/game_crash_after_weather_paradigm/) described the same problem and mentioned that opening the datalinks entry for the WP also crashed the game (and the same was true for the Longevity Vaccine). I compared the original version of helpx.txt (where the datalinks entries reside) with the one in scient's 2.0 patch.  Scient's helpx.txt contains many entries (including for the WP and LV) with very long lines. The original helpx.txt inserts carriage returns to make the lines shorter. So, the problem can be fixed by restoring the original helpx.txt. (Scient's 2.0 patch makes a backup of the original files in the folder \Alpha Centauri\_backup_v2.00.) This fixes the problem.
Title: Re: Possible bug in scient patch 2.0
Post by: bvanevery on July 22, 2019, 06:38:06 AM
Wow, just a Unix vs. Windows /cr vs. /cr /lf problem?  What a crud world we live in.  Well at least it's 90s software, maybe things have gotten better.  (Don't count on it.)

That's a really good reason for a version 2.1.
Title: Re: Possible bug in scient patch 2.0
Post by: Induktio on July 22, 2019, 02:31:49 PM
Quote from: bvanevery on July 22, 2019, 06:38:06 AM
Wow, just a Unix vs. Windows /cr vs. /cr /lf problem?  What a crud world we live in.  Well at least it's 90s software, maybe things have gotten better.  (Don't count on it.)

Why would this be related to EOL types at all? Both the vanilla and 2.0 version of helpx.txt seem to be using Windows CRLF encoding only. I verified this issue and it really exists with 2.0 patch files: looking up either Weather Paradigm or Longetivity Vaccine from datalinks crashes the game every time. No other project entry in datalinks crashes the game, so there's one really OBVIOUS and common problem with C code to look at here.

Both the WP and LV descriptions contain lines that are longer than 255 characters in the new helpx.txt file, so this really looks like a buffer overflow in some parser code. Looking at it with debugger, some function really seems to overwrite part of the stack with WP's text description, so that one is pretty much guaranteed to crash the game.

Also I think Thinker Mod users never experienced this crash because I didn't ship those modified txt files with my distribution. Thinker only contains the binary patches plus a couple of smaller fixes, but nothing to do with helpx.txt. It's really nice Petek managed to finally isolate this crash, it seemed to be affecting some players.
Title: Re: Possible bug in scient patch 2.0
Post by: bvanevery on July 22, 2019, 03:42:52 PM
You're probably right, I was thinking about "different kinds of lines" rather than "buffer overflow".
Title: Re: Possible bug in scient patch 2.0
Post by: scient on July 23, 2019, 04:44:21 PM
I think someone else ran into same issues with modified helpx.txt bundled with my patch.
http://alphacentauri2.info/index.php?topic=21180.msg118308#msg118308 (http://alphacentauri2.info/index.php?topic=21180.msg118308#msg118308)

Fairly certain my own edits to helpx.txt were minimal in conjunction with some exe patches. The rest of the work was done by others, likely not realizing about internal length restrictions.

I still haven't finished parse_string where crash takes place, some really funky logic and nested loops. I'll definitely include some error handling / better parsing to handle situations like this at least for DLL redirect.
Title: Re: Possible bug in scient patch 2.0
Post by: Induktio on July 25, 2019, 07:07:50 PM
I didn't even remember that thread, that being said, the bug exists in the vanilla binary too if one uses the same text files. So in that sense it was not caused by the patch, but we should issue a new version of 2.0 with those long lines fixed. Otherwise people will just keep running into this issue.
Title: Re: Possible bug in scient patch 2.0
Post by: bvanevery on July 25, 2019, 08:01:09 PM
Quote from: Induktio on July 25, 2019, 07:07:50 PM
a new version of 2.0 with those long lines fixed.

AKA a 2.1, a 2.0.1, a 2.01, a 2.0a, something other than just calling it 2.0, so that people don't continue to blithely use the older release.
Title: Re: Possible bug in scient patch 2.0
Post by: Induktio on July 25, 2019, 11:09:30 PM
I think whoever makes these patches knows how to assign version numbers. Originally it was uploaded by Drazhar but he's probably not active, maybe Scient then wants to update it himself? Does Scient know which text types this limit applies to other than the secret project messages?
Title: Re: Possible bug in scient patch 2.0
Post by: bvanevery on July 26, 2019, 05:02:43 AM
It would probably be best not to try to "know it", but rather, follow the practices of the original files everywhere.  And a CHANGELOG as to why it was done.  Maybe a comment if the file takes a comment.
Title: Re: Possible bug in scient patch 2.0
Post by: DrazharLn on August 01, 2019, 11:20:20 AM
I don't think I modified the help files between receiving them from Scient and building the patch.

I'm happy to build a v2.1 with updated helpx.txt

So lines need to be hard-wrapped at X characters? Do we know what X is? Otherwise I guess I can just wrap them at whatever the original helpx had as its max length. (Edit the max length in the GOG helpx.txt is 121 chars, so I'll wrap them there).

Checking line lengths with python, it looks like there are 295 lines longer than 79 characters.

>>> helpx = open("helpx.txt", encoding="latin1")
>>> lines = helpx.readlines()
>>> max(map(len, lines))
735
>>> len([len(line) for line in lines if len(line) > 79])
295

>>> import seaborn
>>> import matplotlib.pyplot
>>> seaborn.distplot(list(map(len, lines)))
<matplotlib.axes._subplots.AxesSubplot object at 0x7fe9b826b6d8>
>>> matplotlib.pyplot.show() # see attachment
Title: Re: Possible bug in scient patch 2.0
Post by: DrazharLn on August 01, 2019, 02:26:27 PM
Well, that was simple enough to do.

Here are the original max line lengths (including carriage return, not including linefeed, because that's just how awk works):

(click to show/hide)

And just the ones that get included in the patch executable:

(click to show/hide)

After running my script the helpx.txt is trimmed to 124 characters. The longest remaining file is conceptsx.txt at 298 characters. The original conceptsx English is 175 chars, original French 614.

I attach my git repo for the scient patch with the code for wrapping lines and some other minor changes. I also attach a beta-version of v2.1 of the patch on the next post.

Edit: fixed the below, too

Where the help text for some categories is also too long and those help fields don't scroll (e.g. Mag Tube), the overflowing text appears outside of the box it should do and is not cleared when looking at the next datalinks entry, which is another slightly annoying bug.

I guess Googly and Scient didn't test this stuff. I know I didn't ;)

Steps to reproduce: go to datalinks, click all categories, hold down arrow.

There are only a few entries that are too long, so I might just revert them myself. Haven't decided yet.
Title: Re: Possible bug in scient patch 2.0
Post by: DrazharLn on August 01, 2019, 02:26:57 PM
Beta v2.1 installer

Edit: re-uploaded with the required changes to helpx.txt to fix the overflow error, and some other minor changes.

Edit: also put the git repo up on github: https://github.com/DrazharLn/scient-unofficial-smacx-patch
Title: Re: Possible bug in scient patch 2.0
Post by: Induktio on August 03, 2019, 07:38:23 PM
Quote from: DrazharLn on August 01, 2019, 11:20:20 AM
So lines need to be hard-wrapped at X characters? Do we know what X is? Otherwise I guess I can just wrap them at whatever the original helpx had as its max length. (Edit the max length in the GOG helpx.txt is 121 chars, so I'll wrap them there).

It would be best to take the longest lines from the original files and use it as a limit that should never be exceeded. I'm not sure about the exact limit before the overflow occurs because I didn't trace the execution line by line. It might be even possible it varies by the file type but that is unconfirmed. Nice command line tools btw. ;)
Title: Re: Possible bug in scient patch 2.0
Post by: DrazharLn on August 04, 2019, 06:37:21 PM
Yup, I wrapped at 124 chars (longest line in original english helpx.txt). There are much longer lines in the French version, but whatever.

Here's the tool I ended up writing to do it.

https://github.com/DrazharLn/scient-unofficial-smacx-patch/blob/master/lineshortener.py

Anyway, 2.1 above has both sets of changes.
Title: Re: Possible bug in scient patch 2.0
Post by: bvanevery on August 05, 2019, 03:49:14 PM
The original French version could be bugged.  Has anyone tested it?
Title: Re: Possible bug in scient patch 2.0
Post by: DrazharLn on August 15, 2019, 04:11:46 AM
The original french files do not cause the crash. I haven't tried to work out why. Possibly there's just one really long line and it's not somewhere that SMAC actually tries to read from.

I have uploaded v2.1 of scient's patch to both the downloads section here and the github release page of the repo.

https://github.com/DrazharLn/scient-unofficial-smacx-patch
Templates: 1: Printpage (default).
Sub templates: 4: init, print_above, main, print_below.
Language files: 4: index+Modifications.english (default), TopicRating/.english (default), PortaMx/PortaMx.english (default), OharaYTEmbed.english (default).
Style sheets: 0: .
Files included: 31 - 840KB. (show)
Queries used: 15.

[Show Queries]