Alpha Centauri 2

Sid Meier's Alpha Centauri & Alien Crossfire => Modding => Topic started by: photalysis on May 09, 2016, 03:24:53 AM

Title: Psst! Pointers on exe-modding?
Post by: photalysis on May 09, 2016, 03:24:53 AM
I must have been inspired by the example of the exe-modders here - that familiar itch to learn coding has returned. I'm particularly interested in modding the AI in SMAX. Competent and clever AI, though rare in 4x games, is extremely important to me. GalCiv II scored major brownie points with me for this reason. This could very well be the carrot-on-a-stick that keeps me motivated; I don't learn things for their own sake.

Which brings me to my question: How do I get started? I assume it entails decompiling terranx.exe into its original source language. Perhaps not a very newb-friendly task, but it could be doable. Speaking of which, what was SMAX coded in? Anything else I should know? I'm given to understanding that the language you choose is important, but in the case of decompiling it will already be decided for me, unless I misunderstand. Nevertheless, knowing that language's particular strengths and applications will influence my decision to go through with it.

I've got sites like codeacademy and stackexchange bookmarked, so I've got the resources to at least know what I need to know, and procure additional texts. Possibly. Maybe.

Thank you in advance - I'm always grateful for tech advice :)
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 10, 2016, 12:38:14 AM
As I wrote on the wiki (http://alphacentauri2.info/wiki/Modding#.EXE_modding_.2F_Programming_changes):

Quote
As the source code of SMAC is not available, these changes have been accomplished by disassembling the binaries, analysing the code, and writing fixes in place within the binary, also in assembly code.

In late 2014, PlotinusRedux released a new kind of patch for SMAC. This patch provides a number of user interface improvements that would have been extraordinarily difficult or tedious to implement in assembly.

PRACX installs a small patch into SMAC's core code, getting it to call an external library written in C++. This library then patches the running SMAC binary further, overriding calls to a number of functions with calls to new C++ implementations provided by the library.

See also: Installing patches, Category:Unofficial patches, Disassembly

## Future work

Yitzi continues work on his patch following the disassembly method. For many bugs and small, engine features, this is a practical way to work, and Yitzi has proven himself to be very good at it.

In late 2015, scient announced that the Mac version of the game had been compiled without obfuscating function names. This is of great help when documenting or trying to understand a disassembled program! A project began to document the game code better and to complete the job of decompiling it. We hoped that we might be able to produce a compilable code base for SMAC, or, if that were not possible, to greatly aid future PRACX-style modifications and reimplemtations of the game.


The database that I mention in the last paragraph may be getting published soon (next week, maybe), which will be the most useful tool for SMACX modding in quite a while.

I must stress that decompilation is not a straightforward process and we are unlikely to generate a compilable code base for SMACX in the short term. In the longer term, with scient's database, it may be possible with some rewriting, but the code base will then still be very large and almost entirely uncommented. Less daunting than the current approach by far, but still quite tricky.

SMACX was written in C, by the way.

Anyway, you're considering programming mods to SMACX and you don't have a lot of experience at the moment. As a first thing, you could try going through the PRACX source code and trying to understand how it works and why it's relevant to non-graphics mods to SMACX. On the one hand, PRACX is written in C++ and even has some comments (though not enough), so it's much easier to understand than just disassembling terranx.exe, on the other hand, the most successful .exe modders have just started by disassembling terranx.exe and tracing it.

If you want to try PRACX, head to github and my repo of the latest code: https://github.com/DrazharLn/pracx (https://github.com/DrazharLn/pracx)

If you want to try disassembling, get ollydbg.

If you want to try decompiling, get IDA (v5 is free for non-commercial use).
Title: Re: Psst! Pointers on exe-modding?
Post by: Buster's Uncle on May 10, 2016, 02:49:50 AM
scient talked to ANYone five months ago and I'm only hearing about it now?
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 10, 2016, 10:33:45 AM
Unless my datsat are wrong that was just from the "scient replied" thread. Could have been late 2014 instead, I wrote that article when I was pretty tired.
Title: Re: Psst! Pointers on exe-modding?
Post by: photalysis on May 12, 2016, 04:12:28 PM
C! I was afraid of this. I'm unsure when delving into a mid-level language is appropriate and have largely avoided it. C++, I am slightly more familiar with (basic data manipulation and I/O, never got around to Boost). So, the SMAX source was originally in C, but the mod libraries were compiled from source in C++. I never considered the possibility of writing different libraries or modules in different languages, or the implications - I had assumed this "wouldn't work." I suppose abstraction would allow the libraries to make calls to functions using lower-level features without any issues, though. But... The identifiers for those functions aren't known, or even relevant without the source. I'm unsure how Yitzi managed to get his code to "hook in."

You say that viable source hasn't been generated from decompilation? And that external libraries were created to work around this, re-directing function calls to over-riding implementations in the library? I have some pre-requisite reading on libraries to understand how this works. Among other topics.

If it is possible to change the behavior of the AI through Yitzi's library(?), I imagine it would be less manual of a task, and certainly easier to grasp. Or... I could also play with the source of one of the SMAC clones floating around here. Options.

Apologies for the late reply, I've been binge-reading wiki articles to get current on terminology. DrazharLn, is it okay if I pm you about books and materials?
Title: Re: Psst! Pointers on exe-modding?
Post by: Buster's Uncle on May 12, 2016, 04:29:00 PM
Yitzi does it by grinding and grinding and grinding away, I think - he's very meticulously detail-oriented.
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 12, 2016, 10:53:35 PM
Yitzi doesn't produce a hook-in library. Yitzi does it the hardcore way and manually edits the assembly code of the SMACX executable.

Which is how everyone does it, apart from PlotinusRedux. Unlike the rest of us, Plotinus knows enough about windows that he was able to develop this C++ library that hooks in quite nicely and allows you to effectively inject new C++ functions into SMACX.

You can PM me if you like, but better to just talk in the open, in this thread.

If you really want to get into enhancing the AI, I suggest we talk about how you want to do that at an abstract level first, rather than jumping straight to code.

What in particular do you think is deficient with the SMACX AI? What kind of things would a competent player do instead?

If you're interested in the theory (and trying not to scare you off): there are a variety of approaches to game AI, from very primitive heuristics based models up to full blown epistemic planners (a fancy name for planning based on knowledge, with particular connotations). What approaches do you know about? What do you feel comfortable learning?
Title: Re: Psst! Pointers on exe-modding?
Post by: Buster's Uncle on May 12, 2016, 11:15:26 PM
Draz, do you already have that university degree in this programming the electronic computing machine stuff, or is that by the end of the month?
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 12, 2016, 11:23:18 PM
I got my degree a while ago :P

I'm a PhD student now, I even teach programming to undergrads sometimes. I don't claim mastery of computing, programming or sciencing, though :)
Title: Re: Psst! Pointers on exe-modding?
Post by: Buster's Uncle on May 12, 2016, 11:32:38 PM
So you're MASTER Draz, now.  Go figure.

I sorta thought you were still 17 or something from when I met you, Boy Wonder.

(When you get that Piled hgher and Deeper in mechanical brains, let's change your display name to DR. DrazharLn... ;nod  Heck - I'm making that a new rule; everybody with a PhD -we have a few- gets Dr. added to their display name.  -I may actually do this to Dr. Mylochka.)
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 13, 2016, 12:50:13 AM
Let's not race ahead: I could well not get the PhD.

Also, I was never awarded a Masters degree (though I have done most of the content of an MSc course), I went straight from Bachelor to PhD Student. So, just DrazharLn BSc (Hons), or "His Majesty, Queen Draz" will do.
Title: Re: Psst! Pointers on exe-modding?
Post by: Buster's Uncle on May 13, 2016, 02:47:20 AM
;lol

You left out Freer of Virgin Slaves... ;lol
Title: Re: Psst! Pointers on exe-modding?
Post by: photalysis on May 14, 2016, 02:45:07 PM
Oh, PR's the one using libraries. I was mixed up there.

You can PM me if you like, but better to just talk in the open, in this thread.
I don't mind if no-one else does, but... As a newbie, the questions I have are nearly endless, and so I thought it would be better to read some material on my own than keep this thread perma-bumped, heh.

Quote from: DrazharLn
If you really want to get into enhancing the AI, I suggest we talk about how you want to do that at an abstract level first, rather than jumping straight to code.
Ah, do you mean formal AI design patterns? I'm interested in this if there's resources on it out there, but the "model" I have in my head is hardly formal. It's the product of an outsider's speculation on how to design AI. More or less, I wanted to create an AI which mirrors my own strategies and tendencies as closely as possible, and does it well.

Speaking of speculation... One of the problems I see with game AI in general, is their lack of ability to do just that. Speculate. As humans, very few of the decisions we make are fully informed; we operate with extremely limited data. I'm philosophically opposed to AI which uses vision or knowledge "cheats" to provide an adequate challenge, this is uninteresting to play against or design. It renders irrelevant entire subsets of gameplay devoted to scouting and espionage, because it is based upon what the opponent knows. It's tragic, and very disappointing, but this has been the standard in strategy games, for the most part.

I have more thoughts on current limitations in AI... This is only one. I don't think it would be too difficult to create ad hoc implementations for specific scenarios to increase its speculative capacity. One example: If an AI spies one of your units on a body of land which it has fully mapped, it can infer that there may be other, unsighted military units on other locations on that continent, and should therefore only send colony pods to that continent with an escort. Sending it to another, equidistant location the AI infers does not require an escort gives a comparative advantage, because it takes less time to establish that colony. This is self-evident to a human player, but not the poor AI!

Quote from: DrazharLn
What in particular do you think is deficient with the SMACX AI? What kind of things would a competent player do instead?
Off the top of my head - nothing in particular, actually. I should have been making notes, perhaps. It's merely many instances where a human player is able to apply a specialized strategy where the AI cannot, netting them a large advantage. I've found in most 4x games, AI factions or empires with an innate production bonus tend to be the most powerful, simply because it allows the AI to persist through brute force, minimizing weaknesses in strategy. Yang certainly might qualify!

Quote from: His Majesty
If you're interested in the theory (and trying not to scare you off): there are a variety of approaches to game AI, from very primitive heuristics based models up to full blown epistemic planners (a fancy name for planning based on knowledge, with particular connotations). What approaches do you know about? What do you feel comfortable learning?
I'm very interested! However, I'm unfamiliar with any of these approaches, as mentioned. Book recommendations would be great. This epistemic planning in particular sounds a bit similar to what I've been getting at :)
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 14, 2016, 04:30:04 PM
I don't think anyone else will mind you bumping a thread, so do ask away. Though perhaps you'd rather talk via voice-comms or instant messaging first to get quick questions done and check we're on the same page.

Regarding books to read, that will depend on your programming and mathematics proficiency a bit. I haven't read any serious books on AI in a while, and I've never read a book exclusively on game AI, but looking through the contents of this book, it could be quite useful.

It's not really interested in strategy games, but the sections on pathing might give you some ideas and the approach in general might inform you about how we can get machines to do ai things.

Anyway, book: https://books.google.co.uk/books?hl=en&lr=&id=gDLpyWtFacYC&oi=fnd&pg=PR13&dq=game+ai&ots=v0wk8TUn6j&sig=dK8w9jEDzCFw119q9BpFVJI6zm8#v=onepage&q=game%20ai&f=false (https://books.google.co.uk/books?hl=en&lr=&id=gDLpyWtFacYC&oi=fnd&pg=PR13&dq=game+ai&ots=v0wk8TUn6j&sig=dK8w9jEDzCFw119q9BpFVJI6zm8#v=onepage&q=game%20ai&f=false)

You can probably find it on amazon or just do the studenty thing and get it from here: http://gen.lib.rus.ec/search.php?req=Programming+Game+AI+by+Example&lg_topic=libgen&open=0&view=simple&res=25&phrase=1&column=def (http://gen.lib.rus.ec/search.php?req=Programming+Game+AI+by+Example&lg_topic=libgen&open=0&view=simple&res=25&phrase=1&column=def)
Title: Re: Psst! Pointers on exe-modding?
Post by: scient on May 14, 2016, 08:49:41 PM
scient talked to ANYone five months ago and I'm only hearing about it now?


Like Rumpelstiltskin just say my name and I appear (maybe with a bit of delay haha). I'll do a more in-depth post in one of other threads related to my work on SMACX.

For learning RCE, I suggest writing small basic programs in C/C++ and then popping the compiled debug binary into IDA/debugger. OllyDbg is great debugger for beginners. For disassembly, IDA is industry standard. There are some others like Hopper for mac. IDA also had a great decompiler plugin that is quite helpful with SMACX code. This way you have your own original source code and can see what's going on under the hood.

Great books under Reverse Engineering section here:
http://dfir.org/?q=node/8 (http://dfir.org/?q=node/8)

The x86 binary (windows) was compiled with an older version of Microsoft Visual C++ (5 or 6).

So I've been talking with DrazharLn about sharing the database like a week ago. He reached out to me via email, my life has been hectic. I have a little bit of time before things get busy again so I'm currently working on cleaning a few things up and merging notes into the database. There is much do to with understanding data structures as well as what certain globals or enums do. I have tried to identify as much as I can as I work on it. Either way, it makes it a lot easier to say find specific combat, base or diplomacy code than blindly look around. Want to understand something related to global map? Just see all references to g_MAP in database. Things like this. It makes finding patterns and understanding code flow more manageable.

Once I finish a draft ready for public, I will host it on here as reference and maybe if people identify more stuff merge that info in future draft. If I run out of time and don't finish want I wanted to, will post it anyway. Either way, it will get posted by first week in June.
Title: Re: Psst! Pointers on exe-modding?
Post by: Buster's Uncle on May 14, 2016, 08:56:49 PM
scient talked to ANYone five months ago and I'm only hearing about it now?


Like Rumpelstiltskin just say my name and I appear (maybe with a bit of delay haha). I'll do a more in-depth post in one of other threads related to my work on SMACX.

For learning RCE, I suggest writing small basic programs in C/C++ and then popping the compiled debug binary into IDA/debugger. OllyDbg is great debugger for beginners. For disassembly, IDA is industry standard. There are some others like Hopper for mac. IDA also had a great decompiler plugin that is quite helpful with SMACX code. This way you have your own original source code and can see what's going on under the hood.

Great books under Reverse Engineering section here:
http://dfir.org/?q=node/8 (http://dfir.org/?q=node/8)

The x86 binary (windows) was compiled with an older version of Microsoft Visual C++ (5 or 6).

So I've been talking with DrazharLn about sharing the database like a week ago. He reached out to me via email, my life has been hectic. I have a little bit of time before things get busy again so I'm currently working on cleaning a few things up and merging notes into the database. There is much do to with understanding data structures as well as what certain globals or enums do. I have tried to identify as much as I can as I work on it. Either way, it makes it a lot easier to say find specific combat, base or diplomacy code than blindly look around. Want to understand something related to global map? Just see all references to g_MAP in database. Things like this. It makes finding patterns and understanding code flow more manageable.

Once I finish a draft ready for public, I will host it on here as reference and maybe if people identify more stuff merge that info in future draft. If I run out of time and don't finish want I wanted to, will post it anyway. Either way, it will get posted by first week in June.
Glad you're not dead, sir. :D

Sooner is better than later - just sayin'.
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 16, 2016, 07:52:39 AM
Hi, Scient :)

Well, there's no big rush, BUncle. Scient's done most of the work, so he should be free to release at his own pace.
Title: Re: Psst! Pointers on exe-modding?
Post by: scient on May 16, 2016, 01:36:18 PM
Have you had a chance to look at draft db? I'm half way through doing the class functions. Some of big ones like Win and Buffer really suck but slogging through them all.
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 16, 2016, 05:04:11 PM
I've only had a brief look: I was away from home this weekend. I'd like to build compilable versions of bits of the code, but I don't know how feasible that is yet.

Idea would be to gradually replace the interesting bits of the code with compilable versions in a library.

Any idea how feasible that is?

The db is a lot more usable than my own one. Good job :)
Title: Re: Psst! Pointers on exe-modding?
Post by: scient on May 16, 2016, 07:46:51 PM
That is actually how I wanted to start having live code to test while debugging. You would have to understand the data structures that are being passed as parameters. Then you could compile your new code and inject a dll along with a patch to the original function to redirect to the new dll. That way, you could slowly shift function by function into the new dll.

Also, if there is anything in your database that I missed let me know so I can merge them.
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 16, 2016, 08:14:09 PM
My database has no manual annotations - I had some annotations of my own years ago, but they probably didn't cover anything you don't have.

Do you have any idea how much work it takes to turn the pseudo-c into c? Presumably lots of work for the 12MB dump of the whole file, (though possibly a load of that is statically linked library stuff we can strip out), but I was hoping that generated c for a smaller section might be quite easy to clean up for an injector library.
Title: Re: Psst! Pointers on exe-modding?
Post by: scient on May 16, 2016, 09:02:10 PM
It is quite easy for simple functions. I think I put up a few on github awhile ago. That's usually what I'd use as a guideline as well as looking over the assembly logic. Once you start getting into the custom data structures it gets more complex translating that over to c. You could probably hack together some code with inlining but it wouldn't be pretty.
Title: Re: Psst! Pointers on exe-modding?
Post by: Yitzi on May 17, 2016, 09:31:51 PM
scient talked to ANYone five months ago and I'm only hearing about it now?

Like Rumpelstiltskin just say my name and I appear (maybe with a bit of delay haha). I'll do a more in-depth post in one of other threads related to my work on SMACX.

For learning RCE, I suggest writing small basic programs in C/C++ and then popping the compiled debug binary into IDA/debugger. OllyDbg is great debugger for beginners. For disassembly, IDA is industry standard. There are some others like Hopper for mac. IDA also had a great decompiler plugin that is quite helpful with SMACX code. This way you have your own original source code and can see what's going on under the hood.

I actually did it in assembly for some time before even learning C, never mind C++.  Assembly is actually a fairly straightforward language if you're up for a bit (or a lot) of detective work figuring out what does what.

I'm wondering, what makes IDA better than Ollydbg?
Title: Re: Psst! Pointers on exe-modding?
Post by: DrazharLn on May 17, 2016, 10:52:13 PM
photalysis, RCE = reverse code engineering, by the way.

Yitzi, I like that IDA gives you a list of functions in the code and other helpful tools for understanding program flow. scient will use it better, I'm sure.
Title: Re: Psst! Pointers on exe-modding?
Post by: scient on May 17, 2016, 11:21:22 PM
For simple code or simple analysis, Olly is fine. For more complex code where you are figuring out structures, enums, globals and functions then this is where IDA shines. Along with the ability to have a database where you can save persistent changes. I can't count the number of times I've lost quick notes I've made in Olly to point where I only use it as a quick and dirty debugger. Most of the debugging I do is with lldb CLI so having any sort of GUI is a bonus haha. I have started to use IDA's debugger and it's great. Especially if you start to mark up the database.
Title: Re: Psst! Pointers on exe-modding?
Post by: scient on May 30, 2016, 01:01:46 PM
I have made leaps and bounds working on the database over the past two weeks. I really am starting to get an idea of all the different classes for the game and how they work together. Well, at least in my head. I've been trying to mark up and identify as much as possible in the database, especially related to the interface code and how it interacts with logic and structures. I have merged 95% of what was in the PPC binary into my database related to function names and custom variable parameters. As of now, my database is more complete along with the huge amount that I have identified on my own. I have tried to keep the same naming conventions so everything stays fairly uniform.

I've decided on a cut off and release date, 6/15. I have done most of what I wanted to accomplish for a public release and should finish up the rest by then.
Templates: 1: Printpage (default).
Sub templates: 4: init, print_above, main, print_below.
Language files: 4: index+Modifications.english (default), TopicRating/.english (default), PortaMx/PortaMx.english (default), OharaYTEmbed.english (default).
Style sheets: 0: .
Files included: 33 - 892KB. (show)
Queries used: 19.

[Show Queries]