Psst! Pointers on exe-modding?

[photalysis]

I must have been inspired by the example of the exe-modders here - that familiar itch to learn coding has returned. I'm particularly interested in modding the AI in SMAX. Competent and clever AI, though rare in 4x games, is extremely important to me. GalCiv II scored major brownie points with me for this reason. This could very well be the carrot-on-a-stick that keeps me motivated; I don't learn things for their own sake.

Which brings me to my question: How do I get started? I assume it entails decompiling terranx.exe into its original source language. Perhaps not a very newb-friendly task, but it could be doable. Speaking of which, what was SMAX coded in? Anything else I should know? I'm given to understanding that the language you choose is important, but in the case of decompiling it will already be decided for me, unless I misunderstand. Nevertheless, knowing that language's particular strengths and applications will influence my decision to go through with it.

I've got sites like codeacademy and stackexchange bookmarked, so I've got the resources to at least know what I need to know, and procure additional texts. Possibly. Maybe.

Thank you in advance - I'm always grateful for tech advice :)

[DrazharLn]

As I wrote on the wiki:

As the source code of SMAC is not available, these changes have been accomplished by disassembling the binaries, analysing the code, and writing fixes in place within the binary, also in assembly code.

In late 2014, PlotinusRedux released a new kind of patch for SMAC. This patch provides a number of user interface improvements that would have been extraordinarily difficult or tedious to implement in assembly.

PRACX installs a small patch into SMAC's core code, getting it to call an external library written in C++. This library then patches the running SMAC binary further, overriding calls to a number of functions with calls to new C++ implementations provided by the library.

See also: Installing patches, Category:Unofficial patches, Disassembly

## Future work

Yitzi continues work on his patch following the disassembly method. For many bugs and small, engine features, this is a practical way to work, and Yitzi has proven himself to be very good at it.

In late 2015, scient announced that the Mac version of the game had been compiled without obfuscating function names. This is of great help when documenting or trying to understand a disassembled program! A project began to document the game code better and to complete the job of decompiling it. We hoped that we might be able to produce a compilable code base for SMAC, or, if that were not possible, to greatly aid future PRACX-style modifications and reimplemtations of the game.

The database that I mention in the last paragraph may be getting published soon (next week, maybe), which will be the most useful tool for SMACX modding in quite a while.

I must stress that decompilation is not a straightforward process and we are unlikely to generate a compilable code base for SMACX in the short term. In the longer term, with scient's database, it may be possible with some rewriting, but the code base will then still be very large and almost entirely uncommented. Less daunting than the current approach by far, but still quite tricky.

SMACX was written in C, by the way.

Anyway, you're considering programming mods to SMACX and you don't have a lot of experience at the moment. As a first thing, you could try going through the PRACX source code and trying to understand how it works and why it's relevant to non-graphics mods to SMACX. On the one hand, PRACX is written in C++ and even has some comments (though not enough), so it's much easier to understand than just disassembling terranx.exe, on the other hand, the most successful .exe modders have just started by disassembling terranx.exe and tracing it.

If you want to try PRACX, head to github and my repo of the latest code: https://github.com/DrazharLn/pracx

If you want to try disassembling, get ollydbg.

If you want to try decompiling, get IDA (v5 is free for non-commercial use).

[Buster's Uncle]

scient talked to ANYone five months ago and I'm only hearing about it now?

[DrazharLn]

Unless my datsat are wrong that was just from the "scient replied" thread. Could have been late 2014 instead, I wrote that article when I was pretty tired.

[photalysis]

C! I was afraid of this. I'm unsure when delving into a mid-level language is appropriate and have largely avoided it. C++, I am slightly more familiar with (basic data manipulation and I/O, never got around to Boost). So, the SMAX source was originally in C, but the mod libraries were compiled from source in C++. I never considered the possibility of writing different libraries or modules in different languages, or the implications - I had assumed this "wouldn't work." I suppose abstraction would allow the libraries to make calls to functions using lower-level features without any issues, though. But... The identifiers for those functions aren't known, or even relevant without the source. I'm unsure how Yitzi managed to get his code to "hook in."

You say that viable source hasn't been generated from decompilation? And that external libraries were created to work around this, re-directing function calls to over-riding implementations in the library? I have some pre-requisite reading on libraries to understand how this works. Among other topics.

If it is possible to change the behavior of the AI through Yitzi's library(?), I imagine it would be less manual of a task, and certainly easier to grasp. Or... I could also play with the source of one of the SMAC clones floating around here. Options.

Apologies for the late reply, I've been binge-reading wiki articles to get current on terminology. DrazharLn, is it okay if I pm you about books and materials?

[Buster's Uncle]

Yitzi does it by grinding and grinding and grinding away, I think - he's very meticulously detail-oriented.

[DrazharLn]

Yitzi doesn't produce a hook-in library. Yitzi does it the hardcore way and manually edits the assembly code of the SMACX executable.

Which is how everyone does it, apart from PlotinusRedux. Unlike the rest of us, Plotinus knows enough about windows that he was able to develop this C++ library that hooks in quite nicely and allows you to effectively inject new C++ functions into SMACX.

You can PM me if you like, but better to just talk in the open, in this thread.

If you really want to get into enhancing the AI, I suggest we talk about how you want to do that at an abstract level first, rather than jumping straight to code.

What in particular do you think is deficient with the SMACX AI? What kind of things would a competent player do instead?

If you're interested in the theory (and trying not to scare you off): there are a variety of approaches to game AI, from very primitive heuristics based models up to full blown epistemic planners (a fancy name for planning based on knowledge, with particular connotations). What approaches do you know about? What do you feel comfortable learning?

[Buster's Uncle]

Draz, do you already have that university degree in this programming the electronic computing machine stuff, or is that by the end of the month?

[DrazharLn]

I got my degree a while ago

I'm a PhD student now, I even teach programming to undergrads sometimes. I don't claim mastery of computing, programming or sciencing, though

[Buster's Uncle]

So you're MASTER Draz, now.  Go figure.

I sorta thought you were still 17 or something from when I met you, Boy Wonder.

(When you get that Piled hgher and Deeper in mechanical brains, let's change your display name to DR. DrazharLn...   Heck - I'm making that a new rule; everybody with a PhD -we have a few- gets Dr. added to their display name.  -I may actually do this to Dr. Mylochka.)

[DrazharLn]

Let's not race ahead: I could well not get the PhD.

Also, I was never awarded a Masters degree (though I have done most of the content of an MSc course), I went straight from Bachelor to PhD Student. So, just DrazharLn BSc (Hons), or "His Majesty, Queen Draz" will do.

[Buster's Uncle]

You left out Freer of Virgin Slaves...

[photalysis]

Oh, PR's the one using libraries. I was mixed up there.

You can PM me if you like, but better to just talk in the open, in this thread.

I don't mind if no-one else does, but... As a newbie, the questions I have are nearly endless, and so I thought it would be better to read some material on my own than keep this thread perma-bumped, heh.

If you really want to get into enhancing the AI, I suggest we talk about how you want to do that at an abstract level first, rather than jumping straight to code.

Ah, do you mean formal AI design patterns? I'm interested in this if there's resources on it out there, but the "model" I have in my head is hardly formal. It's the product of an outsider's speculation on how to design AI. More or less, I wanted to create an AI which mirrors my own strategies and tendencies as closely as possible, and does it well.

Speaking of speculation... One of the problems I see with game AI in general, is their lack of ability to do just that. Speculate. As humans, very few of the decisions we make are fully informed; we operate with extremely limited data. I'm philosophically opposed to AI which uses vision or knowledge "cheats" to provide an adequate challenge, this is uninteresting to play against or design. It renders irrelevant entire subsets of gameplay devoted to scouting and espionage, because it is based upon what the opponent knows. It's tragic, and very disappointing, but this has been the standard in strategy games, for the most part.

I have more thoughts on current limitations in AI... This is only one. I don't think it would be too difficult to create ad hoc implementations for specific scenarios to increase its speculative capacity. One example: If an AI spies one of your units on a body of land which it has fully mapped, it can infer that there may be other, unsighted military units on other locations on that continent, and should therefore only send colony pods to that continent with an escort. Sending it to another, equidistant location the AI infers does not require an escort gives a comparative advantage, because it takes less time to establish that colony. This is self-evident to a human player, but not the poor AI!

What in particular do you think is deficient with the SMACX AI? What kind of things would a competent player do instead?

Off the top of my head - nothing in particular, actually. I should have been making notes, perhaps. It's merely many instances where a human player is able to apply a specialized strategy where the AI cannot, netting them a large advantage. I've found in most 4x games, AI factions or empires with an innate production bonus tend to be the most powerful, simply because it allows the AI to persist through brute force, minimizing weaknesses in strategy. Yang certainly might qualify!

If you're interested in the theory (and trying not to scare you off): there are a variety of approaches to game AI, from very primitive heuristics based models up to full blown epistemic planners (a fancy name for planning based on knowledge, with particular connotations). What approaches do you know about? What do you feel comfortable learning?

I'm very interested! However, I'm unfamiliar with any of these approaches, as mentioned. Book recommendations would be great. This epistemic planning in particular sounds a bit similar to what I've been getting at :)

[DrazharLn]

I don't think anyone else will mind you bumping a thread, so do ask away. Though perhaps you'd rather talk via voice-comms or instant messaging first to get quick questions done and check we're on the same page.

Regarding books to read, that will depend on your programming and mathematics proficiency a bit. I haven't read any serious books on AI in a while, and I've never read a book exclusively on game AI, but looking through the contents of this book, it could be quite useful.

It's not really interested in strategy games, but the sections on pathing might give you some ideas and the approach in general might inform you about how we can get machines to do ai things.

Anyway, book: https://books.google.co.uk/books?hl=en&lr=&id=gDLpyWtFacYC&oi=fnd&pg=PR13&dq=game+ai&ots=v0wk8TUn6j&sig=dK8w9jEDzCFw119q9BpFVJI6zm8#v=onepage&q=game%20ai&f=false

You can probably find it on amazon or just do the studenty thing and get it from here: http://gen.lib.rus.ec/search.php?req=Programming+Game+AI+by+Example&lg_topic=libgen&open=0&view=simple&res=25&phrase=1&column=def

[scient]

scient talked to ANYone five months ago and I'm only hearing about it now?

Like Rumpelstiltskin just say my name and I appear (maybe with a bit of delay haha). I'll do a more in-depth post in one of other threads related to my work on SMACX.

For learning RCE, I suggest writing small basic programs in C/C++ and then popping the compiled debug binary into IDA/debugger. OllyDbg is great debugger for beginners. For disassembly, IDA is industry standard. There are some others like Hopper for mac. IDA also had a great decompiler plugin that is quite helpful with SMACX code. This way you have your own original source code and can see what's going on under the hood.

Great books under Reverse Engineering section here:
http://dfir.org/?q=node/8

The x86 binary (windows) was compiled with an older version of Microsoft Visual C++ (5 or 6).

So I've been talking with DrazharLn about sharing the database like a week ago. He reached out to me via email, my life has been hectic. I have a little bit of time before things get busy again so I'm currently working on cleaning a few things up and merging notes into the database. There is much do to with understanding data structures as well as what certain globals or enums do. I have tried to identify as much as I can as I work on it. Either way, it makes it a lot easier to say find specific combat, base or diplomacy code than blindly look around. Want to understand something related to global map? Just see all references to g_MAP in database. Things like this. It makes finding patterns and understanding code flow more manageable.

Once I finish a draft ready for public, I will host it on here as reference and maybe if people identify more stuff merge that info in future draft. If I run out of time and don't finish want I wanted to, will post it anyway. Either way, it will get posted by first week in June.