AI War 2 v2.090 Released! “Returning From Beta”

Since May 22nd, we’ve been working hard on the game in the beta branch, and now we’re back onto the main branch for everyone to enjoy the new items.  The full release notes start here and finish here.  Let’s see if I can’t summarize that down a bit, though.

The TLDR is that this is almost all prep for multiplayer.  If building and testing multiplayer was a car trip, this would be like getting the car in tip-top shape right before you start the trip.  If the trip is into rough terrain, then good preparation is going to mean a far smoother trip once you get on the road. 

So let’s take a look at that, and then the other bits that are unrelated.

Data Compaction, Many Times Over

Savegames are vastly smaller.  They’re also more efficient to write to disk, and to read.  This was a massive overhaul, but it also directly effects the way we will be transmitting network data messages.

I’ve written about the benefits of this at length in the past, so I’ll skip rehashing that here.  But the short version is that essentially there is a LOT going on in this game, and during networked play we need to be able to fix desyncs on the fly.  The only way to really do that is if we can efficiently send large amounts of correction data in the background while not interrupting your main play.

Back in May, I wasn’t feeling really confident that it would work out well.  It was borderline, but the processing load and the amount of network packets required was going to be iffy.  Any good engineering student knows that you want to build in redundancy: if your bridge can only hold ten tons, and it will have nine tons on it regularly, you’re asking for a terrible accident.  You want to start out with a lot more extra capacity to avoid the tragedy, and you also want to do that from the start rather than trying to strengthen the bridge while people are driving on it.

So that’s where we are now: the metaphorical bridge is ridiculously braced and ready for action.  This was a really massive undertaking, and it was the chief thing that led to the extended beta period here, so hopefully it makes intuitive sense why I chose to do this before getting into a functional multiplayer alpha rather than mixing the two together.

Game Loading Improvements

This might seem unrelated to multiplayer, but one of the things that we have to be really careful of is that all the players have the same data about game units when they load in.

Because this game was built from the ground up with mod support in mind, there are certain things that I realized could be impairing this consistency.  So I put in a bunch of new auto-checking to make sure that things happen the way I expect, and in the process:

  1. Fixed a few issues that were impacting mainly mod authors, making them have far more xml than they needed to.
  2. Made the game load vastly faster for everyone (depends on your computer, but for me it loads in 7-24 seconds now instead of 16-90 seconds.  The big variance is largely based on how busy my hard drives are and what is already cached in RAM from prior runs.
  3. Made it possible for mods to adjust base units (like Raider) and then have those modded changes cascade to all of the descendants of that base unit (Daggers, etc).
  4. Implemented some code that lets us verify that things are consistent regardless of how your OS does the sorting of files in its filesystem.  This sort of thing was going to be a potential barrier to cross-platform play between OSes, but it no longer is.
  5. This also lets you now disable mods and expansions that you have installed from within the game UI, if you don’t want to play with them at the moment.  This is key for multiplayer, where you might not have or want the same mods or expansions as someone else you are playing with, and the people with extra stuff can now just disable that in order to play with one another — no need for someone to reach for their wallet just to play multiplayer with you, if you have extra DLC, for instance.
  6. This prior one also lets us now distribute some mods with the game, but have them be in an “off by default” starting state.  This lets various mods be a lot more easily-accessible, without us having to support them in an official capacity or them affecting every game anyone plays.  The first mod to be included like this is The Spire Railgun Shop by Lord of Nothing.

Rebuilt Compiler Pipeline

This sounds like a yawn sort of item, so I’ll keep it brief.  But it’s something that first Keith and then later I had been working on since 2017 or so, and there were some key limitations with how we were able to link against some of the extra language features that… well, the various multiplayer libraries we’re going to be using (Steam, GOG, and Forge Remastered) all use.

This mean that there was a certain indirection I had to previously build in in order to load those things, and in order to test and change pieces of code in those specific areas it was literally an ELEVEN MINUTE wait for me while I hoped that whatever I changed was okay.  Normally it should be a couple of seconds of turnaround, but because of our compiler chain limitations, these specific libraries couldn’t be linked in the direct fashion we prefer.

This was going to make bugfixing in multiplayer an absolute nightmare, as you can imagine, since being able to routinely test things without eleven minute delays peppered all throughout my day is pretty important.  If I have a one-line typo, I need to be able to fix that and then see the results within a few seconds, not after 11 minutes.  That whole thing was making me dread working on certain aspects of multiplayer, so I put more time into it and finally managed to fix the root problem.  Hooray!

Prepping For Efficient Multiplayer Messages

We have long used a very efficient “serialize by index” (instead of name) method to make savegames smaller and more efficient, but that sort of thing could not work in multiplayer because there was no way to have a shared set of agreed-upon indices between the host and client(s).

The game has now been upgraded to handle fully setting those up so that now the host and clients get those synced automatically when the game starts, and so the ultra-efficient smaller messages now work in multiplayer and not just on disk.

Fireteam Specialists

This is a major new feature that Badger created for the upcoming second DLC for the game, which will be called Zenith Onslaught.  (Please put away any pitchforks — I have not been working on DLC2 at all yet, as my focus is appropriately on multiplayer.  And multiplayer is not something in Badger’s scope to help with).

Anyhow, like fireteams themselves (which were developed initially for DLC1), these new abilities for them have been backported into the base game.

Basically this lets some group, like the AI Hunters, have a sub-group that is chasing just a specific faction.  Or just a certain part of your faction.  Examples:

1. You’re The Little Fish

Let’s say that perhaps the nanocaust is being absolutely terrifying, and they are a third-party against you and the AI.  A faction like them is now able to piss off the AI enough to cause extragalactic war units of the AI to appear.  Those are things like the flenser, at the upper levels, where just one of those is absolutely game-ending for you as a player if you don’t have the fallen spire or some other sort of mega faction going on on your own side.

Anyhow, previously we couldn’t use something like the flenser in this fashion, since the AI would eventually probably turn it on you and end the game prematurely and unfairly.  But now we can keep the flenser fully focused on whatever aggro’d it so much in the first place — the nanocaust, in my current example.

This means that you can be present in the galaxy while the AI is fighting a much larger foe than you are, using much larger firepower.  This is… just plain cool.  It also has strategic ramifications, as you’ll want to avoid the planet with the flenser while it’s fighting the other foe, even though the flenser will never fully turn on you and go wreck all your planets.  It’s just going to be a fight that doesn’t involve you, but that you have to work around, and the flenser will head back out of the galaxy to deal with its war there once it deals with the target here.

(Of course, the flenser is an extreme example.  Mostly you’d see smaller extragalactic war units making an appearance.  But mods, and probably DLC2, could aggro the AI enough to cause a really strong faction like that to get into a war that is larger than the one you yourself are engaged in).

2. A Specialist Group Is After Your Treasure

Right now this is used for Major Data Centers (MDCs) in particular.  Previously when you captured those, the challenge that they presented was exogalactic waves that would spawn against the planet of the MDC while you were hacking it.  If you survived the entire time, then you’d get to keep the MDC.

That mechanic is… overused and kind of boring.  For MDCs, we no longer use exogalactic waves at all.  Instead, the AI hunter fleet gets a bit of extra budget with some specific objectives to go kill your MDCs.  This makes the AI have an appropriate response to you getting the new treasure, but it’s not in the form of this series of kind of hacky waves that come at you over and over.  Instead these are groups of specialists in fireteams who are intent on intelligently taking out that treasure when they can.  This makes the AI seem much more alive and intelligent, versus just a dump of units at you that you have to withstand temporarily.

3. Changes To Player-Allied Factions

Your allies will no longer go around popping warp gates, which was causing AIP to increase.  Instead the AI is now able to use these new mechanics with specialist hunter teams to hunt your allies appropriately.  In other words, the AI is now able to mount a proportional response to actions, but without your allies having to do something annoying like destroying warp gates that you might prefer consider existing.

Other Items

  • Holy cow, the one bad thing about an extended beta is that it can delay certain fixes.  One of those was that all of the AI (and I think Outguard units as well) had the equivalent of mark 1 hull and shields for the last couple of months if you weren’t on the beta branch.  That’s going to be a bit of a jump in difficulty on campaigns that were vanilla if you were on the last non-beta one all this time — apologies.
  • There were a bunch of other bugs fixed that were less likely to affect everyone, but which were still annoying and which are great to finally have on the main branch.
  • A bunch of achievements that were not triggering correctly now do.
  • The Parasitic Starting Fleet has had its balance improved, and should be more attractive to start with now.
  • AI ships from spire debris are now a lot more dangerous.
  • The Imperial Spire Fleet now gives you vision on their planets so that you can watch the fireworks.
  • Multiple copies of the same faction type (whether they are friends or enemies to one another) now try to spread themselves out in the galaxy to make things a lot more interesting.  If your playing with good and evil marauders both at once, then they won’t be neighbors most of the time, now.
  • We improved how we are doing our fixed-integer math, which fixes some math rounding issues that could lead to a bit of odd data with certain bonuses on the UI in particular.
  • Fixed a very unusual bug where the AI was basically able to get frigates “for free” as part of their reinforcements in the rare cases that they actually used them.
  • Make the Marauder less likely to suicide themselves against Praetorian Dragons.
  • During gameplay, when you click into the Galaxy-Wide Options screen it now shows you ALL of the values, not just the ones you can edit.
    As with the factions screen, it just has the ones that are non-editable as text values, but you can examine your settings this way and also see their tooltips.
  • If the game runs into an exception while trying to start up, it now gives you a visual error rather than the loading process just seeming to hang.
  • The Nanocaust and Dark Spire shouldn’t be able to rebuild a Nanobot Center or a Locus for at least a few minutes after a previous one on that planet was destroyed.
  • Cross planet waves against a specific faction now will stay focused on that faction (and not go off and attack the player).
  • The “brighter color” hex and color for faction teams now uses new logic by -NR-SirLimbo to make sure that faction names should always be nicely readable even if their actual colors are normally very dark.
  • Various more voice lines have been integrated into the game.
  • The game now has a Notification for when Dark Spire Loci are warping in.
  • It turns out there were some sort of edge cases where the AI king could be dead but its faction would not have gone through the “we have lost” logic. We don’t yet know how that could possibly be possible, but we now have a safety check from then on in the future that makes sure that even if the kings death was somehow missed when it happens, it now recognizes that it happened in the next few seconds and will mark the faction as defeated.
  • Fixed a SUPER longstanding bug where it would sometimes write “Was looking for a wormhole (some stuff) but couldn’t find one.”  This was also causing some bad ship behavior.   It’s been around, but rare, for years now.
  • Fixed a “fun” bug where some enemy ships with fireteamss would be unable to decide between targets; they would get partway to one target, then turn around as if going to the other target. This looked like a Buridan’s ass style problem with the targeting code.  The actual problem was that Fireteams were incorrectly declaring themselves winners of a battle without showing up.
  • Fixed what appears to have been a relatively old bug (somehow?) where the max strength of a lot of fleets was not being properly calculated. It seems like this may have slipped by for a few months, or since a bit before DLC1 somehow.
  • Also fixed a bug that probably goes back to the start of fleets, where instead of the current strength of fleets on the fleets sidebar, it was showing their total strength. For command station fleets, it was showing a max that they could not even hit, too.

More to come soon!

But here’s some further reading on what we’re doing in detail:

Multiplayer Schedule, And Why

It feels like an endless wait for multiplayer, doesn’t it?  I hear you on that — I think it probably feels even longer for me, since I’m the one working directly on it daily and it’s been so many things.

I wanted to take a minute to step back and address WHY things are taking so long.  You may or may not know this, but we actually had fully functioning multiplayer for this game back in 2017, before turning it off because we couldn’t reliably try to keep sync in the game while building out so many features like we were doing until late 2018.

It seems like it should have been a simple matter of just turning back on that old code, right?  Well… unfortunately as we developed the game, it got a lot more complex.

Essentially, the first AI War existed in a perfectly deterministic state where, after initial game sync, nothing had to be sent between clients and the host except for commands from the players and the higher-level AIs.  It also sent some basic check messages to make sure that results were not diverging on your machines — and if it detected a divergence, then it would flag that as a desync and stop you from playing until you saved, reloaded, and had the clients reconnect.

That determinism was achieved through a lot of really rigorous coding practices, and using fixed-integer math instead of floating-point math, and only using one thread for the main simulation.

In AI War 2, we can’t really enforce how rigorous modders’ code is; we also are using many threads, all of which are designed to be deterministic but which… turns out sometimes are not because of exceptions that we can’t solve in a deterministic way without slowing the entire game way down; and we have to use some floating point math here and there (forcefield pushback, for one) in order to get the accuracy and performance the game needs.

So what this means is that the entire premise for multiplayer in AI War 2 got more complicated.  The game is MOSTLY synchronous, MOSTLY deterministic, and does a great job at running blazing fast.  But desyncs can and will happen, probably with some frequency depending on how different your PC specs are from the people you are playing with (or what errors in rigor have been made in mods you want to use), and the old approach of stopping the game and making you reload and reconnect is absolutely unacceptable.  You could be running into those sort of issues once an hour, or once a minute, depending on the circumstances.

That means that, while we strive for determinism as much as possible, we also need to have automatic desync repair.  This is familiar more for action games (which exist in a state of constant desync by nature) rather than strategy games.  I have experience with this from the A Valley Without Wind games, but they have an incredibly small fraction of the amount of data that AI War 2 has going on.  It’s a much less intense environment as a multiplayer coder.

Anyhow, this has meant that the environment has to be extra well-prepared for sending efficient messages, since there will be more of them.  It also has to be able to fix small problems in a spot-fix fashion without re-syncing the entire game.

These added challenges meant that I needed to restructure some substantial bits of the code that were working perfectly fine if you look at them from any other context.  I still have some areas to deal with for that, but the list is getting much smaller.

We’re still on track to have multiplayer “in the next few months,” but I realize I’ve been saying that for months now.  The really good news is that alpha of multiplayer should be way more smooth, and a lot shorter of a period before we can get to the point of it being fully polished. 

I’m keenly aware that as soon as folks can play multiplayer, they will do so; and if it’s a broken mess on the first alpha, and it takes me months to make meaningful improvements to it, I will lose my testers.  So I’m being extra cautious and hitting all the areas that I expect I will run into before even starting the first alpha.  That should ultimately be a lot less stressful on me in a time-pressure sense once we start doing multiplayer alpha.

What’s Remaining For Multiplayer?

So what is the todo list, then?  It’s getting shorter!

  • Now that the compiler chain is improved, I need to get Steam and GOG linked in a better way, and also get Forge Networking linked in that fashion.  This will be FAR easier than what we were doing before, and more in line with how those respective libraries expect you to use them.  I’m very excited about this being possible, finally.
  • The actual cycle of multiplayer messages being sent and received, and that keeping the game flow going, is done and has been for years (literally since early 2017), but I need to reintegrate that into our core dlls and improve the efficiency of some bits of it.  It’s not large amounts of code, thankfully.
  • I need to redo how the initial world state is sent to new clients joining in, but that will also be quick — thanks to the rework of world save data styles and sizes.
  • Our old style of desync-detection from 2017 has been stripped out now, and I need to add in a new kind of rolling desync detection that will find problem entities and then allow us to fix them.  To keep this fast and relevant, I’m only going to care about a few things: position, health, and shields.  If those match, then probably everything else is close enough.  How to handle the time differential on this is really challenging, and only in the last couple of months did once facet of this occur to me.  I have a couple of solutions in mind, but it’s going to be… “fun.”
  • The way that we send information in multiplayer is by GameCommand, and right now those are generic and bloated (they are overly multi-purpose).  I am going to replace those with a new set of custom GameCommands that us developers and modders can use as-needed and have them  take full advantage of the new smaller data sizes.  Fun fact, this will also make single-player run very slightly faster when a lot of things are going on.
  • The really big one that remains is making sure that the cross-machine identifiers (PrimaryKeyIDs) are consistent between machines.  I don’t fully have this figured out yet, but I think that the interim state of it will essentially be that there are occasionally too many messages being passed around because of rolling sync errors.  I will probably punt this issue into something I look at during the alpha, so that I can gauge what sort of impact it really has on performance, and where the problems are coming from.
  • After that it’s more or less a matter of making sure that the lobby fully works as we expect, and various other small UI systems to get multiplayer basically playable.  A lot of work went into the lobby in particular in the last few months to make that as close to as ready to go as possible.
  • There are then whatever changes we need to make to balance in order to make things “feel right,” which will be a matter of working with the multiplayer alpha and beta testers.  A lot of things we already did in the past, like making science collection a humanity-wide thing that each player gets a copy of, rather than something people have to do individually (what a pain that was in AIWC).  We will have to scale waves like we did in AIWC multiplayer, or in some other fashion.  But a lot of the difficulty scaling is inherently handled by AIP being higher when you have to take more planets in multiplayer.

So, to sum up the plan for the short term:

  1. I need to re-link the networking libraries using the new compiler chain.  This might take a couple of days.  I’ll do it on a beta branch in the next week or so, at least for a bit, because it will potentially break the ability to log achievements on some machines or OSes until I get it fixed back properly (assuming it doesn’t work on try one).
  2. Then I will pull the network-loop code back into the main game, which is a part of one day job.
  3. Then I need to generalize that to work for all three networking code paths (Steam, GOG, Forged), which probably will only take a couple of days at most.
  4. Then I need to re-code GameCommands to be more efficient and special-purpose.  This is probably a job that is a couple of days long, and will potentially lead to widespread bugs for a week or so after it.
  5. Then I need to implement V1 of the desync detection and correction code, which should probably only take a few days.
  6. Then I need to get the basic connection interfaces working, so that you can choose to connect to players via Steam, GOG, or a direct IP address (forged).  This will be a moderate challenge, because Steam and GOG both have APIs that I’m pretty unfamiliar with in this exact area.  Once the connection is going, it’s much easier.  But with the new compiler chain, at least I can hopefully get these going in a matter of days or a week rather than it being something that takes multiple weeks.
  7. At that point we should be able to reasonably declare that the multiplayer alpha is ready, as you should be able to connect to friends and play with them, and the desyncs would cause network performance degradation rather than something permanent that your game can’t recover from.
  8. Then it’s a matter of the other features on multiplayer, mainly regarding things like donating fleets between one another, and/or whatever else we come up with that is desirable.
  9. Oh, and during that period if we’re seeing network degradation or other issues due to the constant need to sync errors, then that will be to be investigated and improved.  But those things are most of what the focus of the alpha/beta will be on.

How long will each of those line items take in reality?  I guess it depends on what gotchas I run into that are unexpected at the moment, but you can always watch the release notes to see how things are coming.  A lot of the really hefty things that required major reworks of the game are done, so that’s a win.

There are some things that I’d probably prefer to do sooner than later, like allow you to enable and disable expansions/mods from the main menu without a restart of the game, so that might take a chunk out as well.  But it should make things easier on everyone as they get more into multiplayer.

Please Do Report Any Issues!

If you run into any bugs, we’d definitely like to hear about those.

The release of this game has been going well so far, and I think that the reviews that folks have been leaving for the game have been a big help for anyone passing by who’s on the fence.  For a good while we were sitting at Overwhelmingly Positive on the Recent Reviews breakdown, but there have been a lot fewer reviews lately and so that has definitely had a material negative effect.  Go figure.  Having a running selection of recent reviews definitely is helpful, but at least we have a pretty healthy set of long-term reviews.  If you’ve been playing the game and enjoying  it, we’d greatly appreciate it if you’d drop by and leave your own thoughts, too.

More to come soon.  Enjoy!

Problem With The Latest Build?

If you right-click the game in Steam and choose properties, then go to the Betas tab of the window that pops up, you’ll see a variety of options.  You can always choose most_recent_stable from that build to get what is essentially one-build-back.  Or two builds back if the last build had a known problem, etc.  Essentially it’s a way to keep yourself off the very bleeding edge of updates, if you so desire.

The Usual Reminders

Quick reminder of our new Steam Developer Page.  If you follow us there, you’ll be notified about any game releases we do.

Also: Would you mind leaving a Steam review for some/any of our games?  It doesn’t have to super detailed, but if you like a game we made and want more people to find it, that’s how you make it happen.  Reviews make a material difference, and like most indies, we could really use the support.



AI War 2 v2.062 Released! “Savegame/Networking Data Compaction”

This is on the beta branch, as it’s a freaking huge number of changes.  The more testers the better, please!  Full notes here.

How To Play A Beta Branch Build

To play this on Steam, please go under betas and choose current_beta from the drop-down list. We would really appreciate some testers on this so that we can get back out of beta status as quickly as possible! There is not currently a way to get the beta versions on GOG, but we won’t be in that status for more than a week, knock on wood.

We want to make sure we didn’t break anything with all the substantial changes in here, but we fully expect some savegames to throw cosmetic errors at the very minimum. Please report and upload those here:

What’s New?

  • The biggest thing for me personally is the new data compaction stuff, which I have been pouring hours into in preparation for multiplayer.  The last thing I want is for multiplayer to be functional, but immediately laggy as soon as we have the desync repair stuff going.  I needed to start with efficient data storage and transmittal so that there wouldn’t be a point where I need to disappear for a week to redo all that after we’re into multiplayer’s alpha.
  • I’m not entirely done with my data compaction shenanigans, but I’m very excited about how it has turned out so far.  I do expect it to result in a number of user-facing errors, though, so hence the beta, which I really hope you’ll partake in.
  • Badger fixed a number of bugs, including several achievements that were not triggering.  Also that all the AI units have had the equivalent of mark 1 shields/hull for the last two weeks??
  • Meanwhile, Badger also implemented another major fireteams upgrade, this one which is intended for the upcoming DLC2 but has been backported to the base game.
  • The first use of this fireteams code is to let the AI really fight some other faction other than you with giant fists, without killing you in the process.  Aka, let it fight the Marauders or the Nanocaust using REALLY BIG AND SCARY GUNS if those two factions are taking over the galaxy… but don’t have that be game-ending for you.  You’re not the problem, after all, from the AI’s point of view.
  • The second use of this is to let us have certain sub-groups of the AI faction, most notably Hunters right now, which have a sub-subgroup that just focuses on one objective after you harass them.  The example in place right now is Major Data Centers (MDCs).  Those always triggered an Exogalactic Wave, previously, which was boring — more ships come in increasing waves, and you survive them or don’t.  It felt kind of uninspired.  NOW, instead, the hunters get a major buff to a certain sub-team that explicitly hunts the MDC and its protectors… and if it wins against that, eventually, then they just leave you alone — “job’s done, bye!”
  • The way that the hunter deals with allied factions to you is a lot more interesting, too, rather than your allies causing rapid AIP gains by killing all the warp gates and aggroing the AI in that way.
  • These are some major strides forward towards multiplayer being a smooth and fun experience as we get into that, and towards the DLC2 that will come out around the time multiplayer fully releases (multiplayer will be free).   Please note that none of the DLC2 stuff is slowing down multiplayer at all, as Badger is not working at all on multiplayer and I’m entirely focused on that and related things for now.
  • Oh, also, for the sake of modders we’ve documented some new things that will help with making sure mods are multiplayer-compatible.
  • Note that if a mod you like turns out NOT to be multiplayer-compatible, the game should still be playable.  You’ll just have a ton more desync-repair messages going back and forth, related to the mod.  This is an excellent example of why I want those desync-repair messages to be absolutely as efficient as they possibly can be, which means representing all our data in as tiny a format as possible.

More to come soon!

But here’s some further reading on what we’re doing in detail:

Why Do We Care About Compaction?

I explain what I mean by this below, and go into detailed benchmarks.  The release notes have a lot of other detailed info.  But a TLDR of why this matters is certainly a good place to start:

  • Smaller savegames are easier to transmit across the network when you’re initially synchronizing your game from a host to clients.
  • Doing it via our compaction method, versus compression, uses FAR less CPU cycles, and thus makes the transfer of even small, frequent bits of data run more smoothly.
  • Doing it via our compaction method makes even small bits of data… smaller.  Compression often requires a lot of data before it makes much difference.
  • Given that we are going to have frequent small commands being sent back and forth just to have the game run at all, and then we ALSO are going to have the desync-repair data going back and forth that is still small (but substantially larger at the scale of network messages), this is pretty critical.
  • So the TLDR of the TLDR is that we use less bandwidth and less CPU to do the same thing, and thus your game will run way more smoothly in multiplayer.
  • And it has the side benefit of making savegames a lot smaller, and also faster to save.

Data Compaction vs Compression

What the heck is data “compaction?”  Well, in the most direct sense here, it’s a term I made up.    The way I’m defining the term, for purposes of what we’re doing here, is “making data smaller at the level of small objects, as we put it into a binary stream.”

Wait, what?  Basically… it’s something we do as we go along, and it works on very small pieces of information.  A single integer.  One ship and its data.  That sort of thing.

Compression, by contrast, requires a lot of processing AFTER you put in your data to some sort of format.  So to compress a bunch of data in AI War Classic, for instance, we write roughly 30MB of data for a savegame  into a temporary buffer, and  then we run GZIP compression on that, which turns it into maybe 1MB that we can store on the disk or send across the network.

That 30:1 ratio is legitimately what AIWC was seeing, on average, by the  way.  The downside is that it only really works well for very large amounts of data, and it also requires a lot of processing every time it wants to send or save.  You’ll notice that AIWC has a noticeable hitch in itself every time it saves the game.  AI War 2, by contrast, doesn’t use compression and thus doesn’t have that sort of hitching.

AI War 2 is a lot smarter about how it represents its data to start with (using binary formats directly  instead of a unicode-based format), and uses clever tricks to also know when not to send data of certain sorts (if I don’t have X properties, then don’t bother tracking them).  So this realistically converts what would have been 30MB in AI War Classic into something that might be… 5 MB in AI War 2.

Beyond that, though, requires a lot more work, because we actually have far more data here than in AI War Classic — enough that despite our better formats and such, we’d be back up around 10 MB or so for savegames.

Several years ago, perhaps four now, Keith LaMothe came up with a great new concept he called “buzzsaw binary,” and we posted benchmarks on how incredibly efficient it was at storing data.  After working out some kinks with it, we’ve been using that for the last few years, and it saves blazing fast, as you’ve probably noticed.

Basically, this takes each data point that we want to store in binary and writes it directly into a bitstream (which C# does not officially have — they only have bytestreams).  As it writes to  our custom bitstream, it analyzes each piece of data and figures out if it can make it any smaller than would normally happen for that kind of data.  The types of data that Keith built support for were string, 32bit integer, and 64bit integer.  The last of those also gave us support for our own FInt fixed-integer format, and by extension also some limited float support (thought we avoid float as much as possible).

The plan was always to go further, and he left notes to himself to do so directly in the code, actually.  With this format, storing a byte was actually highly inefficient, taking on average 10 bits rather than 8. But we never even tried to directly store bytes, so that was kind of irrelevant.

In most programming languages, integer numbers can be stored as 8, 16, or 32 bits, with or without the ability to have negative numbers (called signed numbers).  In the prior edition of buzzsaw binary, we assumed all numbers were signed and could be up to 64 bits, and wrote them out in a manner that still produced results that were often 16 bits or less for a 32bit number.  That was a savings of 50% or more, in lots of cases.

A big part of the savings is what to do with “default case” numbers, such as those that are 0.  How many bits should you take to store that zero?  By default, a 32bit number will take 32 bits to say 0, and a 64bit number will take 64 bits to say 0.  In buzzsaw binary, regardless of the bit level of the number, it has always taken us 1 bit to store a zero.

One tricky thing is that we use a lot of -1s as defaults for various reasons, and that was something that required 10 bits to store. Ack!

So for me, I went in and finally figured out Keith’s code, and added a ton of comments to make it clear what was going on.  I added explicit support for

  • 8bit (aka byte) and 16bit (aka short) numbers
  • And some special cases for:
  • numbers of 16 or more bits that cannot be negative
  • numbers of 16 or more bits that cannot be any smaller than -1
  • and 8 bit numbers (bytes) that are frequently 0

Those are… strange categories, I know.  But when you look at the data we use, and apply the same sort of buzzsaw binary approach, but with more hinting from the higher-level program (as well as more appropriate number formats for variables instead of int32 for everything), then you wind up with some truly amazing compaction.  I’ve detailed the macro results of that here.

Bytes now take an average of 4.74 bits in a savegame, down from 10ish in our previous implementations.  In the benchmarks I just not the percentage relative to the normal 8 bits. “32 bit numbers that cannot be less than -1” are averaging out at 2.71 bits instead of 32.

Overall file sizes for the game dropped in this build to commonly about 70% of what they previously were, and as low as 28%  in one case.  The actual data format change only represents a savings of 10%, sadly, but it’s still a win, even if it is a minority of the gains.

Please Do Report Any Issues!

If you run into any bugs, we’d definitely like to hear about those.

The release of this game has been going well so far, and I think that the reviews that folks have been leaving for the game have been a big help for anyone passing by who’s on the fence.  For a good while we were sitting at Overwhelmingly Positive on the Recent Reviews breakdown, but there have been a lot fewer reviews lately and so that has definitely had a material negative effect.  Go figure.  Having a running selection of recent reviews definitely is helpful, but at least we have a pretty healthy set of long-term reviews.  If you’ve been playing the game and enjoying  it, we’d greatly appreciate it if you’d drop by and leave your own thoughts, too.

More to come soon.  Enjoy!

Problem With The Latest Build?

If you right-click the game in Steam and choose properties, then go to the Betas tab of the window that pops up, you’ll see a variety of options.  You can always choose most_recent_stable from that build to get what is essentially one-build-back.  Or two builds back if the last build had a known problem, etc.  Essentially it’s a way to keep yourself off the very bleeding edge of updates, if you so desire.

The Usual Reminders

Quick reminder of our new Steam Developer Page.  If you follow us there, you’ll be notified about any game releases we do.

Also: Would you mind leaving a Steam review for some/any of our games?  It doesn’t have to super detailed, but if you like a game we made and want more people to find it, that’s how you make it happen.  Reviews make a material difference, and like most indies, we could really use the support.



AI War 2 v0.749 Released! “Release The Hounds”

Are you a programmer interested in helping out on a multithreading issue?  Be sure to see the bottom (or this forum thread).

Case Study of Modding: Marauders

Before talking about the release or multithreading, this is a great time to talk about the power that our Modder #1 (and volunteer developer to boot), Badger, has been able to exert thus far.  He’s made a ton of factions, but right now let’s talk about Human Marauders, which were in the first game as well.

If you don’t remember them from the first game, that’s because they weren’t too exciting; marauders were units that would periodically show up from the gravity well edge (not a wormhole) and cause some trouble. They were hostile to the player only, and were generally pretty insignificant.  It added a tiny bit of spice, but not much.

Enter Badger.

He started from scratch when implementing these in the new game.  His version of Marauders are hostile to everyone.  They will attack any system they deem “weak enough to take,” then drop in from the edge of the gravity well.  If they destroy all the defenses, then they start building Outposts, turrets and additional ships. They’re already starting to defend the planet as their own.  If you leave the planet alone, the Marauders will keep making outposts, and each outpost will get stronger (ie it will build stronger defensive ships and more turrets).

Once an outpost hits Max Rank, it starts to build Raiders (powerful starships). Once the Marauders have built enough Raiders, they will attack adjacent-through-wormhole systems that they think are “weak enough to take.” If you leave outposts at Max Rank, the marauders will be able to attack more and more often — their “Attack Budget” gets bonuses based on how many Max Rank Outposts there are. Also the Max Budget gets increased every time they capture a planet.

Fun fact: If you take out all the defenses on a swath of AI planets around your empire (but haven’t actually captured or defended the planets) then the Marauders will rapidly take all of them over and potentially become a real threat.  To you… and the AI and other factions.
TLDR: Instead of just launching quick raids, these Marauders are here to take their own stab at conquering the galaxy.

People have been giving lots of balance feedback on the Marauders, as well as positive impressions for that faction on Discord.  This also illustrates two pretty key points:

  • The marauders are an example of what can be done with the modding tools of AIW2, but which was basically impossible for AIWC even for us as developers.  Gives a bit of context as to why making this sequel was important.
  • It’s also a really good example of how a “Decent but not exciting” AIWC faction turned into something way cooler in AIW2, so buy AIW2 to get cooler stuff. 😉  There are a lot of factions like that (all-new ones and revised ones), even though we’re pivoting the core mechanics to be more like AIWC.  For anyone worrying that this is just AIWC HD, please fear not!

The Actual Release

Okay, right: release notes here.

Where to even start?  This release is pretty sizeable.

  • There’s been a bunch of balance to minor factions.
  • Mercenaries got some extra oomph.
  • AI Waves were previously capped at 100 units (cough, whoops), but now will gleefully flood your planets properly.
  • There are auto-build options for your convenience now available in the game settings.
  • The salvage and reprisal mechanics from the first game are added back in.
  • Golems are now available again — five of them, anyway!

…and all of that stuff is just what Badger, a volunteer, has put in this release.  To say we’re indebted to him is an increasingly sharp understatement with every week.  Holy smokes.

On my end of things, I upgraded the game to Unity 2018.2 and mono-.NET 4.6.  Some performance improvements were possible from this, and a lot more multithreading options.  A few boosts happened out of the gate, and I was able to explore (and then discard) the Lightweight Rendering Pipeline as an option.

I spent a fair bit of time on the multithreading problem, but at this point I’ve been hitting a wall where I can’t get the performance any higher.  I’m sure with more time I could figure it out, but in the meantime there are bigger fish for me to fry in terms of performance blockers (namely the front-end vis layer’s extreme performance hits, which I talked about two releases ago).

Multithreading Help Wanted

Last release I asked for some help on the multithreading problem, which was clarified here.  At this point, I’ve basically hit a wall in my ability to improve the threading without getting really excessive in my expenditure of time.  There are a variety of other things that really need my attention right now, so I’m having to put this on the back burner.

That said, I’ve open-sourced our multithreading code and our core sim loop, so if there are any kind souls who want to take a look at it and help more directly with revisions, that would be super appreciated.  I know folks were a bit hampered by being blind to the code before.

How to get at/test code:

  1. Make sure that Steam has updated you to the latest build of the game (0.749 or later).
  2. Inside the install folder for the game, open the AIWarExternalCode.  There you’ll find a AIWarExternalCode.sln visual studio solution.
  3. You can open that with visual studio community edition 2015 or 2017, and then it should let you compile directly to the GameData/ModdableLogicDLLs folder.
  4. Running the game after having compiled a new version into that folder will run it with your changes in place.
  5. If you want to use a different IDE, or even no IDE and just compile manually, you can do so.  Badger has scripts for doing so on linux, for example.
  6. The actual scripts of relevance are in the AIWarExternalCode/src/Sim folder (and its subfolders), which has a variety of C# files that I’ve added comments to.  There are other files that call into and out of those files, but all of the multithreading bits and the main-thread-bits that call them are all there.
  7. I’ve made a mantis ticket on our bugtracker that has the savegame, plus instructions on how to see the issues.

Any help is appreciated!  And I’m happy to answer questions.  I have made a forum thread that is probably the best and easiest place for us to discuss this.

Right, Back To The Release

Anyway, that’s enough out of me.  This new release is pretty cool.



I again wanted to mention: we have a new Steam Developer Page.  If you go there and follow us, you’ll be notified about other upcoming releases (including this one, of course).

Pivoting AI War 2: Bring The Fun!

Hey all — Chris here.

We’ve hit a juncture point with AI War 2. We’ve built a lot of cool things, learned a lot, and now it’s time for a soft reboot. The current plan is to pivot the gameplay to very closely resemble the original AI War, but on our new engine, and then build up from that foundation.

Achievements So Far

So very much is going right with this game, from a technical level and an engine standpoint.

  • The game is crazy moddable.
  • It’s multithreaded to take full use of modern computers.
  • The 3D aspect is working out well.
  • We’ve figured out a variety of new tricks that definitely do improve on the first game, and can be kept.
  • The UI has already been dramatically improved by the introduction of a tabbed sidebar in the main view, and streamlining of several other mechanics that felt very difficult in the past. Eric (as a volunteer) has been a godsend for the UI, and we have pages and pages of more designs from him that we’re going to be working on in the next two months or so.
  • Badger (as a volunteer) has been an incredible tester, volunteer developer, and general help to getting us this far at all; he’s created the Nanocaust faction, as well as a new and better implementation of both Human Resistance Fighters and Human Marauders, among many, many other things.
  • Folks like zeusalmighty, chemical_art, Draco18s, and Magnus have been wonderful sources of thoughtful feedback, commentary, testing, and even map creation.
  • We’ve got art for over 130 distinct units (not counting different mark levels), and we’re set up well to finish off the rest of the art despite the staff changes noted below.
  • We’ve got over 1500 lines of spoken dialogue from more than 25 actors, focusing primarily on the human side at the moment; we have a few hundred lines of AI-side taunts and chatter, some of which is recorded but just not processed yet.
  • There are hundreds of high quality sound effects for a varied battlefield soundscape (with distance attenuation if you’re far away, and positional 3D audio if you’re down in the thick of it), all routed through a tuned mixer setup for optimal listening to all the various parts.
  • We have a set of music from Classic that is over four and a half hours long, and the new music from Pablo is partly in, but mostly set to be mastered and integrated within the next week or two.
  • There’s also a ton of map types, many of them new, and with a lot of sub-options to make them even more varied.
  • We’ve created half a dozen custom Arks as backer rewards, we have another four in various stages of completion, and there’s a lot of cool variety from those folks.
  • With a lot of the other custom art-related rewards (custom flagships, fortresses, and gold merc paint jobs — 21 backers affected in all, when you include the custom Ark folks), since we’re having staffing changes in that particular area, we’ve offered alternative options to those backers, yet said we’d honor the original reward if they prefer. We’ve had a mix of both responses, both of which are fine, and things are proceeding well there.
  • Other backer rewards are either already delivered (game keys of all sorts, many of the custom Ark rewards, backer badges, antagonistic AI voice line writing), or something that are still on the todo list but easy to handle prior to 1.0 (custom wallpapers, planet names, cyber ciphers, antagonistic player voice lines, custom AI personalities, and other non-art merc bits).

New Teaser Trailer!

On that note, here’s an excellent new teaser trailer that Chris and Craig created together. It’s light on details, but it’s just a teaser, after all:

The Sticking Point

The new game just isn’t living up to the first one’s legacy. We started out with a lot of design shifts away from the original AI War, and the design just hasn’t been as robust or fun as the original.

  • In AIW2, so far, there was no real sense of logistics. Things felt too simple.
  • The combat was basically getting you to just “fleetball” all the time, though that wasn’t our actual intent.
  • The defensive options felt too limited no matter what we tried, and player Arks wound up sitting away in a corner with their offensive fleet having to return home frequently to help with defense.

The Two Paths

We’ve done quite a lot of engine work to make the actual game that runs on top of it mostly data-driven, so we have a pretty decent amount of flexibility here. For the last few months, we’ve been chasing various issues in gameplay, trying to tidy those up, but it just kept feeling less and less “like AI War.” So, we had two options:

  1. Keep doing that and hope for the best, particularly that it magically starts feeling “like AI War” again.
  2. Go back and actually make AI War again, at least the base game, and then build from that foundation rather than starting way off somewhere else.

As you have likely already gathered, we’re going with option 2. As players, Keith and I have been really let down by how different certain sequels felt from their predecessors, and we really didn’t want to do that to you folks.

We want this to be the sequel you truly wanted, that takes the original game and then goes forward in a refinement fashion. Total Annihilation turns into Supreme Commander, not SupCom becoming SupCom 2. Age of Empires 1 begets AOE2, not AOE2 morphing into AOE3. All of those games listed are good, but there’s a reason that the second in each series is typically more acclaimed than the third.

Future Growth

We do know that some of you backed for something more radical in departure from the original game. Why have the same old experience again? That’s certainly a valid point, and that’s why we talk about this as being a foundation for future growth.

Look at how much the first game grew from version 1.0, way back in 2009, through six expansions and version 8.0 in 2014. They’re radically different games. That said, we were constrained at every turn by an engine that was designed for street racing, and that we were trying to take offroad. That just doesn’t work.

The new engine for AI War 2 is so robust and flexible that we can take it street racing, offroad, or underwater. Maybe we can have our cake and eat it too, at least eventually? Based on the underlying engine, there’s nothing stopping us from having n factions, xyz ships, and all sorts of new sub-games and mechanics on top of it if the response to the baseline is positive enough.

One example: We’ve floated a variety of crazy ideas about hacking in the last few weeks, for instance; and while those are Way Out Of Scope right now, there’s nothing stopping us from implementing those exact systems or something like them a year or two from now, once we know the baseline game is fun and feels “like AI War.”

Second example: in the preliminary design document we’re working on, check out the section way at the bottom about using Arks as champions. That’s something that we want to attempt sooner than later, and it could be an enormous leap forward on the “radical new ideas” front. Same with the mercenaries section in that document.

Schedule Changes

At this point, we’re looking at Early Access (the “fun point” fulcrum) being sometime in July. That will give us a lot of time to further implement Eric’s UI and refine some visual elements and whatnot while we’re at it. Obviously, schedules change, and this is a tight one on the side of Keith’s core gameplay work.

THAT said, the transition toward the fun point is going to come in 5 overall waves of core features from Keith. The 1st wave being minimum set of units to have a functional, winnable and losable game; the 2nd focusing on core variety; 3 and 4 focusing on various toys on human and AI sides; and 5 wrapping up the last toys as well as adding the minor factions noted on the design doc as being pre-fun-point. (Nemesis and Spire are both post-1.0)

Hopefully we’ll have a general idea of our progress, and people’s reactions to it, throughout those five waves.

After Early Access starts, there’s a bunch more stuff to add and tune, and we think the 1.0 can still be October. Some of the stretch goal content (Spire, interplanetary weapons, possibly some merc stuff) may be after 1.0, but that was always the plan, anyhow.

Staff Changes

All the above said, this is not coming without cost; it’s a major financial blow to the company, and unfortunately we can’t afford to keep our longtime artist Blue after April. She’s been with us for five years, and will be sorely missed, but we’ve known for a while this might be something that had to happen (as did she).

We’re basically folding back down into a quasi-one-man company, although that’s giving me too much credit. I’ll be the only full-time employee, at any rate. Keith is part-time and has been for some time. With the AI War 2 project being almost a year over schedule, something had to give. For myself, I’ve taken on a lot of debt, and am about to take on more.

We Remain Committed

You better bet that the game is going to come out; we’re working hard to make this truly shine, not just as a half-baked, unenjoyable mess. We’re determined that this will arrive at 1.0 as something that we can be proud of and that you can enjoy for many hundreds of hours.

This Isn’t an Engine Overhaul

We want to emphasize this! The AI War 2 engine framework isn’t changing much. The engine we built basically kicks butt, with all the moddability and support for advanced UIs and multi-threading, and so much more.

What’s changing is what we do with that engine, back towards something we know was fun on a different (much worse) engine. That solid baseline will be something we can have confidence in, and will be a great place from which to grow.

Example question: “Is the engine is flexible enough to go back to the original vision of mobile Arks as your king unit, and no stationary home command station?” Answer: an emphatic YES. The engine is so flexible that you can designate a king-unit option in XML and select it through the interface. That king-unit could be a squadron of fighters if you want, or the largest spirecraft with steroid stats. All of that can be done, at this very moment already, without any need for more than XML edits.

The 40+ Page Design Document

Measure twice, cut once. We’ve just spent the last week going back and planning things. Here’s the detailed design document.

In general there are a few upcoming stages:

  1. Working on getting it to match the AIWC base game. (The Pre-Fun timespan.)
  2. Players declare it is as fun as the base game of AIWC was. (The “Fun-Point.”) We may take it to Early Access at this point?
  3. We start bringing in more features. (The “Post-Fun-Point.”)
  4. We release the game to 1.0, probably in October.
  5. We do more stuff to meet our obligations as well as our personal goals. (The “Post-1.0 period.”)

At this point, Keith and I are feeling like the feature set as planned for the pre-fun-point is pretty darn huge on its own, and then there’s a variety of stuff planned for pre-1.0 that makes it even larger. We weren’t trying to expand the scope, but such is life.

There are also a number of ideas of varying tentativeness for after the fun-point that we want to try, such as bringing Arks in as a champion style. Things like that should really make the game feel like it has been taken to the next level compared to the first.

Looking for Modders!

Did you know:

  • ALL of the game data is in XML in AI War 2?
  • Adjusting ship stats is as easy as using a text editor to change a few numbers?
  • Adding new ships is just a copy-paste and then edit situation in those same XML files? You can use temporary graphics, and we can do real ones later.
  • All you need is Visual Studio 2015 Community Edition (which is free) or similar in order to edit tons of pieces of code for the game.
  • You can program map types with ease, GUI things with pain (that’s just UGUI for you), and make AI tweaks and similar somewhere in the middle of those two poles?

We’ll provide as much help as we can in getting you the info you need, and documenting all of this as things go on. If you have questions about where anything is, you can always ask Keith or Chris. Badger probably also knows, and before long we hope to have a solid stable of folks who know this well enough to help others.

Further, I feel it’s worth pointing out:

  • If you disagree with us about something relating to balance, you have the option of tuning the numbers yourself in your local copy and then showing us why we’re wrong. (Of course you can still ask us to do it, as has always been the case — but we’re no longer a bottleneck.)
  • If you make something particularly cool, then with your permission we’re happy to integrate that into the main game as an option that people can access without having to download something separate.
  • We wouldn’t have some of the cooler features that the game has right now, like the Nanocaust or some of the more interesting Dyson Sphere behaviors, if it wasn’t for Modder #1 — Badger. We know there were more of you who wanted to get involved in that sort of capacity, and now’s as good a time as any.

What do we WANT from modders?

A good question was raised: what are we really asking of modders, here? Honestly, that depends on the modder.

Some folks like putting in interface bits to solve personal pain points that they had with the original interface. Others have ideas for creative extra factions — for instance the Nanocaust — and we’d love to have those be something that you’re working on as we move toward 1.0, rather than as we move toward 2.0. If it’s all the same to you, anyway, it’s more valuable to us sooner than later, if that makes sense?

But in general, it’s kind of a “hey, if poking around at games like this is your sort of thing, we’re throwing a party and you’re invited.” We’re happy to show you around the house, not just throw you into the deep end of the pool without floaties.

Short Term Goals

We’re going to be aggressively pursuing the Fun Point, with Early Access to follow; and meanwhile building up and refining the UI, controls, and so forth to be the best that they can be.

Long Term Help

On the further volunteering end of things: if you want to help out with any sort of balance testing or custom unit design using the mechanics that we decide on as final, then the XML is easy to edit, and our doors are always open on our forums and on mantis.

Thanks for your continued support!


AIWar 2 – “AIW2ModdingAndGUI” Modding Tutorial

This video is directed at external modders, for whom we’ve now set up a unity project that comes with the game and which makes it easy for you to add your own ships, models, sounds, visual effects, and more.

This does NOT cover code additions or changes, and data changes like balance statistics, etc. Those are handled through the AIWarExternalCode visual studio project, and the GameData/Configuration xml files, respectively.

Actually creating content for any of the above is beyond the scope of this particular video, which is more about how to get set up and find what you need. There is an ongoing series of videos that I’ve created that does cover all sorts of topics related to this, however.

And in the future I’ll create more videos that are more concisely focused on specific modder-oriented topics.

This video requires you be on version 0.109 or later of the game.

Thanks for watching!